The spring term is over

The qualitative class is over. I really enjoy it. I enjoy the materals Dr. Shulman selectc for us, enjoy the guests' presentation, and also enjoy the whole process of doing usability testing. Anything you have done is what you actually learned.

The class is over, but my interest of qulitative research just begins. The blogger will continue to record my learning shedule and notes. It is a good way to track back my ideas. During I wrote my final paper, I found it is important to take notes. It helps me clear my mind, help me to decide next steps, and help me generate more ideas to present my paper. This summer, I plan to catch up the rest materails which I have no time to read when I was busy on my final project. And keep posting my insights on this blogger.

Project Progress (week 10)

During the spring break, I did the first data collection of my project. Although there is just one participant so far, I already got some valuable experiences.

First, I got some advices about the questionnaire design. There are some misleading questions making the participant feel confused and I need to correct them. The misleading questions influenc the participant'sperceptions about the website. He checked the number iondicating high difficulty in the ranking sclae becasue he felt frustrated when searching for the answer. It might cause problems on validity of testing participants' "true" feelings about the website.

Second, there are several serious problems about the website found by the participant which were not found by me when designing the tasks. I am excited to see whether the next participant will find the same problems or some other different issues. I found myself actually "learning" from the subjects.

The first usbility testing took us two hours. It is logner than I expected. One reason is that the participant #1 is not only an experienced web user, but also has experiences of creating website and online database system. He is more sensitive to deficiencies of website design than general users and pointed out many details. The data collected from "thinking aload" is the part I feel the most valuble. The other reason is that he is my friend. So he is willing to spend more time getting familiar the website and trying to find as many problems as possible. This is the advantage to recruit people whom researchers know to be participants.

The participant #1 is not an Engligh native speaker. But I found he doesn't have difficulty to complete the test. So I decide keep using Engligh to do the test even though the participant uses the same mother tongue with me.

Here are just few things I learned from this testing. When I finish transcription, I beleive there will be more ideas coming out.

Reading Response (Week 9)

This week's G & C chapter 8 talks about interviewing. It is relevant to me because my project involves interviews. During reading this chapter, I wrote down any ideas coming up in my mind on my "qualitative research notebook."

I decide not to take notes during the interview. Because I am not a native English speaker, what I hear will need to be translated to my language in my mind for understanding. If I take notes during the interview, my concentration on the participants will be distracted. I think the better way for me is to take notes immediate after the interview session is over.

I plan to carry out my first pilot study next week, so I also read the Silver's chapter 4. It also talks about the interviews. The other concern mentioned in Silver's book for interviewing is that: we have to be careful about the "voice and experience" issues (Silver, p. 114-116). This issue causes problems if the researcher directly sees what the participants said as the facts or experiences they actually have. Sometimes it is hard to know participants' retrospective account is what it was really like.

This issue reminds me the importance of "triangulation" or "multiple methods" to enhance the credibility of the data. G & C p. 129 mentioned the other way to enhance credibility related to interview technique: to interview a number of different people in different organizational positions (if the site is in a particular organization), and the cridibility of data you gathered is enhanced if it can be confirmed from several sources.

In my project, I also take the same principle into consideration. I try to involve several data collecting methods: observation, questionnaire, interview, and screen mouse activities recording to enhance the credibility. But at fist, seeking for similarities among the data collected by different ways is not my goal. The purpose of the usability testing is trying to find problems. Even though I always got different opinions from participants, I am still successful on the purpose of "understanding the topic." Especially in the stage of pilot testing, I will keep open mind and expect for the diversity emerges.

Reading Response (Week 8)

Silver's Chapter 10:
This chapter talks about suggestions for writing the research paper. After reading it, I was encouraged somehow. The most valued tip for me is that "Do not put off writing; write as you go." This tip is also mentioned by Dr. Shulman in the class.

The case study illustrated in p.337 gives a good example demontrating the benefit of "writing as you go" which is to help to clarify the research ideas. Something wrong only can be realized when conducting the actual writing.

"If you delay your writing, you are asking for trouble (p.344)."

G & C's Chpater 9:
This chapter talks about two group discussion techniques, focus group and NGT.
Focus group is one of the most common-used techniques in usability testing. Because people might bring up different opinions when coversing with others, it is usually the complement to the individual interviews. Focus group is also seen as a good avenue to obtain suggestions and information for define the research focus or for the questionnaire/protocol design in the preliminary stage.

I decide to conduct the pilot study first and then decide whether to use focus in my project.

Project Progress (Week 8)

Because of the three mid-term exams for the last week and this week, the progress of my project totally slowed down. I already had a big all-over picture for my final paper but have not started the data collection. It has been the eighth week, I feel a little bit nervous. I plan to at least do the pilot study by the end of the spring break.

Annotated Bibliography

The list below is the bibliography for my project. I extract the sentences or paragraphs from the articles and think about how I will use in my final paper. The italic words in the brackets are the notes for sections or places in which I am planning to put the paragraphs or sentences as the reference.

Borgman, C. L., Solvberg I., & Kovdcs L. (Eds.) (2002). Evaluation of digital libraries: Testbeds, measurements, and metrics. Retrieved Oct 26, 2006, from http://wwwold.sztaki.hu/conferences/deval/
presentations/DELOSWorkshop4OnEvalreport.pdf

Digital libraries may be seen as new forms of information institutions, multimedia information retrieval systems, or information systems that support the creation, use, and searching of digital content (Borgman 2002). (Definition of Digital Library; Literature Rview)

Borgman, C. L., (2000). From Gutenberg to the Global Information Infrastructure: Access to Information in the Networked World. Cambridge, Mass.: MIT Press, 34-52.

In Borgman et al., 1996, the scope of digital library was broadened to encompass two complementary ideas:

• Digital libraries are a set of electronic resources and associated technical capabilities for creating, searching and using information.
• Digital libraries are constructed, collected and organized, by (and for) a community of users and their functional capabilities support the information needs and uses of that community. In this sense they are an extension, enhancement and integration of a variety of information institutions as physical places where resources are selected, collected, organized, preserved and accessed in support of a user community. These information institutions include, among others, libraries, museums, archives and schools, but digital libraries also extend and serve other community settings, including classrooms, offices, laboratories, homes and public spaces.

The above definition in Borgman et al., 1996 is the most complete one I have been ever seen which extends the scope of digital libraries in several directions, reflecting the contributions of scholars from a dozen disciplines. It moves beyond information retrieval to include the full life cycle of creating, searching and using information. Rather than simply collecting content on behalf of user communities, it embeds digital libraries in the activities of those communities and it encompasses information-related activities of multiple information institutions. And the most important is that the definition point out that museums are also included in the information institutions proving the digital library as an extension, enhancement and integration of themselves. (Definition of Digital Library; Literature Rview)

Jeng, J. H. (2006). Usability of the digital library: An evaluation model. Unpublished doctoral dissertation. Rutgers, The State University of New Jersey, New Brunswick, New Jercey.

· Methodology (Model, criteria, & instruments)

· Definition of digital library (Literature Review)

· The “Future Research” section suggests there should more study focusing on cultural variables. (Motivation)

· To verify the generalization of Jeng’s model (Motivation). Trying to find our more variables affecting user experiences, new findings, and new model for Museum website.

· Synthesis of previous findings is also considered in her design of instruments.

Gorman, G.E. & P. Clayton, Qualitative Research for the Information Professional: A Practical Handbook (2nd Ed.)

Huberman, A.M. & M.B. Miles (ed.), The Qualitative Researcher’s Companion

Saracevic, T. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49 (2), 350-69.

Tefko Saracevic (2000) analyzed and then isolated some 80 evaluation studies and suggested an evaluation must involve selection and decisions related to:

1. Construct that was evaluated

2. Context in which evaluation was conducted

3. Criteria that were chosen as a basis for evaluation

4. Measures reflecting selected criteria to record the performance

5. Methodologies that were used

A clear specification on each of these is a requirement for any evaluation of digital libraries (Saracevic, 2000). Therefore, this project follows the lines of these five aspects as the conceptual framework for evaluation. (Conceptual Framework)

Saracevic, T. (2004). Evaluation of digital libraries: An overview. Retrieved Oct.23, 2006, from http://www.scils.rutgers.edu/~tefko/DL_evaluatio
n_Delos.pdf

Glaringly missing from this list are evaluations of operational digital libraries in institutions, i.e. as instituted in academic libraries, museums, national libraries, public libraries and the like. Considerable amounts of statistics are being collected about these libraries, but as yet they have not been a subject of evaluation. As to evaluation, institutional digital libraries are a terra incognita. Yet, there are thousands of them. (Definition; Motivation)

Silverman D. (ed.), Interpreting Qualitative Data (3rd Ed.)

Screen Recorder Comparison

This weekend, I tried to search for any free or trial version screen recorder for the use in my project. And I downloaded three softwares and compared their advantages and disadvatages.

• Screen Video Recorder: 15 days trial version. The recorded mouse movement seems not very smooth. The audio recording only lasts about 30 seconds (maybe because it is a trial version).

• ZD screen recorder 2.5.0: Freeware. The recorded mouse movement is smoother. 5 minutes limitation recording.

• Auto Screen Recorder 2.1 pro: Freeware. No time limitation for both audio and screen recording. The recorded mouse movement is smooth. The text is clearer than the other two. The sounds and movements match very well. Although the post-processing is slower, this software is the most recommended so far.

Reading Response (Week 5)

The focus of this week is "fieldwork." Very practical. G & C's chapter 5 details prepared works for fieldwork. One of the "key factors for selecting the focus or topic" in P.68, which is "Is the proposed site neutral?", reminds me of the similar consideration for usability testing:

The usability testing has its root in the CS field. Originally, system designers/programmers use colleagues or themselves’ judgments to design and test their product before release. And it results in many hard-to-use systems coming up because those designers did not take "real users" into the consideration. Jeffrey Rubin (1994) wrote it in his book, Handbook of Usability Testing which is the classic reference in usability research, "the most important factor leading to the neglect of human needs has been that in the past, designers were developing products for end users who were much like themselves." A familiar site can be dangerous.

G & C mentioned (p.68) that most text on qualitative research exhort novice to use neutral sites for data collection, sites in which they are not personally involved as employees or colleagues, or in which their interest is not predetermined by existing relationships.

However, we still have to acknowledge that using familiar site is a bonus if you can put personal perceptions aside. I think it is really well said by G & C (p.69): The advice should be to use a site in which one’s powers of observation will not be clouded by personal preconceptions. And a experienced researcher develops an ability to use familiarity as a facilitator rather than a bias-prone burden.

Experiences shared

I asked my friend who has experiences of usbility testing in a company last summer. She shared with me about how they did testing. Because she was asked to promise never to share data, reports, and ideas with anyone, I only got the information of the overall process.

Steps:
1) test plan
2) protocol (script for moderator)
3) set up software and hardware for recording
4) go through the test with user
5) observe and take notes
6) edit the videos
7) write the report
8) present to designers and other stakeholders
9) put document and video on intranet

How to sample?
Rule of thumb: target audience of the system/product/website
Target audience might be different in age, gender, experience, etc. To avoid bias, don't recruit your subjects from only one category.

Number of participants?
5 will be enough for general usability test; 15 for card sorting tests.

The company my friend worked for has its own customer database, so they can always recruit people from there via emails. They need to take many criteria (age, location, status, etc.) into account by having screening questions to filter out unqualified people. But they only had 2 weeks for recruitment, which is too short, hence people responding and participating were not as representative as they were supposed to be.

The data sources, research question & theories of the project

A list of possible data sources
Best possible source:

• Contact with the usability group in Carnegie Museum of Art to request for the previous research data (or documents), web log records, or user study reports.

• Contact with Mr. Real who is the staff conducting the website in Carnegie Museum of Art. I can get information about collection backgrounds and system architecture from him. The information will help me design my questionnaire, interview protocol and testing tasks.

Sub-optimal source:
If there is no “oven ready” data (or not appropriate/adequate to fit my goal) in Carnegie Museum of Art, I need to collect data directly from participants. The testing will be laboratory-based and the data include:

• The results of observation
• Real-time issues captured by observers during the testing sessions (take notes)
• Transcripts of interviews
• Questionnaires
• Video and audio recording (digital files)
• The conversation of focus group meeting
• Cursor activities recorded by tracking software
  

Worst case:
If the above data sources are not available, the only way I may do is to synthesize findings and analyze the issues from the previous literature. And maybe doing comparison of the content and design with other museum websites with similar functions and features. I hope the worst case won’t be happened because I want to use user-centered methods in my study.

A summary of possible relevant theories
There is little research related to usability of museum website and most of the theories focus on the planning and design. Fortunately, the Carnegie Museum website can be defined to be one type of digital libraries by its nature and characteristics (I will elaborate it in my future paper). The usability evaluation of digital libraries has already evolved for about two decades. There are a few studies or concepts which shape my understanding of this topic so far. The following paragraphs are the summaries for them.

T. Saracevic (2000) suggests a conceptual framework for digital library evaluation which must involve selection and decisions related to:

1. Construct that was evaluated: What was evaluated? What was actually meant by a “digital library”? What elements (components, parts, processes, etc.) were involved in evaluation?
2. Context in which evaluation was conducted: selection of a goal, framework, viewpoint or level(s) of evaluation. What was the basic approach or perspective? What was the level of evaluation? What was the objective(s)?
3. Criteria that were chosen as a basis for evaluation: What parameters of performance were concentrate on? What dimension or characteristic were evaluate?
4. Measures reflecting selected criteria to record the performance. What specific measure(s) to use for a given criterion?
5. Methodologies that were used: for doing evaluation. What measures and measuring instruments were used? What samples? What procedures were used for data collection and for data analysis?
   

A clear specification on each of these is a requirement for any evaluation of digital libraries (Saracevic, 2000). My project will follow the lines of these five aspects as the conceptual framework for usability testing.

Judy H. Jeng’s dissertation (2006) is the recent effort on looking for a better way to evaluate digital library. She created an evaluation model and, in the end of the study, she generalizes the model, instruments, and methods for use in academic digital libraries. I expect my study can make further efforts based on her research and reexamine the validity of her model and generalization by applying it to the Carnegie Museum of Art Website. In addition, I also expect to find out new issues and variables not covered or confirmed by previous studies.

M.R. Patel (2003) examined the viability of a Category-Based Usability Theory, which indicates that usability of websites should be accounted for on the basis of the category the website is in. While Web design experts have provided general design guidelines, it is believed that with different site types, design guidelines may differ. In his study, Patel verified the hypothesis that “if the conceptual model of the user is to be observed then true usability must come from analyzing current practices of Web design of the most popular websites within a category or field, and not merely by following recommendations from Web usability experts.” My study will support this notion, and carefully decide the scope of generalization.

At last, one of the important relevant concepts is user-centered theory. Principally, user-centered theory argues for the user as an integral, participatory force in the process. Users are encouraged and invited to have a say, in other words, and thus they are physically or discursively present in the decision-making process... (Johnson, 1998, p. 30-32). My study support user-centered theory and adopt user-centered approach to carry out the testing.

A Workable Research Question
This research project is going to carry out a usability testing for Carnegie Museum of Art (CMOA) Website. Through the testing, the study could provide re-design recommendations to Carnegie Museum of Art Website in the end and re-examine the internal and external validity of usability evaluation model designed by J. Jeng in applying it to test CMOA Website. The study also interests the relationships among variables (effectiveness, efficiency, satisfaction and other unexpected variables) which influence user experiences.

Reference

1. Johnson, Robert R. (1998). User-Centered Technology, a Rhetorical Theory for Computers and Other Mundane Artifacts. Albany, NY: State University of New York Press.
2. Jeng, J. H. (2006). Usability of the digital library: An evaluation model. Unpublished doctoral dissertation. Rutgers, The State University of New Jersey, New Brunswick, New Jercey.
3. Saracevic, T. (2000). Digital library evaluation: Toward an evolution of concepts. Library Trends, 49 (2), 350-69.

Reading Response (Week 3)

The readings for this week contain many dimensions for evaluating qualitative research. What interests me the most is the issue of the generalizability (external validity) of qualitative research.

I found that there is one point these four articles all mentioned and all make efforts on emphasizing its importance, that is, the importance of clear and detailed description. As H & M motioned in the book that thick description are vital and it appears to the census shared by many qualitative researchers in the recent literature. Clear and detailed description serves as a means of allowing decisions about the extent to which findings from one study are applicable to other situations. From this view, it is important for examining or improving one’s generalizability of qualitative research.

The same view is also shared by Gary King in his book. He wrote in the paragraphs illustrating four characteristics of scientific research that scientific research uses explicit, codified, public methods to generate and analyze data whose reliability can therefore be assessed. Because all methods have limitations, explicitness is that those limitations can be understood and addressed. In addition, the methods can be taught and shared. This process allows research results to be compared across separate researchers and research projects to be replicated, and scholars to learn.

Through explicitness, thick decription, clear and detailed description, qualatative researchers are able to increase reliability and generalizability of their reseach and also be able to more or less compensate the weakness of qualitative research by its nature.

Reading Response (week 2): Silverman's Chapter 1 & 2

I am a slow reader but I am glad to find that Silverman's preface and chapter 1 & 2 are worth reading slowly and staying up this late. After the reading, I looked back to the usability evolution proposal I wrote for the last term's Introduction to Doctorial Study class. What a mess! Here I want to share with you how a novice researcher makes mistakes.

The following paragraph is what I wrote in the “Methods and Instruments” section in my proposal:

"In order to counterbalance the weakness of each method, this study combines several different methodological approaches to assess different aspects of the site with usability criteria. The main phase (phase three) of the project focus on the user-centered evaluations consisting of questionnaire, formal usability testing, card sorting and card description, as well as a combined label intuitiveness and category membership expectation test. Before carrying out the user-centered evaluations, focus group meetings and a pilot study will be conducted in the phase two.”

In this study, I desire to use multiple methods because I want to get many different aspects of the studied target (i.e. the studied website). Some methods gather only quantitative data (e.g. questionnaire) while others gather only qualitative data (e.g. focus group). And the formal usability testing even contains both the two kinds of data because it involves observation, open-ended questions for comments, and Likert scale for satisfaction ranking. Too many methods increase the difficulty and complication of data gathering and analysis. It means that the researcher will have to learn many more data analysis skills. This research design is time consuming and neglects the original purpose which is to give re-design suggestions in the phase of testing. It means the more efficient way is needed.

I think one of the solutions is to break down this study into several small narrow-downed studies. One aspect of the website usability testing is focused at one time instead of desiring to cover all aspects of evaluation.

"Ask the IRB" session

Before I went to the "Ask the IRB" session, I am not quite familiar with IRB or Human Subjects Review Board. Then I knew that IRB is established to improve human subjects protection. After the session, I did some study about the ethics of research. Here are some notes I think we, as researchers, should keep in mind when conducting any research.

• All researchers must be aware of and attend to the ethical considerations related to their studies. This need is important for all types and methods of research.
• Two of , perhaps the most basic and important, ethical issues in research are concerned with participants' informed consent and freedom from harm.
  
• Informed consent ensures that research participants enter the research of their free will and with understanding of the nature of the study and any possible dangers that may arise. Collecting information on participants or observing them without their knowledge or without appropriate permission is not ethical.
• Freedom from harm is focused on not exposing subjects (participants) to risks. It involves
  issues of confidentiality (for example, to protect participants from embarrassment or ridicule) and issues related to personal privacy.
  
• Access to data should be limited to persons directly involved in conducting the research. An individual participant's performance should not be reported or made public using the participant's name, even for an innocuous measure. Lack of privacy may lead to harm.
  
• The use of confidentiality or anonymity to avoid privacy invasion and potential harm is common. Anonymity means that the researcher does not know the identities of participants in the study. Confidentiality is that the researcher knows the identities of participants but promises not to release them to anyone else.
  
• Deception is another ethical dillemma. It is recommended not do the initial research studies using a topic that requires deception.

Ethical Issues in Qualitative Research
There are features of qualitative research that raise additional issues not typically encountered in quantitative research:

1. Qualitative research plans typically evolve and change as the researcher's immersion in and understanding of the research setting grows. In a real sense, the research plan is "in process" and only generally formed when presented to the IRB. As the plan evolves with added understanding of the context and participants, there is increased likelihood that unanticipated and unreviewed ethical issues will arise and need to be resolved on the spot.
   
2. Second, qualitative researchers typically are personally involved and engaged with the research context and its participants. The closeness between participants and researcher helps to provide deep and rich data, but may also create unconscious influences that raise issues for objectivity and data interpretation.

We might encounter ethical issues other than what I mentioned above in our research since there are many dimensions to the ethical conduct of research. However, I think the fundamental ethical rule is that participants should not be harmed in any way. Respect and concern for your own integrity and for your participants' dignity and welfare are the bottem lines of ethical research.

Reference:
Gay, L.R. & Airasian, P. (2003). Educational Research: Competencies for Analysis and Applications. New Jersey: Pearson Merrill Prentice Hall.

Getting Started!

Here is the first time for me to create Blog for academic purposes. I think it is a good way to share opinions, ideas, and thoughs outside the classroom. It also a good way to examine our own path of doing research project. I am expecting to see how much benefit we can get from blogging.