hopper, 1993 [2, abstract, overview, toc, switchboard, references]

2.5 Data Collection

Elites are considered to be the influential, the prominent, and the well-informed people in an organization or community. Elites are selected for interviews on the basis of their expertise in areas relevant to the research. Valuable information can be gained from these participants because of the positions they hold in social political, financial, or administrative realms. Elites can usually provide an overall view of an organization or its relationship to other organizations. Elites are also able to report on their organizations' policies, past histories, and future plans. However, in the course of the elite interview, considerable variation will occur in the degree of control, with the respondent occasionally assuming the questioner's role. (Marshall & Rossman, 1989, p. 94)

1. Explicit Purpose. The researcher should have a specific purpose for the interview and should make this clear to the informant so the person knows where the interview is going. While avoiding an authoritarian stance, the researcher should gradually take control and direct the talk into those channels that will discover the relevant knowledge of the participant.
 
2. Ethnographic Explanations. During the entire series of interviews the researcher must provide ethnographic explanations to the participant. These are concerned with explanations of what the project is all about, statements about why the researcher is writing things down or making a tape recording, encouragement to the informant to describe the participant's culture in his or her own terms, and explanations of the interview process and the reasons for asking various kinds of questions.
 
3. Ethnographic Questions. The questions asked are, of course, the reason for the interview. What is asked, of whom it is asked, and how topics are followed up determines the quality of the field notes and the ethnographic record that is subsequently written. (Spradley, 1979, p. 59)

Collecting data in their natural setting from carefully selected individuals was valuable for revealing complex interconnections in social relationships, and resulted in the discovery of nuances that were not apparent in any widely referenced publications.
 
The researcher's personal contact with key participants illuminated events and relationships that were critical during courseware development, but were not frequently or freely written about. Personal contact also often provided contacts to other participants and their viewpoints that would have not been available otherwise.
 

The following projects, organizations and participants were the focus of this study:   Project: ESCAPE (HyperCard and HyperNews) Organizations: Educational Research and Information Systems (ERIS, Purdue) Participants: Hopper, Lawler, LeBold, Putnam, Rehwinkel, Tillotson, Ward   Project: TODOR (BLOX) & Mechanics 2.01 (cT, Athena) Organizations: Athena and Academic Computing (AC, MIT) Participants: Bucciarelli, Daly, Jackson, Lavin, Schmidt   Project: Physical Geology Tutor (AthenaMuse) Organizations: Center for Educational Computing Initiatives (CECI, MIT) Participants: Davis, Kinnicutt, Lerman, Schlusselberg   Project: Context32 (Intermedia, StorySpace) Organizations: Institute for Research and Information Scholarship (IRIS, Brown) Participants: Kahn, Landow, Yankelovich   [See the Switchboard for further information.]

 
As in any form of research, the choice of methods also required some sacrifices as the price for important benefits. The types of interviewing techniques used in this study for data collection relied heavily upon cooperation of a small and select group of participants. Because the generalizability of this study to other efforts in the near future were a primary premise behind this research, it was critical to compensate for this weakness by including multiple perspectives from within three different organizations, and then triangulate between them during data analysis to determine which phenomena appeared most repeatedly and consistently across efforts. This type of effort is also referred to as " Situational Analysis." In this form analysis, a particular event or activity is studied from the viewpoint of all the participants. When all the views are pulled together, they provide a depth of perception that can contribute significantly to understanding (Borg and Gall et al, 1989, p. 403).
 
The variety of methods and data collection procedures in this study required the role of the researcher to vary throughout data collection and analysis. The researcher's roles ranged from actually being a participant during the ESCAPE project, to being an outsider during interviews for Context32, TODOR, ME201 and PGT. This range of roles proved to be both an advantage and a disadvantage. It is critical to the research that the researcher was a key person in the design of the ESCAPE project. While this allowed the researcher access to highly detailed knowledge about courseware development, it also maximized her observer bias. This background also resulted in confidence about courseware development issues, so that she found it less difficult to establish competence during interviews with developers. On the other hand, this perspective also increased her biases towards the perspective of some participants. She may have been particularly biased against those whose main concern dealt with the administration of hardware and operating systems, or faculty issues, due to a relative lack of experience in those areas.
 
Due to the qualitative nature of this study, not only did the role of the researcher change during the study, but so did her background. In fact, one critical reason for the selection of qualitative research was that the researcher would be the main instrument of data collection. Unlike paper and pencil formats, she was able to adapt to unexpected situations, and take advantage of the changing realities that were anticipated in the research sites. The changeable nature of the researcher as a research instrument is the greatest weakness of qualitative research studies, while remaining one of its greatest strengths.
 
In most qualitative studies, changes in the interviewer take place as increasingly expert views develop as an outcome of carefully arranged experiences. Change in the interviewer's perspective is the desirable outcome of the research process.
 
Ironically, a contradiction exists when the predominant method used to capture the results of qualitative research experiences is field notes. It is a well known phenomena that novice observers pick out different things than experts in the same situation, so researchers will inadequately capture, remember and thus interpret their earlier novice experiences relative to their later expert ones. Changes in what is observed, and how it is interpreted, results in a major threat to external validity called instrumentation, which refers to errors or biases that are introduced when the instruments used to collect research data are thought to remain constant, but in fact change.
 
Data collected through field notes early in a study can not be comparable to data collected later, because the researcher as the instrument used to gather data is unavoidably and even necessarily changed over the course of the study. In addition, it leads to the precious time spent early in a study to be relatively less effective. In order to compensate for this, technological means of capturing the situation were used during all interviews. This made it possible for the data gathered early in the study to be reviewed again in great detail later. The researcher was able to use periodic reviews of electronically captured data from early in the study to take advantage of her increasingly "expert" ability to interpret it.
 
Every interview was recorded with audio or video tape. After each interview, the exact contents of the interview were transcribed. Later in the study each transcription was reviewed again at least one more time for accuracy, and to allow the researcher to reexperience the interview from a more developed perspective. Once the tape was transcribed, copies of the original interview logs were created, and divided into segments based on topics of conversations. Each segment was edited to remove extraneous, repetitive and unclear conversation (roughly 20-50% edited out).
 
A continuous process of literature review was also essential to continue apace of emergent themes, as well as to keep up with numerous references provided by participants. Before and after the researcher interviewed participants in each of the three settings, numerous published and unpublished documents were often obtained. An elaborated literature review was also made near the end of the study, in order to focus on the key themes that emerged based on cumulative experience. The accumulated documents were used extensively to conduct "Historical Analysis" (Borg and Gall et al, 1989, p. 403), which allowed triangulation between a larger number of accounts than were available through interviews alone. In addition to this value as sources of triangulation opportunities, documents were an efficient form of supplementary data because their inherent and overt structure made then easy and efficient to manage than the audio and video data. They were also much easier to access for immediate follow-up data collection for clarification and omissions, which was important in this study to help compensate for the limited access to the sources of information afforded by the "elite interview" form of data collection.
 
The "vignettes" or passages from interviews were combined with selected passages from the project director's and producer's other published reports and publications to result in the total data distribution represented in the following table.
 

 

Note: In the cumulative data distribution, publications appear to have been the primary source of data for this study. This is a misleading impression resulting from the criteria used for final passage selection to include in published reports. When key points were emphasized by participants during interviews that were also available in published sources, the data used were drawn from the published sources to avoid unnecessary imposition on participants for the time consuming review process needed before passages from interviews were published. For this reason, while much of the data did come from published sources, the relative emphasis on key points was determined through interactions with participants during interviews. In some cases, data collection efforts included searching for relatively obscure published passages to illustrate extremely important points highlighted in interviews. While the passages finally used in this report came from previously published sources, the emphasis on them was determined through experiences and insights gained from personal interviews.

© Mary E. Hopper | MEHopper@TheWorld.com [posted 12/04/93 | revised 04/12/13]