• Home
• |
• Contact Us
• |
• User Guide

Data Collection Guidelines

XI. Data Collection

Webpage last modified: 2008-Sep-10

Introduction

Collecting comparable data in multiple nations and cultures is a highly complex task, in which one can expect to encounter a variety of languages and cultural contexts. Even in a single locale, the target population may not be one homogenous population but a collection of language and cultural groups. Some of the languages involved may not even have a standard written form. The study may need to take wide variations in respondent literacy into account. The geographic topography may be difficult (e.g., remote islands or mountainous regions). Weather and seasonal impediments (e.g., monsoons) may make the harmonization of fielding times across different locales impractical. Some populations may be inaccessible because of migration patterns or only accessible under special circumstances (e.g., miners in camps, or populations in which the men go on long hunting or fishing trips). Other individuals may have refugee or undocumented status. People living in shanty-type housing may not be included on a given sample frame. While homeless populations are often not included by definition, the number and definition of the "homeless" may differ considerably from location to location. Outside events such as natural disasters or political upheavals may also pose major challenges for data collection.

Countries also vary widely in both their survey research infrastructures and in their laws and unwritten rules and customs pertaining to data collection and data access. Certain modes of administration may be inappropriate or not feasible in some situations. In addition, the size and composition of nonresponse will likely vary due to differences in contactibility and cooperation. Some countries officially prohibit survey research (e.g., North Korea and Burma) or, to date, severely restrict data collection on some topics, or restrict publication of results (e.g., China and Iran) [19].

While a survey conducted in a single country might face one or more of the challenges mentioned above, the probability of encountering multiple hurdles is much higher in a large-scale, cross-national study. What is atypical in the one-country context often becomes the norm in cross-national contexts. Moreover, the assumed homogeneity and common ground that may, broadly speaking, hold for a one-country study contrasts with the obvious heterogeneity of populations, languages, and contexts encountered in multinational studies. Because of the heterogeneity of target populations in cross-cultural surveys, allowing some flexibility in data collection protocols can reduce costs and error.

These guidelines are intended to advise data collection decision-makers within each participating country. However, it should be noted that, in some cases, a coordinating center dictates data collection decisions across all countries involved. The European Social Survey, for example, mandates the mode in each country, while the ISSP allows a certain amount of flexibility. See Study, Organizational, and Operational Structure for more details.

Because difficulties in data collection can be extreme in majority countries, these guidelines heavily emphasize the challenges of data collection in such contexts.

Guidelines

Goal: To collect data which is comparable across survey locations while minimizing total survey error and survey costs.

1. Assess the feasibility of conducting the research in each target country and culture.
   Rationale

   Local knowledge can be critical to understanding cultural traditions and customs, possible limitations, and the feasibility of the research. Experienced researchers, interviewers, and key stakeholders familiar with the topic or the population under study can help assess concerns and suggest potential solutions.

   Procedural steps

   • Plan a feasibility study, gathering input from the coordinating center on topics to be included. These might include the survey topic, intended mode of administration, respondent burden (e.g., length of interview, complexity of topic), and proposed methods of dealing with nonresponse.
   • Gather information from people who are familiar with data collection in the area. If possible, conduct focus groups and one-on-one interviews with individuals within the contracted survey organization and others who have previously collected data within the country or location.
   • Solicit the help of local collaborators or researchers.
     • Local collaborators or researchers can conduct the feasibility studies.
       • Provide them with a detailed description of the protocol, including the proposed mode, nonresponse reduction techniques, timing, interviewer training, remuneration, monitoring, and the general framework for data collection.
       • Request feedback on all aspects of the proposed study.
     • Elicit information from these sources and any relevant administrative bodies on:
       • Population issues (e.g., local knowledge about the survey, family structure and household listing issues, literacy levels, and cultural norms).
       • Logistical issues (e.g., seasonal accessibility and dangerous areas).
       • Issues related to mode choice (see Guideline 2).
       • Human protection issues (e.g., legal and/or cultural permissions which may be necessary to conduct the study) (see Ethical Considerations in Surveys).
   Lessons learned

   • One-on-one interviews are useful because participants in focus groups may be unwilling to express objections to the data collection protocol in a group setting.
   • While outside input is often helpful, recognize that negative feedback may, in part, reflect uncertainty rather than concrete obstacles. Such feedback can, however, alert researchers to constraints that require attention. For example, in an early survey of mass media communication behavior in the Middle East, experts predicted that data collection would not be possible in Arab countries because, the experts believed, the populace would think that the interviewers were agents of the government. The experts also suggested that women could not be hired as interviewers and that it would be impossible to survey previously unsurveyed groups, such as the nomadic Bedouin tribes. The research team, however, was successful in their data collection efforts [3].
2. Select a mode of administration that is appropriate for the survey topic and feasible for the country or culture.
   Rationale

   Whether dictated by the coordinating center or left to individual survey organizations (see Study, Organizational, and Operational Structure), selecting the mode(s) in which the survey will be administered is a major design decision. It affects survey cost, survey error, instrument design, and field planning. There is no one "best" mode; rather, mode(s) should be chosen based on appropriate tradeoffs of cost and error. In an international setting, cultural norms, literacy levels, and logistics will further constrain mode selection.

   Surveys can be conducted in numerous ways: face-to-face (FTF), by telephone (either conducted by an interviewer or using Interactive Voice Response (IVR)), through the mail, or over the web. The survey instrument can be paper-and-pencil in format (PAPI) or computer assisted (CAI). It can be be interviewer-administered or self-administered. This guideline will focus on face-to-face, telephone, and mail modes. Little research has been conducted on IVR, web surveys, or other, newer modes in cross-cultural settings. In addition, we have no strong sense of their current viability in multiple contexts around the world. More methodological research is needed in this area.

   Procedural steps

   • Assess the advantages and disadvantages of each mode. The general considerations for mode choice are [2]:
     • Cost:
       • Generally, mail surveys are the least expensive to implement, while face-to-face surveys are the most costly.
     • Speed of completion:
       • Data collection can usually be completed in the shortest period of time with a telephone survey.
     • Complexity of concepts to be measured:
       • Face-to-face and mail surveys can include visual aids.
       • Interviewers in either telephone or face-to-face surveys can help respondents understand complex concepts.
     • Target population and sampling frame (see Sample Design):
       • Often the sampling frame(s) available will dictate the choice of mode. For example, if an up-to-date sampling frame with names and addresses is available, mail or face-to-face surveying may be feasible. If there are no pre-existing frames, the survey organization may have to construct an area-based frame.
       • If a large percentage of the population has telephone service, a telephone survey may be feasible.
     • Sample dispersion:
       • If the target population spans a wide geographic area, the cost of contacting respondents in person may be prohibitive; in this case, mail or telephone surveys are alternatives.
       • If the sample is widely dispersed and mail or telephone surveying are not viable options, consider clustering to reduce interviewer travel costs (see Sample Design).
     • Interest in the topic:
       • If respondents do not seem interested in the topic, telephone or face-to-face interviewers can explain the purpose of the study, answer questions, address concerns, and draw attention toward other desirable aspects of the survey (e.g., the survey's contribution to building knowledge in important areas).
     • Nonresponse:
       • Face-to-face surveys generally achieve higher response rates than telephone surveys and, especially, mail surveys.
     • Interviewers and sensitive topics:
       • Research indicates that respondents in interviewer-administered surveys, whether telephone or face-to-face, tend to underreport socially undesirable behaviors (e.g., drug use) and overreport socially desirable behaviors (e.g., religious service attendance).
     • Instrument:
       • Complex questionnaires with branching and fills are difficult for respondents to complete on their own. Interviewers may be trained to use questionnaires with branching and fills but may still make errors. Consider using an interviewer-administered mode and/or an automated instrument with programmed skip patterns.
   • Using the findings from the focus groups and one-on-one interviews as a guide (see Guideline 1), determine what particular challenges the specific culture or country presents, including:
     • Logistical issues
       • In some areas, telephone and/or mail service may be unreliable.
       • Limited access to telephone or internet means that these modes are not appropriate.
     • Technical capabilities and infrastructure
       • Some areas may lack electricity, posing a challenge for computer-assisted interviewing.
     • The characteristics of the target population
       • Written materials are difficult to use in countries or cultures with low literacy rates.
       • Interviewer matching or tailoring may be a challenge.
   • Consider using a combination of modes to solve design problems while keeping costs low [8].
     • Problems which may be addressed include reducing coverage bias by correcting the sampling frame, increasing response rates, and reducing social desirability bias.
     • Modes can be combined in any of several ways:
       • Multiple modes can be used concurrently within a culture or country. For example, households that can be easily reached by telephone could be surveyed on the phone while the remainder is interviewed face to face.
       • Different modes can be used concurrently in different cultures or countries. A specific mode may be optimal for one country or culture while a different mode is the best choice for another.
       • Different modes can be used sequentially. The survey can be administered primarily in one mode with additional modes offered as part of a nonresponse follow-up program. Similarly, the first wave of a panel survey could be conducted face to face, with subsequent waves switching to a less expensive mode.
     • It is crucial to understand the cost and error implications prior to implementing a mixed mode design.
     • Because the sources and level of error differ from one mode to another, using a combination of modes can affect data comparability.
     • Planning decisions regarding the standardization of survey specifications may dictate one mode for all populations.
   Lessons learned

   • Mode choice in cross-cultural surveys is driven by a combination of cost and country- or culture-specific considerations. For example, complete lists of telephone numbers are available for almost all residents of Sweden, making a telephone survey a potentially good choice in that country. In general, the response rate in a telephone study can be expected to exceed the response rate using a mail survey, and costs should be much lower than in a face-to-face survey. However, if the majority of the target population is not literate or a reliable telephone service is unavailable, face-to-face interviewing may be the only feasible mode [28].
   • While a mixed-mode design can reduce cost, it may also create a layer of complexity for coordinating centers. The Gallup World Poll implements a mixed mode design in which the telephone is used in countries where 80% or more of the target population is covered, and face-to-face in countries with lower telephone coverage. The costs of telephone surveys are much lower than face-to-face modes[2], so overall costs are reduced. Since breakoffs are less common in face-to-face surveys than telephone surveys, even when the questionnaire is much longer [2], additional questions were added to the face-to-face questionnaire. Gallup has therefore developed two different sampling protocols, questionnaire versions, and data collection procedures for face-to-face countries and for telephone countries [11].
   • In a cross-national context, the impact of mode can be confounded with cultural differences. For example, the ISSP originally planned to have all countries use self-completion methods. However, low literacy rates in some countries necessitate the use of interviewers in these locales [28]. Differences between countries on survey estimates may be substantive or may be a result of mode effects.
3. If face-to-face interviewing is selected, establish procedures for dealing with issues specific to this mode.
   Rationale

   Many cross-cultural projects attempt to keep the mode of administration constant by choosing face-to-face data collection, as it generally has the best sample coverage properties, highest response rates, and does not require sample persons to be literate. In order to collect comparable data, surveys that are conducted in multiple countries or cultures must establish a standard data collection protocol. At the same time, the protocol will sometimes need to allow for modifications required by local norms, conditions or customs.

   Telephone, mail, and even web modes may be used in cross-cultural surveys, although the implementation of face-to-face surveys presents a number of logistical challenges not faced in other modes.

   Procedural steps

   • Contact local authorities for clearance to collect data at the site; if necessary, negotiate with local authorities or militias to gain access to sample areas.
   • Take measures to ensure interviewer safety.
     • Inquire about potential safety problems, such as civil unrest and high crime areas.
     • Decide whether interviewers should be chaperoned.
     • Have interviewers visit their segments during the day of or before the first day of data collection. They should check for potential hazards and for safe havens during this visit.
     • Have interviewers tell their supervisors and family members when they plan to leave for the field, the approximate location of the segment, and when to expect them back.
   • Have interviewers carry the following items in the field to establish their legitimacy:
     • Official identification from the survey organization.
     • Official letters to local authorities describing the study.
   • Provide adequate transportation for staff and supplies.
     • If maps are unavailable or unreliable, consider the use of local guides or GPS instruments.
     • Arrange to secure fuel and oil and to maintain the vehicles used by the field staff; this may present logistical problems in some majority countries.
     • Arrange for emergency transportation in the event that a field team member becomes ill or injured and needs immediate medical attention.
     • Arrange for backup transportation.
     • Secure housing accommodations in more remote areas prior to fieldwork or have the team carry their own (e.g., tents or mobile homes).
   • Match interviewer and respondent characteristics (e.g., race, ethnicity, or gender) when cultural norms so dictate (see Interviewer Recruitment, Selection, and Training).
   • If physical measurements are taken as part of the survey,
     • Check the cultural acceptance of taking such measurements.
     • Calibrate the equipment regularly.
   • Provide all members of the field staff with access to a reliable line of communication with their supervisor. This will allow them to report progress and problems and to transmit the survey data as quickly as possible.
     • Majority countries may have weak communication capacities, especially in rural areas.
     • Cellular or satellite phones may be a worthwhile investment for teams in remote areas.
   • Aim to conduct the interview in a setting which affords visual, physical, and auditory privacy.
     • Privacy is critical for keeping respondents' answers to the survey questions confidential.
     • Although complete privacy is ideal, it is impossible to achieve in some cultures. Interviewers should attempt to keep the interview as private as possible, while still respecting cultural norms. This may involve self-administration on more sensitive questions or use of the randomized response technique [32]. Another alternative may be to keep any others present occupied while the targeted respondent completes the survey.
     • Customs may vary among countries. In some, it may be unacceptable to have any interviewer come to the respondent's home, or it may be unacceptable for an interviewer of opposite sex to the selected respondent or informant to enter the home.
     • Privacy increases the likelihood that respondents will answer honestly about sensitive behaviors, such as sexual practices or drug use. What is considered "sensitive" may vary among countries or cultures; administration practices may need to differ accordingly.
   Lessons learned

   • Because responses to some survey questions can be affected by other individuals present during data collection, it is optimal—but not always possible—to conduct face-to-face surveys in private. In a face-to-face fertility survey of women in what is now Bangladesh, privacy was difficult to establish; most interviews took place in the presence of the respondent's mother- or sister-in-law. This may have affected responses to sensitive questions [5].
   • Similarly, men in some parts of Africa were found to object to confidential interviews with their wives or children. The interviewers were instructed to conduct interviews in a place that was visible to the male heads of household but out of earshot. [4]
4. Establish a clear protocol for managing the survey sample.
   Rationale

   Nonresponse can be assessed and/or reduced with an effective sample management monitoring system. In addition, a good sample management system facilitates evaluating interviewer workload and performance.

   The study structure may specify what sample management systems are used. In cross-cultural surveys with strong centralized control, a single sample management system may be specified in the contract with local survey organizations. If an electronic system is used, coordinating centers may play a role in monitoring fieldwork. See Study Organizational and Operational Structure for details.

   Procedural steps

   • Use a coversheet or an electronic sample management system to track each sample element during the study (see Appendix C for an example of a paper coversheet).
     • Interviewers using paper coversheets have found that they work most efficiently if they sort the coversheets by (1) appointment time and/or (2) geographical location. The same sorting procedures should be available in electronic sample management systems.
   • Structure the field staff to aid them in working the sample efficiently.
     • Give supervisors the responsibility of assigning sample elements to interviewers and reassigning them when necessary.
     • Do not allow interviewers to switch sample elements among themselves without the explicit approval of the supervisor.
     • In a face-to-face study, ensure that sample elements are assigned in a way that minimizes travel costs.
     • In a face-to-face study, decide whether interviewers will work alone, in pairs, or in traveling teams (see Interviewer Recruitment, Selection, and Training).
     • Decide whether interviewers and respondents should be matched on some characteristic(s) (see Guideline 1).
       • If the respondents' characteristics are unknown prior to data collection, develop procedures to make on-the-spot matching possible. For example, to facilitate gender matching, send interviewers into the field in male-female pairs.
   • Train interviewers to use a random selection technique, such as a Kish selection table, for within-household respondent selection (see Appendix A for an example of a Kish table) [22].
   • Have interviewers complete a call record or contact form each time they attempt contact, whether or not the attempt is successful (see Appendix D for an example of a call record).
     • Use disposition codes to describe the outcome of each contact attempt.
     • Distinguish (1) completed interviews with eligible persons, (2) non-interviews (eligible persons), (3) non-interviews (unknown if eligible persons), and (4) non-interviews (ineligible persons).
   • Assign a final disposition code to each sample element in the gross sample at the end of data collection; include any new sample elements that may be created or generated during data collection (e.g., for additional family members or through half open intervals).
     • Provide a clear explanation and training to interviewers before they are allowed to assign final disposition codes.
     • Take into account that, in some agencies, only supervisors can assign final disposition codes.
   Lessons learned

   • An effective sample management system can clarify the causes of nonresponse. When the Amenities and Services Utilization Survey (AVO) was conducted in the Netherlands in 1995, interviewers were not asked to record detailed disposition codes for each call. As a result, refusals could not be distinguished from noncontacts. When the study was repeated in 1999, detailed disposition codes were collected. Researchers were then able to see that, after three unsuccessful contact attempts, refusal was the more probable explanation [29].
   • Not all survey organizations will be familiar with sample management practices. Allow some time in training for interviewers to become familiar with the sample management system (see Interviewer Recruitment, Selection, and Training).
5. Reduce nonresponse as much as possible.
   Rationale

   Increasing the response rate can improve the accuracy of probability statements about the population. Surveys follow an inferential paradigm that assumes a 100% response rate in a probability sample will provide unbiased estimates [13]. Although the nonresponse rate alone does not predict nonresponse bias [13], it can be a predictor of the potential for nonresponse bias. Furthermore, response rates have been dropping differentially across countries due to noncontact and, increasingly, reluctance to participate [9].

   Procedural steps

   • Depending upon cultural norms, gain the support of any "gatekeepers" (e.g., community leaders or elders) before attempting to reach individual households.
   • Publicize the survey locally to raise awareness and encourage cooperation.
     • If most of the population is literate, consider posting colorful, attractive leaflets. Simple presentations can also inform low literacy populations.
     • Also use word-of-mouth channels or local dignitaries (doctors, teachers) as appropriate.
   • Send pre-notification letters to sampled households if feasible.
     • The letter should (1) explain the purpose of the survey, (2) establish the legitimacy of the survey agency and the interviewer, (3) assure confidentiality of answers, (4) notify the household that participation is voluntary, and (5) provide contact information for the agency.
     • Be aware that survey sponsorship may affect both nonresponse and the accuracy of the actual measures. For example, some respondents may fear repercussions if they do not respond to a survey sponsored by a government agency. While this fear may dramatically increase response rates, the quality of the data may be dubious; respondents may feel that their responses are not genuinely confidential if the survey is sponsored by a government agency, and they may not respond openly. In addition, ethical issues arise in such situations (see Ethical Considerations in Surveys).
   • Attempt to contact the respondent.
     • Be aware that different modes face different obstacles.
       • Telephone surveys may encounter caller ID and/or answering machines; these devices may be used by potential respondents as screening mechanisms, thus preventing contact.
       • Face-to-face surveys must contend with general at-home patterns of sample persons, locked apartment buildings, gated communities, or long absences from home.
       • In mail surveys, it is difficult to disentangle the effects of noncontact, refusal, and a poor sampling frame. If a mail questionnaire is not returned, the respondent may not have picked up mail at that address, may have decided against participating, or may never have received the questionnaire because the address was incorrect.
     • In a face-to-face survey, train the interviewers to make observations of the housing unit to assess likely at-home patterns.
       • Some survey organizations allow interviewers to ask neighbors about targeted but not contacted respondents; others prohibit such questions because of confidentiality concerns.
     • In interviewer-administered surveys, train the interviewers to use a predefined grid showing different blocks of time across the week when the interviewer must attempt to contact respondents.
       • This practice increases the probability of reaching the respondent at home.
       • The times of day when persons are most likely to be at home vary by culture. For example, individuals in the United States are more likely to be reached on evenings and weekends [15].
       • Alternatively, and as possible, specify the minimum number of times that attempts must be made during daytime hours, during evening hours, and during the weekend. Incorporate culture-specific information about likely at-home patterns, such as normal workdays, normal work hours, and holidays.
     • For cost efficiency, specify the maximum number of contact attempts that should be made before the final disposition code is assigned.
   • Monitor response rates continuously, and produce reports of daily response rates in order to identify data collection procedures that are more or less successful at increasing participation.
   • If allowed by local custom, offer an incentive for participation [27].
     • Present the incentive as a "token of appreciation" for participating in the survey, not as payment for the response.
     • Make the token reasonable; it should not be so large that it might raise suspicion about the researcher's or agency's motives or be somehow coercive. It should be generally proportionate to the respondent burden.
     • Ideally, provide the incentive prior to the interview. Incentives promised upon the completion of the interview also increase participation, but to a lesser degree.
     • Document the use of incentives, including amount and type, time of implementation, and any special strategy, such as increasing the amount of the incentive in the final weeks of the study.
     • For financial incentives, interviewers may be asked to record that an incentive was given to a respondent; similarly, the respondent may need to sign to indicate receipt.
   • Train interviewers to use culturally appropriate reluctance aversion techniques (see Interviewer Recruitment, Selection, and Training).
     • Minimally, train interviewers how to answer anticipated respondent concerns [14].
     • Be aware that local customs and legal limitations may prohibit any attempt to recontact someone who has declined to participate in the survey.
   • Consider assigning supervisors or more experienced interviewers to difficult cases to increase contact and cooperation.
   • Consider switching modes to increase contact and cooperation (see Guideline 2).
     • Some studies in the United States employ a mixed mode design in which the least expensive mode is used initially, after which time progressively more expensive modes are implemented in order to reduce nonresponse.
     • Different modes may produce different survey estimates. These mode-specific differences in measurement might be acceptable to the investigator if nonresponse is reduced.
     • If more than one mode is expected to be used and budget permits, examine possible mode effects prior to the start of data collection.
       • Test for mode effects by administering key questions or questionnaire sections to a randomly split sample of respondents similar to the targeted population (e.g., asking the questions on the telephone for one group and in-person for another).
       • If it is not possible to test for potential mode effects beforehand, check for differences in responses at the end of data collection.
       • Ascertain whether respondents surveyed in each mode produce similar response distributions on key variables before combining their responses for analysis.
   • Minimize nonresponse bias as much as possible.
     • Nonresponse bias is a function of both the response rate and the difference between respondents and nonrespondents on a particular statistic [15]. Because nonresponse bias is statistic-specific, response rates alone do not indicate nonresponse bias.
     • Estimate the effect of nonresponse bias on key survey estimates, if possible (see Guideline 9).
     • If possible, use weighting and imputation [16] (see Data Processing and Statistical Adjustment).
   Lessons learned

   • While the literature has clearly established the positive effects of prepaid and cash incentives upon response in minority countries, it is possible that incentives may affect the propensity to respond differently in majority countries. For example, offering a choice of incentives may be more effective at increasing response rates than simply offering a prepaid incentive. Furthermore, in areas with rampant inflation, the value of cash incentives may decrease dramatically within a short period of time.
   • Response rates are not necessarily good indicators of nonresponse bias, but nevertheless tend to be used as a proxy for bias. In a health study of the elderly in Scotland, healthy individuals were more likely to participate than unhealthy individuals. Because of this difference between the respondents and nonrespondents, the estimate of health was biased even though response rates reached 82% overall [6].
   • The same incentive may affect response rates differently across countries or cultures. In the German General Social Survey (ALLBUS), the same incentive (€10) was offered to all respondents. The authors examined cooperation rates for Moroccan and Turkish immigrants. The authors found that the incentive affected cooperation differently by ethnicity and gender: cooperation rates increased as a result of the incentive for Moroccan women, but did not increase for Moroccan men, Turkish men, or Turkish women [31].
6. Time data collection activities appropriately.
   Rationale

   A specific survey estimate may determine the timing of data collection activities; for example, a survey about voting behavior will necessarily be timed to occur around an election. Data collection activities may be hampered by inappropriate timing. Face-to-face data collection, for example, may be impossible during a monsoon season, an earthquake or a regional conflict.

   Procedural steps

   • Based upon feasibility studies (see Guideline 1), evaluate environmental, political, and cultural considerations which might affect the timing of data collection. These could include:
     • Extreme weather patterns or natural disasters.
     • War, civil war, military rule, militia rule, or the possibility of hostage taking.
     • Religious and secular holidays or migratory patterns of nomadic people.
   • Establish a specific start and end date for data collection.
     • Because unexpected events can interfere with data collection activities, remain somewhat flexible to allow for unexpected events.
   Lessons learned

   • Coordination of data collection activities across countries or cultures can be difficult or impossible. The Afrobarometer measures public opinion in some sub-Saharan African countries. The coordinators for the Afrobarometer note that data collection is especially difficult during national election or referendum campaigns, rainy seasons, times of famine, and national or religious holidays. Since such events vary across countries and cultures, fieldwork activities are spread over a full year [21].
   • Timing of data collection activities may be related to the topic of the survey or statistics of interest. The Comparative Study of Election Systems (CSES), for example, studies elections around the world and therefore must time data collection activities according to local election cycles [20].
7. Institute and follow appropriate quality control measures.
   Rationale

   If errors are caught early, they can be corrected while the study is still in the field. Improvement made during data collection may introduce some measure of inconsistency in the data, however. This trade-off should be considered before any action is taken [17].

   Procedural steps

   • Evaluate the effectiveness of data collection protocols regularly. Include:
     • Sample management systems.
     • Contact protocols.
     • Reluctance aversion protocols.
   • Observe the interviewers throughout data collection [23]; monitor them more frequently early in the study, less frequently as the study continues.
   • Review a random sample of coversheets on an ongoing basis to ensure that correct eligibility and respondent selection procedures are being followed.
   • If the survey is being conducted in a centralized telephone facility, follow established monitoring procedures [7]:
     • Monitor in relatively short (e.g., one-hour) shifts; this is cost-effective and reduces supervisor fatigue.
     • Use probability sampling to ensure that the number of interviews monitored is proportional to the number of interviewers working each hour.
     • Monitor new interviewers at a higher rate than experienced interviewers.
     • Select from eligible cases in which the phone is still ringing so that the supervisor is not forced to wait for new interviews to begin in order to start monitoring.
   • If feasible, audio record face-to-face interviews for review.
     • Determine whether cultural norms permit taping.
     • Store any tapes safely and securely (see Ethical Considerations in Surveys).
   • If physical measurements are being taken, periodically re-test the interviewers on the use of the instruments that record the measurement values.
   • Provide interviewers with feedback, both individually and as a group [7] [23].
     • Provide immediate, individual feedback if there has been a critical error.
     • Provide routine, individual feedback for self-improvement.
     • Offer group feedback to focus efforts on improving the process.
     • Evaluate the following with respect to interviewers [2]:
       • Knowledge of the study objectives.
       • Administration of the survey introduction.
       • Administration of household enumeration and respondent selection procedures.
       • Reluctance aversion efforts.
       • Contact efforts.
       • Rapport with the respondent (e.g., having a professional, confident manner).
       • Standardized interviewing techniques (e.g., reading questions as worded, probing, and clarifying).
       • Data entry procedures.
       • Administrative tasks (e.g., submitting timesheets in a timely fashion).
       • Ability to meet production goals and maintain productivity.
       • Administration of specialized study-specific procedures (e.g., procedures for taking physical measurements and administering tests of physical performance or cognitive ability).
   • Whenever possible, recontact or reinterview approximately 10-15% of each interviewer's completed cases, selected at random [1] [26].
     • If recontacting the respondent, verify that the interview took place, inquire if the interviewer acted professionally, and ask factual questions (e.g., mode of data collection, interview length, incentive, household composition, and key survey topics) [1].
     • If reinterviewing the respondent, ask a sample of factual questions that do not have heavily skewed response distributions, were not skipped by many respondents, are scattered throughout the questionnaire, and have answers which are unlikely to have changed between the time of the interview and the verification check [10] [30].
     • Conduct reinterviews within a time period that is not so long that respondents will have forgotten about the survey or so short that respondents will remember all the details of the survey [10].
     • Make sure recontacts and/or reinterviews are made with the original respondent and that questions refer to the same time period as that asked about in the original interview [10].
   • For approximately 5% of each interviewer's finalized non-interviews, perform random checks with households to verify that ineligibility, refusal, or other status was correctly assigned. Checks may be done by telephone, in person, or by mail, as needed.
   • Monitor quality indicators consistently throughout the field period; use an electronic system or note them in a daily log book [30]. Include the following:
     • Distributions of key variables.
     • Hours per interview per interviewer, for the study as a whole, and by respondent groups of interest.
     • Number of respondents approached, interviews completed, incomplete interviews, and contact attempts.
     • Response, refusal, and non-contact rates [30].
     • Outcomes of all contacts and review of disposition code assignment.
   • Create statistical process control charts (SPCs) to provide timely information on key aspects of the data collection process [25].
     • Use the charts to detect observations that are not within pre-determined limits (often within one standard deviation of the mean).
       • A common use of SPCs in survey organizations is to assess nonresponse reduction methods over the field period. Using these charts, the impact of interviewer effort on response rates can be easily assessed.
       • See Figure 1 for an example of an SPC that assesses this kind of efficiency. This SPC is divided into phases of data collection; at each phase, the sample size is reduced because cases in the previous phase are contacted and either participate, refuse to participate (a "hard" refusal), are found ineligible, or are unable to participate. Shades of red, or points in which the word "Test" appears, indicate weeks outside the predetermined limits.
     • Give extreme observations additional attention and try to determine the root cause.
     • Refer to the charts when deciding whether to release additional samples, further monitor interviewers, and/or offer additional training sessions.
   • Set contact limitations, determining:
     • The point at which additional attempts to contact a sample element are inefficient.
     • Whether respondents cooperating after a certain number of contact attempts are significantly different from others on key indicators.
   • Identify potential interviewer falsification.
     • Implement silent monitoring in centralized facilities, use audio-recordings and recontacts in field studies, and analyze outliers in the data to detect falsification [1].
     • Check responses to stem questions for each interviewer. Questions that have a stem-branch structure—in which specific responses to "stem" questions require the interviewer to ask a number of "branch" questions—can be at increased risk for falsification. If a particular interviewer has recorded responses to stem questions that consistently preclude the interviewer from asking the branch questions, the interviewer may be falsifying data.
     • Examine paradata, such as keystroke data and time stamps, by interviewer to identify potential falsification.
     • If falsification of data is suspected, contact the respondents involved over the telephone [10]. If respondents cannot be reached via telephone, send out a brief mail questionnaire with a pre-paid return envelope [2].
     • If falsification of data is suspected, investigate the interviewer's other work and remove the interviewer from all data collection activities until the issues have been resolved [1].
     • If irregularities or falsified data are discovered, redo the interviewer's cases and delete all of his or her recorded data [1] [2].
   Lessons learned

   Process and progress indicators are often interdependent. Therefore, improving one process or progress indicator may negatively affect another. For example, the pursuit of higher response rates can actually increase nonresponse bias if the techniques used to obtain the higher response rates are more acceptable and effective in some cultures than in others [13] [18].

8. Document data collection activities.
   Rationale

   Documenting procedures is an essential part of the data collection process. Process documentation is essential for timely intervention. In addition, by understanding what was done in the field, the data are more easily interpreted and understood.

   Procedural steps

   • Document the following:
     • A summary of feedback from the feasibility studies.
     • The interview or data collection process.
     • A description of the mode(s) used.
     • A description of the mode-specific protocols.
     • A description of the sample management system.
     • A description of any paradata collected.
     • Special approaches to reduce nonresponse, including any incentives and nonresponse follow up.
     • Outcome rates by key respondent groups, including response, refusal, noncontact, and other nonresponse rates.
     • Structure of the field staff (e.g., size of interviewer groups and supervisor/interviewer ratio).
     • Timing of the fieldwork for each country or cultural group.
     • A description of quality control procedures and protocols, including:
       • Interviewer monitoring procedures.
       • Outcomes of interviewer monitoring, such as hours per interview and any falsification rates.
     • Any validation study descriptions and outcomes (see Guideline 9)
9. When possible, conduct validation studies to estimate bias.
   Rationale

   As noted in Guideline 5, response rates alone are not good indicators of nonresponse bias; understanding nonresponse bias and making subsequent post-survey adjustments require information about the nonrespondents. Similarly, measurement error bias can only be estimated when "true" values for survey variables are known or can be modeled (i.e., using latent class analysis). Validation studies can increase confidence in results, assist with post-survey adjustments (see Data Processing and Statistical Adjustment), and address potential criticisms of the study. However, while the interpretation of survey estimates can benefit greatly from validation studies, conducting them may be difficult and/or prohibitively expensive.

   Survey methodological experiments are designed up front and the outcomes are carefully documented. While these experiments may or may not directly benefit a given study, they are extremely important for the development and building of a body of knowledge in cross-national survey methodology, on which future studies will be able draw.

   Procedural steps

   • Collect data on nonrespondents, if possible, to estimate nonresponse bias [13].
     • One approach is to study sample elements that initially refused to be interviewed.
       • Draw a random sample of such initial nonrespondents and attempt to interview them under a modified design protocol (e.g., increased incentives or a shorter interview).
       • This approach assumes that people who were initially reluctant to participate are identical to nonrespondents on key variables; this may or may not be a valid assumption [24].
       • Document the data collection procedures, including the proportion of initial nonrespondents included in the validation study, mode of administration, and any additional incentive [17].
     • A second approach is to compare respondents and nonrespondents on statistics of interest using information contained in outside records (e.g., population register data).
       • Complete external records for all sample elements may be difficult to find, inaccurate, or outdated.
       • These benchmark data are rarely available for statistics of interest.
     • A third approach is to calculate response rates within subgroups (e.g., racial, ethnic, or gender groups).
       • This approach assumes that subgroup membership is related to the propensity to respond, and assumes that biases in demographic variables are informative of biases in substantive variables.
     • A fourth approach is to compare estimates to similar estimates generated from outside surveys.
       • While estimates similar to estimates from these benchmark surveys can increase credibility, the key survey variables may not exist in the benchmark survey. Furthermore, coverage, nonresponse, and measurement error differences in the benchmark survey are largely unknown.
     • A fifth approach is to examine the impact of post-survey adjustments on the estimates by comparing unadjusted and adjusted values.
       • The use of this approach strongly assumes that the models used to adjust for nonresponse fully capture the nonresponse mechanisms at work. While some amount of nonresponse bias may be controlled using these adjustments, they will rarely-if ever-fully control nonresponse bias.
       • See Data Processing and Statistical Adjustment for more information on post-survey adjustments for nonresponse.
   • Use methodological studies to assess measurement error.
     • One approach is to use laboratory techniques, such as cognitive interviews (see Pretesting), vignettes, response latency, and behavior coding, to assess potential measurement error.
       • This approach assumes that laboratory measurements are comparable with those obtained in the survey.
       • Many laboratory experiments in the United States do not use probability-based samples; therefore, errors detected in the self-selected laboratory sample may not be representative of errors in the target population.
     • Another approach is to check outside records for the true value, or a proxy of the true value, of the measure.
       • The researcher must have access to the outside records.
       • This approach assumes that the outside records are complete and error-free.
       • It may be difficult to match the respondent to the outside record.
       • Document record collection procedures, including a description of the records and their quality.
     • A third approach is to embed a randomized experiment within the survey to assess differences in survey estimates among different measurement conditions. In this situation, respondent should be randomly assigned to the experimental conditions (e.g., interview mode — see discussion in Guideline 5).
   • Consider using other methods of assessing measurement error.
     • Reinterview respondents. Reinterviews are especially useful in determining interviewer falsification [10], but may also help assess other forms of measurement error. Document all aspects of the reinterview procedure, including:
       • The respondents who were eligible for the reinterview component of this study (e.g., random 10% of respondents), as well as the total number of respondents selected and how many completed the reinterview.
       • The questionnaire used in the reinterview.
       • The mode of administration of the reinterview.
       • The interviewers who administered the reinterview (e.g., any project interviewing staff, specially designated interviewers, supervisory staff, clinicians, self administered, etc.).
       • The time interval between administration of the main interview and the reinterview (e.g., reinterviews were conducted 1-2 weeks after the main study interview).
     • Collect paradata that may be correlated with measurement error (e.g., number of keystrokes, length of interview).
   Lessons learned

   Supplemental studies can be difficult and expensive to implement, but they are useful for validating survey results. For example, a study of discharged patients at a French hospital found no difference in patient satisfaction ratings between early and late respondents. The authors interpreted this finding to indicate that there was little evidence of nonresponse bias in their estimates of patient satisfaction. However, it is unclear if the differences in estimates were due to nonresponse bias or to measurement error [12].

Return to top

Previous chapter | Next chapter | Home

© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to\n publish any of this material in any form.

View Chapters

• Introduction
• Study Structure
• Tenders, Bids, & Contracts
• Ethical Considerations
• Sample Design
• Questionnaire Design
• Translation
•      Translation Assessment
•      Language Harmonization
•      Translators
•      Translation Budgeting
•      Translation Scheduling
•      Translation Tools
• Adaptation
• Survey Instrument Design
• Pretesting
• Interviewer Recruitment & Training
• Data Collection
• Harmonization of Data
• Data Processing
• Dissemination
• Assessing Quality
• Glossary
• Bibliography