I. Study, Organizational, and Operational Structure

Webpage last modified: 2008-Oct-29

Introduction

The following guidelines outline the various organizational and operational aspects that should be considered in the structural design of a cross-national project or any project involving multiple languages or cultures. In particular, the focus is a cross-cultural survey that is implemented only once or the first wave of a longitudinal cross-cultural study, although most of the guidelines are also applicable to additional waves of a longitudinal study.

Cross-cultural studies are organized in many different ways, and each method has its advantages and disadvantages. Key determinants that will affect the structure of such studies include the source(s) and flow of funding, the availability of human and technical resources, both centrally and at local levels, the best way (mode) of contacting and interviewing respondents, and the research infrastructure. These factors may vary from country to country, and all will influence how the overall study is organized, designed, and later implemented. Although increased spending does not necessarily improve survey quality, there is a minimum cost involved in assuring quality. The inverse relationship between survey error and survey cost generally drives the target level of quality of the survey (final product).

One organizing structure that has been used often in large-scale, cross-cultural studies, and that will be referenced throughout these guidelines, is some form of a coordinating center. The coordinating center generally includes people from different countries, institutions, and affiliations, and assumes varying levels of organizational responsibility or guidance for some or all of the aspects of the study. Examples of different coordinating centers can be found in information on the websites of survey programs such as the International Social Survey Programme, European Social Survey, Afrobarometer, and World Mental Health Initiative.

Guidelines

Goal: To establish the study's overall structure and locus of control at all levels and across all aspects of the study's design and implementation, and to communicate this structure and the procedural consequences of the structure to each participating country and survey organization.

  1. Identify and specify the chosen organizational structure for a study at the supranational, national, and, as necessary, subnational level.
    Rationale
    Procedural steps
    Lessons learned
  2. Clearly define and then implement specifications for every phase of the survey, keeping the study's research goals in mind at all times.
    Rationale
    Procedural steps
    Lessons learned
  3. Define quality standards at all stages of the survey process [8] [9].
    Rationale
    Procedural steps
    Lessons learned
  4. Document every stage of the survey lifecycle.
    Rationale
    Procedural steps
    Lessons learned

Glossary

Adaptation
Changing existing materials (e.g., management plans, contracts, training manuals, questionnaires, etc.) by deliberately altering some content or design component to make the resulting materials more suitable for another sociocultural context or a particular population.
Coding
Translating nonnumeric data into numeric fields.
Confidentiality
Securing the identity of and any information provided by the respondent to ensure, to the greatest extent possible, that public identification of an individual participating in the study and/or his individual responses does not occur.
Coordinating center
A research center that facilitates and organizes cross-national research activities.
Coversheet
Electronic or printed materials associated with each case that identify information about the case (e.g., the sample address, the unique identification number associated with a case, and the interviewer to whom a case is assigned). The coversheet often also contains an introduction to the study, instructions on how to screen sample members and randomly select the respondent, and space to record the date, time, outcome and notes for every attempt.
Editing
Altering data recorded by the interviewer or respondent to improve the quality of the data (e.g., checking consistency, correcting mistakes, following up on suspicious values, deleting duplicates, etc.). Sometimes this term also includes coding and imputation, or the placement of a number into a field where data were missing.
Interviewer falsification
Intentionally departing from the designed interviewer guidelines that could result in the contamination of the data. Falsification includes: 1) Fabricating all or part of an interview—the recording of data that are not provided by a designated survey respondent, and reporting them as answers of that respondent; 2) Deliberately misreporting disposition codes and falsifying process data (e.g., the recording of a refusal case as ineligible for the sample; reporting a fictitious contact attempt); 3) Deliberately miscoding the answer to a question in order to avoid follow-up questions; 4) Deliberately interviewing a nonsampled person in order to reduce effort required to complete an interview; or intentionally misrepresenting the data collection process to the survey management.
Longitudinal study
A study where elements are repeatedly measured over time.
Mean Square error (MSE)
The total error of a survey statistic; specifically, the sum of the variance and the bias squared.
Mode
Method of data collection.
Non-interview
A sample element is selected, but an interview does not take place (for example, due to noncontact, refusal, or ineligibility).
Overrun
The exceeding of costs estimated in a contract.
Pretesting
A collection of techniques and activities that allow researchers to evaluate survey questions and/or survey procedures before data collection begins.
Quality
Achieving excellence for all components related to the data.
Quality assurance
Statement of confidence that quality requirements will be fulfilled.
Quality control
Process focused on fulfilling quality requirements.
Recontact
Having another staff member (often a supervisor) attempt to speak with the respondent after the interview is reported, in order to verify that the interview was completed according to the specified protocol.
Reinterview
The process or action of interviewing the same respondent twice to assess reliability (simple response variance).
Reluctance handling
Techniques that can reduce reluctance to participate in potential respondents, thereby increasing the overall response rate.
Respone rate
The number of completed interviews divided by the total estimated number of eligible sample persons.
Sample element
A selected unit of the target population that may be eligible or ineligible.
Survey error
The total error of a survey statistic; specifically, the sum of the variance and the bias squared.
Working Group
Experts working together to oversee the implementation of a particular aspect of the survey lifecycle (e.g., sampling, questionnaire design, training, quality control, etc.).

References

[1] Aitken, A., Hörngren, J., Jones, N., Lewis, D., & Zilhäo, M. J. (2003). Handbook on improving quality by analysis of process variables. Luxembourg: Eurostat.

[2] Biemer, P., & Lyberg, L. (2003). Introduction to survey quality. Hoboken, NJ: Wiley.

[3] Federal Committee on Statistical Methodology. (1983). Statistical policy working paper 9: Contracting for surveys. Washington, DC: Office of Management and Budget.

[4] Fitzgerald, R., & Jowell, R. Measurement equivalence in comparative surveys: The European Social Survey (ESS)—from design to implementation and beyond. Unpublished manuscript.

[5] International Organization for Standardization. (2005). Market, opinion and social research—terms, definitions and service requirements. No. ISO/DIS 20252. Geneva.

[6] Jowell, R. (1998). How comparative is comparative research? American Behavioral Scientist, 42(2), 168-177.

[7] Office of Management and Budget. (2006). Standards and guidelines for statistical surveys. Washington, DC: Office of Information and Regulatory Affairs, OMB. Retrieved May 27, 2008, from http://www.whitehouse.gov/omb/inforeg
/statpolicy/standards_stat_surveys.pdf

[8] United Nations. (2005). Household surveys in developing and transition countries. New York: United Nations, Department of Economic and Social Affairs.

[9] Worcester, R., Lagos, M., & Basañez, M. (2000). Cross-national studies: Lessons learned the hard way. Paper presented at the 2000 WAPOR/AAPOR Conference.

Further Reading

Heath, A., Fisher, S., & Smith, S. (2005). The globalization of public opinion research. Annual Review of Political Science, 8, 297-333.

Lynn, P. (2001). Developing quality standards for cross-national survey research: Five approaches (Working Paper No. 2001-21). Colchester: University of Essex.

© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to publish any of this material in any form.