Cross-Cultural Survey Guidelines

This Chapter is Under Construction

V. Questionnaire Design

Webpage last modified: 2008-Jun-20

Introduction

These guidelines will present current practice and options for the design, realization, and testing of questions deliberately intended for implementation in multiple cultures and languages. The models and strategies discussed are drawn from a variety of disciplines, including quality of life research, educational testing, marketing research, and attitudinal survey research.

The aim is to present a clear but basic picture of options available in developing questions for comparative studies, the procedures involved in each, and advantages and disadvantages of different approaches. The emphasis is on assisting readers to make informed choices about developing questionnaires for their own studies, but these guidelines will also provide the basis for informed appraisal of existing questionnaires, whether for reasons of analysis or otherwise.

While comparative questionnaire design generally involves issues related to question translation, question adaptation, survey instrument design, and pretesting questions, these issues are addressed in separate guidelines (see Translation, Adaptation, Survey Instrument Design, and Pretesting).

Guidelines

Goal:To ensure the equivalence of survey questions across cultures and languages (to the extent possible) and to make it possible to assess equivalence of survey measures and results.

Ensure that questionnaire design adheres to basic questionnaire design guidelines for general survey research.
Rationale

There are key elements of questionnaire design that researchers need to know in order to embark on any question or questionnaire design project, and which are not comparative in focus. While those with extensive experience in designing questionnaires may not need to review topics and areas of research necessary for sound questionnaire design, others may find it useful. These are not meant to provide guidance on general design issues (see Study, Organizational, and Operational Structure).
Procedural steps
- Review survey methods literature and research on general questionnaire design, including:
  - Theories contributing to question design [Harkness 2007].
    
    Cognition and survey research, including question and response processes.
    
    Response styles and bias.
    
    Pragmatic aspects of the survey interview event.
  - The craft, technique, and art of asking questions (see Appendix) [Bradburn, Sudman & Wansink 2004] [Converse & Presser 1986] [Groves et al. 2004 chap. 7] [Fowler 1995], including:
    
    Factual questions.
    
    Behavioral questions.
    
    Attitudinal questions.
    
    Socio-demographic questions ("background" variables).
    
    Open questions.
    
    Closed questions.
    
    Scales.
    
    Visual stimuli.
  - Questionnaire design and mode (see Survey Instrument Design).
  - Questionnaire design and replication.

Select an approach to comparative question design based on the study design.

Rationale

Procedural steps

Establish a lead team or working group responsible for questionnaire design that is comprised of qualified participants from the countries and the coordinating center, including external experts as needed (see Study, Organizational, and Operational Structure).
Determine what options there are for developing questions appropriate to the study design within design constraints; the key approaches can be differentiated as follows [Harkness, van de Vijver, & Johnson 2003] [Harkness 2007]:
- Adopt questions, adapt questions, or write new questions.
- Ask the same questions (ASQ), as different questions (ADQ), or use a mix of methods.
Determine whether differences in cultures, regions, or languages require adaptation or translation of source questions (see Translation and Adaptation).

Select an approach to comparative questionnaire design based on options appropriate to the study design and their advantages and disadvantages (see table below):

Ask the same questions (ASQ).
Decentering.

Source questionnaire and translation.
Ask different questions (ADQ).

Mixed approaches--etic (common questions)-emic (cultural-specific questions) mixed approaches.
Other approaches.

Approach to Design	Advantages	Disadvantages
ASQ
Decentering Translate source
ADQ
Mixed etic-emic questions
Other approaches

Develop protocols for the implementation of a given approach for development of concrete questionnaires for multiple cultures and languages.
Rationale

There are many conceptual and technical challenges faced in actually realizing questionnaires in the comparative context. For example, there are constraints on realization imposed by such as the characteristic of a given languages system and associated typographical conventions, or the chosen mode of data collection.
Develop qualitative and quantitative protocols for assessing the quality of questions across survey implementations.
Procedural steps
- Qualitative methods of assessment:
  - Various pretesting techniques (focus groups, cognitive testing, pilot studies, behavior coding (see Pretesting).
  - Expert appraisals (by such groups as target population members, substantive experts, question design experts, translators).
  - Field debriefings (interviewer and respondent).
- Quantitative methods of assessment:
  - Exploratory analyses.
  - Multi-trait multi-method (MTMM).
  - Item-response theory (IRT).
  - Differential item functioning (dif).
  - Split-ballot testing.
  - Monolinqual / bilinqual testing.
- Combinations of qualitative and quantitative methods of assessment.
Fully document questionnaire design decisions and design implementation and quality assurance protocols.

Appendix [under construction]

[A high-level summary of basic guidelines for crafting survey questions, based on References provided; further reading will be recommended].

Glossary

Adaptation: The deliberate technical or substantive modification of questions to better fit a new sociocultural context or particular population.
ADQ (Ask different questions): An approach to question design where researchers collect data across populations or countries using the most salient-population-specific questions on a given topic that or demonstrated to tap a construct that is germane or shared across populations.
ASQ (Ask the same questions): An approach to question design where researchers collect data across populations/countries by first deciding on a common source questionnaire in one language and then producing whatever other language versions are needed on the basis of translation.
Attitudinal question: A question asking a respondent to evaluate a particular entity with some degree of favor or disfavor.
Behavioral question: A question asking a respondent to report actions or behaviors.
Closed-ended question: A survey question in which the respondent is presented with a set of response alternatives from which to choose an answer
Decentering: A classical model of comparative question design in which two different cultures are asked the same questions but the questions are developed simultaneously in each language. Thus, the process removes culture-specific elements from both versions. Decentering can thus be seen to stand between ADQ and ASQ models.
dif (differential item functioning): Item bias as a result of systematic differences in responses across cultures due to features of the item or measure itself, such as poor translation or ambiguous wording.
Emic (culture-specific) question: A question based on concepts or constructs that are culture-specific in constellation or significance and cannot be assumed to be shared across populations.
Etic (common) question: A question based on concepts or constructs that are universal and shared across cultures.
IRT (Item response theory): A theory that guides statistical techniques used to detect survey or test questions that have item bias or differential response functioning (see dif).
MTMM (Multi-trait multi-method): A structural equation modeling technique used for construct validation and testing the equivalence of measures in cross-cultural research (constructs are the "traits" and "methods" are the cultures).
Factual question: A question in which a true value exists for a particular respondent.
Open-ended question: A survey or interview format that allows respondents to answer questions as they choose. Unlike a closed question format, it does not provide a limited set of predefined answers.
Socio-demographic questions: Background questions about respondent characteristics such as age, marital status, employment status, and education.

References [incomplete]

Bradburn, N., Sudman, S., & Wansink, B. (2004). Asking questions: The definitive guide to questionnaire design—For market research, political polls, and social and health questionnaires. San Francisco: Jossey-Bass.

Converse, J. M., & Presser, S. (1986). Survey questions: Handcrafting the standardized questionnaire. Newbury Park, CA: Sage.

Fowler, F. J, Jr. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: Sage.

Groves, R. M., Fowler, F. D., Jr., Couper, M. P, Lepkowski, J. M., Singer, E., & Tourangeau, R. (Eds.) (2004). Questions and answers in surveys. In Survey methodology (chap. 7). New York: Wiley.

Groves, R. M., Fowler, F. D., Jr., Couper, M. P, Lepkowski, J. M., Singer, E., & Tourangeau, R. (Eds.) (2004). Evaluating survey questions. In Survey methodology (chap. 8). New York: Wiley.

Harkness, J. A. (2007). Comparative survey research: Goal and challenges. In E. D. de Leeuw, J. J. Hox, & D. A. Dillman (Eds.), International handbook of survey methodology. New York: L. Erlbaum Associates.

Harkness, J. A., van de Vijver, F. J. R., & Mohler, P. (2003). Cross-cultural survey methods. New York: Wiley.

Payne, S. L. (1980). The art of asking questions. (Original work published 1951). Princeton, NJ: Princeton University Press.

Return to top

Previous chapter | Next chapter | Home

Last modified: 2008-Jun-20

**This Chapter is Under Construction**