• Home
• |
• Contact Us
• |
• User Guide

Survey Instrument Design Guidelines

VIII. Survey Instrument Design

Webpage last modified: 2008-Oct-16

Introduction

The design and implementation of a given survey instrument can be viewed separately from questionnaire design per se. Survey instrument design focuses less on questionnaire content and much more on the design of the actual survey instrument that delivers the questionnaire content. In this sense, design includes the format, layout, and other visual aspects of the presentation or context of survey questions, such as how prior and following questions appear related (or not) to specific questions. In some instances, questionnaire design and instrument design overlap. Mode decisions, for example, shape the technical format of questions as well as their wording.

These guidelines will use the more general terms "survey instrument" or "instrument" when describing procedures or features that apply to both paper and computer-assisted or Web surveys, and the term "application" — which suggests the need for at least some programming — when discussing procedures for development of computer-assisted or Web survey instruments.

Study design decisions include whether the survey is self-administered or interviewer-administered and whether it is computer-assisted or administered on the Web (see Study, Organizational, and Operational Structure and Data Collection). It involves decisions about data output, coding, and data documentation (see Data Processing and Statistical Adjustment and Dissemination of Survey and Statistical Data). If the survey is computer-assisted or implemented on the Web, it also involves decisions about programming and the user interface (what respondents or interviewers see on the computer screen and how the computer interacts with them). All of these decisions may have an impact on instrument design, which affects survey implementation primarily in three ways:

1. How easy it is for an interviewer or a respondent to use the survey instrument and to provide responses (the "usability" of the instrument).
2. How easy it is to program a computer-assisted or Web survey and to test it.
3. How easy it is to code, output, analyze, and document survey data.

Instrument design may lead to measurement error, including error resulting from context effects. In the case of cross-cultural survey research, error related to instrument design can stem from problems in each implementation in a target culture or language. Poor design of survey instruments may also increase nonresponse error at the levels of both the sample element (unit nonresponse) and the survey question (item nonresponse).

Guidelines

Goal: To maximize the usability of cross-cultural survey instruments, to minimize measurement error and nonresponse error due to survey instrument design, and to ensure good documentation of survey instruments and data.

1. Ensure that instrument design is appropriate to the method of administration and the target population.
   Rationale

   Design requirements for self-administered surveys differ from design requirements for interviewer-administered surveys. For example, self-administered surveys have no interviewer to help repair misunderstandings. There is also limited opportunity to "train" respondents on how to respond to a self-administered survey. Computer-assisted instruments, which involve human-computer interaction call for their own additional design features. Target population characteristics (household or establishment, age, education, literacy, computer literacy, etc.) may also influence instrument design decisions. For example, self-administered surveys are best administered to populations likely to have high literacy rates; computer-assisted surveys are best administered to target populations with an expected high familiarity with computers, or in situations in which data collection can be facilitated by interviewers, teachers, or other aides.

   Procedural steps

   • Determine whether to develop an interviewer- or self-administered instrument and whether to use a paper or computer-assisted instrument. Here are some points to consider:
     • Self-administration may lead to better data quality for surveys with sensitve questions; self-administered components may be combined with interviewer assisted components of surveys.
     • There may be constraints in some contexts that make it difficult to collect data with computer-assisted or Web survey instruments. For example, countries with low Internet penetration cannot be included in a general population Web survey.
     • Computer-assistance may lead to higher data quality in long and complex surveys or those with embedded experiments.
   • Determine the appropriate instrument design for the method of administration (see Data Collection):
     • It is important that interviewer-administered instruments make it easy to perform required tasks in the order in which they are expected to be performed. For example, interviewer tasks such as referring to show cards or other aids, reading questions, providing definitions, probing responses, and recording responses should be displayed in the order of their likely occurrence.
     • Similarly, it is important that self-adminstered instruments make it easy for respondents to distinguish instructions (such as "Select one"), read questions, navigate correctly through the instrument, and enter responses [9]. For example, instructions should appear where they are needed, such as "Start here" before the first question, and response entry instructions after the question text (e.g., "Tick all that apply").
     • Whether interviewer- or self-administered, design should help to minimize burden placed on interviewers and/or respondents, which increases as instruments increase in length and complexity.
   • Determine whether there are additional design considerations related to characteristics of the target population, such as for children, the elderly, or the visually or hearing impaired [8].
   • Ensure that all such considerations are reflected in the instrument design specifications (see Guideline 2).
   Lessons learned

   • DeLeeuw, Hox, and Kef [8] describe the results from a number of Dutch surveys of special populations using computer-assisted interviewer and self-administered components, in which instrument design and administration were tailored to target population needs. They concluded that well-designed computer-assisted instruments both improve the quality of data and minimize the burden experienced by interviewers and respondents.
2. Develop complete instrument design specifications for the survey instrument, specifying culture-specific guidelines as necessary.
   Rationale

   Instrument design specifications guide formatting and/or programming of the survey instrument or application; they ensure design consistency across culture-specific instruments (to the extent possible) and facilitate post-production data processing, harmonization, documentation, and analysis. Specifications should clearly outline the questionnaire and its contents, provide guidelines for formatting the survey instrument, and suggest appropriate adaptation of an instrument design for other cultures and languages. Note that similar guidelines are necessary for a data entry application (see Data Processing and Statistical Adjustment). Generally this guideline is relevant to formatting of elements in either paper or computer-assisted instruments, although a few may relate to only one or the other (e.g., paper size). Guideline 3 adds guidelines that are relevant to computer-assisted applications and their interface designs.

   Procedural steps

   • Provide application formatting guidelines that facilitate the translation of languages (see Translation), specifying scripts, character sets, fonts, spacing, and so on, for target languages [1] [2] [14] [17]; formatting guidelines should address aspects of design such as:
     • Language- and region-specific characters sets.
       • The International Organization for Standardization (ISO) 8859 Character Set has language specific groupings, for example, ISO 8859-1 for Western Europe and ISO 8859-2 for Central and Eastern Europe.
     • Differences in languages and scripts; for example:
       • Japan has one language, but several scripts, which can be mixed.
       • China has one official language, Mandarin, but many regional dialects, and two main scripts, Simplified Chinese (mainland China) and Traditional Chinese (Taiwan).
     • Differences in fonts that support different character sets; in general:
       • Avoid complex or ornate fonts.
       • Provide interline space to ensure clear separation between lines and to accommodate underlining.
       • Provide space to accommodate changes in line heights.
       • Provide a source application design that can be adapted to translated instruments with differing text directionality; the three types of text directionality are: (1) Left-to-right (Latin, Cyrillic, Greek, Thai, and Indic languages); (2) Left-to-right and vertical (Chinese, Japanese, and Korean); (3) Bi-directional (Hebrew and Arabic; characters displayed right to left, with Latin characters displayed left to right).
       • Provide flexibility in layout of the instrument to accommodate expansion or contraction of text during translation. For example, use a larger font and/or margins for an English instrument, if translating from English into other languages would increase the amount of space required for text in culture-specific instruments.
     • Differences across languages in punctuation (e.g., the different question marks in English and Arabic, ? and ؟, respectively)
     • Language- or culture-specific differences in the ways characters are sorted alphabetically, including diacritics (accent marks above or below letters, e.g., é), ligatures (multiple letters treated as single typographical units, e.g., æ, œ, and ß), character combinations (e.g., ch follows h in Czech), and uppercase and lowercase letters. For instance, the ligature Æ sorts after Z in Swedish, but after A in German.
     • Differences in paper sizes.
       • Provide guidance on pagination in order to avoid inadvertent context effects (for example, two related questions appearing on one page in one survey and on separate pages in another).
     • Differences in formatting of data objects, for display of information and recording of responses, including:
       • Date and time.
       • Calendar, holidays, and start of week.
       • Numeric formatting, e.g., thousands and decimal separators.
       • Names and addresses.
       • Telephone numbers.
       • Currency and monetary values.
       • Sizes and measurement (e.g., metric versus imperial units, temperature, clothing sizes, etc.).
   • Provide guidelines for both paper and computer-assisted surveys for the consistent formatting of specific text elements, such as question text, respondent or interviewer instructions, and so on. These might include, for example:
     • Display question text more prominently than response options.
     • Distinguish interviewer or respondent instructions, for example, in a smaller font of a different color, or italicized in parentheses.
     • Place text elements where and in the order they are needed based on interviewer or respondent task demands; for example, a show card instruction precedes question text and a probe instruction follows it.
     • Evenly space response options in a scale, grid, or table, so that they appear of equal weight or prominence.
   • Provide guidelines for the formatting of specific question types, and examples for each type in the application; these may include:
     • Enumerated response options.
     • Tick [Check or Select] all that apply.
     • Short or fixed-length text.
     • Open-ended text.
     • Numeric response.
       • Response entry masks (e.g., ____/__/__ for YYYY/MM/DD).
     • Multi-part questions and question series; for example:
       • Year / Month / Day.
       • Address / contact information.
       • Demographics question sets.
       • Amount-per-unit [e.g., income per day / week / month / year].
     • Randomized questions, response options, or sections.
     • Answer scales.
       • Fully-labeled scale.
       • Partially-labeled scale.
       • Table, grid, or matrix.
     • Text fills (variable question text).
     • Context indicators, that is, visual indicators that help respondents or interviewers understand where they are in an instrument (for example, current household member, vehicle, or source of income).
     • Progress indicator (for example, highlighting on a progress bar the current section or module in a Web or computer-assisted survey).
     • Question-level help.
     • Validation or consistency checks and post-collection edits. In computer-assisted surveys with programmed consistency checks, there is a distinction between:
       • Hard consistency check (interviewer or respondent cannot continue until the check is resolved).
       • Soft consistency check (interviewer or respondent may continue without resolving the check).
   • Provide examples of key question types and elements for all target languages and cultures (see Appendix A).
   • Add information to the instrument specifications that facilitates recording responses and the creation of a data dictionary and development of a codebook (see Data Processing and Statistical Adjustment and Dissemination of Survey and Statistical Data; see also Appendix B); for example, specify:
     • Variable names and labels.
     • Open question formats; consider space provided, which may need to differ across languages.
     • Pre-coded response options. If necessary, specify international standards for codes and classifications, such as occupation, language, country of origin, and religion (for example, specifications for the European Social Survey states that codes for respondent's language(s) specifies that the respondent's industry codes are based on the ISO-639-2 code frame).
     • Code number conventions (e.g., Yes=1, No=5 or No=2). Note that code numbers generally are not shown in self-administered questionnaires); in the example given, No=5 is sometimes used to minimize error in interviewer-administered surveys, because 5 (No) is farther away from 1 (Yes) on an interviewer or coder computer keyboard.
     • Missing data values, e.g.,
       • Not applicable.
       • Refusal.
       • Don't know.
       • No answer (other).
       Note that possible code values may be constrained by computer-assisted interviewing and coding software, or the statistical software used create survey datasets. Specifications should indicate the values required in the final datasets and in final data documentation.
     • Data input formats, including scales that use metaphors (such as ladders or thermometers).
     • Question-by-question objectives (for interviewer-administered surveys).
     • Interviewer or respondent instructions.
       • Respondent show card instructions.
       • Skip instructions.
       • Response format or data entry instructions.
     • Universe statements (e.g., "Universe [for this question]: Women ≥ 45").
     • Variables to construct or recode during postproduction.
   • Provide guidelines for the use of color, graphics, images, maps, and icons;
     • Ensure that colors used in application design do not have any negative connotations in specific cultures. Color has different meaning across cultures and research has found there are cultural differences in color preferences. Any choice of colors should be validated by experts on particular cultures [2] [15]. This may involve harmonization to a set of "culture-neutral" colors across instruments, or adaptation of some colors across instruments as necessary.
     • Ensure that any maps used in surveys are drawn to scale.
     • Ensure that images are displayed using comparable typographical units across survey implementations.
     • Ensure that graphics, images, and icons convey comparable meaning across cultures and do not have negative connotations in specific culture, or adapt them as necessary.
   • If using multiple data collection methods, include specifications for how question formats would differ across methods (e.g., interviewer-administered with a self-administered component, Web and mail, or in-person or telephone). For instance:
     • A computer-assisted self-interviewing screen might have only one question and input field per screen (to minimize respondent burden), whereas an interviewer-administered computer-assisted screen might have multiple questions and multiple input fields.
     • Self-administered instruments may be developed without response codes (the respondent clicks on a response option, checks a box, or clicks on a radio button), whereas many interviewer-assisted computer-assisted instruments require numbered response options for entry of responses, if numbers are the only possible form of input.
     • Software constraints may also necessitate alternate specifications, for example, if different software were used for Web and computer-assisted telephone interviewing components.
   • Based on the guidelines specified above, as well as the interface design guidelines that follow, prepare a survey instrument specification with all survey content for the instrument as well as a data dictionary, which represents the contents of the survey dataset.
     • At the beginning of the instrument specifications, provide an overview of the survey instrument, including the order of core modules and required placement of culture-specific modules (see an example in Appendix B).
   Lessons learned

   • Seemingly small differences in instrument design across culture-specific surveys can influence responses across cultures. For example, scales that are not formatted consistently, response options with misaligned check boxes, differences in the relative amount of space allowed for open responses, and differences in the physical placement of follow-up questions have been shown to lead to missing data or unusual response distributions across surveys [18].
   • In the International Social Survey Program (ISSP) one country lost the last item in a scale when copying the scale from A4 size paper to 8 ½ by 11 inch paper [19].
3. Develop interface design guidelines for computer-assisted and Web survey applications.
   Rationale

   Interface design has an impact on the respondent-computer or interviewer-computer interaction, influences user performance, and may affect data quality. Design should not only maximize usability, but should also be consistent across survey implementations. Therefore, it is important to provide clear guidelines for design of questions, error messages and screen elements for computer-assisted and Web surveys (see Appendix A for an example of basic design guidelines). Note that similar guidelines are necessary for a data entry application (see Data Processing and Statistical Adjustment).

   Procedural steps

   • Establish the key principles for interface design, which should lead to effective quality assessment of instrument screen design (see Guideline 4). These include:
     • Consistency.
     • Visual discrimination among screen elements, so that interviewers and respondents quickly learn where different components are located and thus where on the screen to look for what type of element (e.g., interviewer probe instruction).
     • Adherence to a culture's normal reading behavior for each language and script, based on issues such as text directionality (see Guideline 2).
     • Display instructions at points appropriate to associated tasks.
     • Elimination of clutter and unnecessary information or other display features that distract interviewers and respondents.
   • Provide guidelines for the layout and formatting of screen elements, including:
     • Question text, which should be the primary focus of the screen.
     • Response options that convey that either only a single mutually-exclusive response (e.g., radio buttons if computer-assisted or Web) or that multiple responses are possible (e.g., check boxes if computer-assisted or Web).
     • Response input fields, which should convey length of response expected. For example:
       • An open-end response entry field provides a field as wide and with as many lines as the expected length of response.
       • The width of an integer field is as many number character lengths wide as the expected input, that is, one character length for a one-digit integer, two-character lengths for a two-digit integer, etc.
     • Instructions, which should appear as expected in relation to task demands; for example, a reference to a respondent booklet or show card should appear before question text, and a probe or data entry instruction after question text.
     • Online help or assistance, including format of help text and design of navigational aids that facilitate opening and closing help text windows.
     • Error messages, warnings, and consistency checks; these clearly identify the nature of the problem, reflect actual question wording if necessary, and convey how to resolve the problem (see [16] for an example for more detailed guidelines on design of error messages).
     • Context markers (for example, instrument section labels, household member, and so on).
     • Additional information may be required for Web self-administered surveys, such as contact information and graphic and/or text identification of the sponsoring organization.
   • Provide examples of screens with correct formatting of elements, for all question types (see Guideline 1) and all languages and cultures (see Appendix A).
   Lessons learned

   • There is increasing evidence [5] [6] [7] [9] that the visual design of computer-assisted and Web instruments can impact data quality. For example, providing an input box or field that allows entry of 10 characters with no guidance on input format can lead to poorer data quality than if the survey question more precisely calls for an integer of up to three digits; for example, instead of "20," "90" or "100" in an entry field with a width of three (___), a Web survey respondent enters "40 to 50" in a field with a width of 10 (_________).
4. Establish procedures for quality assurance of the survey instrument that ensures consistency of design, adapting evaluation methods to specific cultures as necessary.
   Rationale

   As discussed in Guideline 3, research shows that instrument design can affect data quality positively or negatively. Thus, it is important that pretesting (see Pretesting) of comparative survey instruments include procedures for assessing the quality of the design of the survey instrument and adaptations for specific cultures and languages, not just the quality of the content. This includes the evaluation of the use of color, graphics, images, maps, and icons. As indicated earlier, such evaluation procedures may require adaptation across cultures.

   Procedural steps

   • Provide a clear set of instrument specifications and/or data dictionary for the instrument and culture-specific adaptations, which will facilitate testing and assessment of the instruments.
   • Identify members of the cross-cultural questionnaire development, adaptation, and translation teams (see Questionnaire Design, Translation, and Adaptation) with expertise that enables assessment of the design of survey instruments. Such experts may include substantive experts, survey methodologists, linguists, and usability professionals, and should include someone with an understanding of response styles across cultures. This may require recruiting additional team members with knowledge of specific cultures.
   • Identify appropriate instrument evaluation procedures for the comparative surveys under evaluation. These may be more or less extensive based on whether survey organizations in the targeted cultures previously have used specific guidelines, instruments, and survey software. Most questionnaire pretesting tools (see Pretesting) may be used to evaluate instrument design as well as questionnaire content and data collection procedures. These include:
     • Expert review or heuristic evaluation, in which one or more experts evaluates the instrument design against a set of evaluation criteria or heuristics, for example:
       • Consistency and adherence to design guidelines.
       • Error prevention.
       • Usefulness of documentation, definitions, help, error messages, and other feedback to users.
       • Ease of navigation.
       • Ease of recognition of specific question or instrument elements and actions required.
     • Review of an instrument, data dictionary, or codebook to ensure adherence to instrument specifications for naming and labeling of variables and response options. This should include comparison across instruments or data dictionaries for all survey implementations.
     • Laboratory or in situ tests of instrument design with users or participants with similar characteristics to target interviewers or respondents. These are called usability tests when evaluating computer-based instruments, but they also may be used to evaluate paper instruments. Since culture-specific response styles affect how participants respond to questions about usability [3], every effort should be made to match tester and participant characteristics, language, and cultural background.
   • Collect metrics or measures from all instrument evaluation procedures that will lead to informed decisions about question- or screen-specific or global design changes that need to be made (see Pretesting). Examples include:
     • Questionnaire length and section and item timings.
     • Audit trails for computer-assisted or Web applications, which can include item time stamps, keystrokes, mouse actions, and functions invoked. Gathering some of these requires programming that captures information directly from the respondent's computer (Heerwegh [12] provides sample programming code on the Internet.)
     • Behavior codes or event codes based on video or audio recordings that reflect problems using the survey instrument. Such methods are appropriate for both paper and computer-assisted instruments.
     • Qualitative anlaysis of cognitive and usability testing.
     • Heuristic evaluation or expert review.
   Lessons learned

   • Research has shown [4] [11] that techniques for evaluating the effectiveness of paper materials and computer software work very well in the evaluation of the design of survey instruments. For example, usability evaluation methods (commonly used in the development of software to assess the quality of user interfaces) and traditional pretesting methods such as conventional pretests, cognitive interviews, and behavior coding can be used to identify instrument design problems as well as problems related to question content.
5. Provide complete documentation of guidelines for development of source and target language or culture-specific instruments.
   Rationale

   Comprehensive documentation of survey instruments or applications is an essential component of study documentation and comes into play at all stages of the survey lifecycle (questionnaire development, pretesting, data collection, post processing, and data dissemination and analysis). Complete and consistent guidelines for specifying and designing instruments leads to survey data that meet the quality requirements of users—statistical agencies throughout the world define quality (see Assessing Quality for Cross-Cultural Surveys) in terms of characteristics of comparative survey data and documentation, which include aspects such as relevance, accuracy, timeliness, accessibility, intrepretability, coherence, comparability, professionalism, and cost and burden (see, for example [10]). Documentation of instrument design specifications plays a significant role in ensuring data quality in these respects. The rapid increase in computer-assisted data collection methods makes it increasingly possible to provide well-documented survey data.

   Procedural steps

   • Provide documentation of the guidelines used for survey instrument design.
   • Provide documentation of quality assessments of the survey instruments, and the outcomes of decisions made to revise the instrument design.
   • Provide specifications for the final source instrument, based on Guideline 1, Guideline 2, and Guideline 3. This should include the instrument specifications and a data dictionary.
   • Provide alternative specifications for target languages or cultures as necessary. For example, if the source instrument is computer-assisted, but it is necessary to develop a paper instrument for one or more locations, separate specifications should be developed for these paper instruments.
   • Provide paper and/or electronic copies of all culture-specific instruments, to facilitate comparison of instrument design across implementations.
   • Provide question-level metadata (question text, response options, instructions, text fills, population universes, definitions, etc.) in an electronic format to facilitate linking and comparing metadata for all survey instruments (e.g., eXtended Markup Language data files). Some computer-assisted data collection software now makes this possible.
   Lessons learned

   • Survey instrument design and documentation of design guidelines and specifications can affect the quality of data produced and disseminated, and the ability of users to effectively analyze survey data. Hert [13] conducted studies of users "interacting" with statistical data in order to understand how to better meet their needs. In one study she found that the completeness and quality of available question-level survey instrument documentation and metadata affected users' selection of variables for analysis. In particular, she found that users used a number of mechanisms for identifying appropriate variables for analysis, including what they knew about variable naming conventions, how particular questions relate to other questions, and even coding categories, if the question text did not provide enough information for selection. These findings reinforce the need for clear documentation of instrument design guidelines and instrument specifications, and for these to be readily available to data users.

Appendix A

Instrument Design Guidelines

Following are examples of some basic guidelines for design of interviewer-assisted computer-assisted instruments using Blaise interviewing software [20].

Text Characteristics

• Display question text on a light background color (cream), in mixed case, and in 12-point Arial, black.
• Display instructions in 11-point Arial bold blue.
• Display response categories:
  • those read out to the respondent, in 12-point Arial black.
  • those not read out to the respondent, in 11-point Arial bold blue
• Use underline for emphasis, sparingly.
• Place optional text in (parentheses).
• Display in-text references to function keys and numbers to type in mixed case within square brackets, for example, [Enter], [1], [F12], and [Ctrl-R].

On-Screen Instructions and Other Information

• Place references to interviewer aids (e.g., an event history calendar or show card instruction) and the question text in the upper left corner of the screen, above the question text.
• Place instructions that precede the question flush left with the question;
• Use icons to distinguish special instructions:
  • for respondent booklet instruction.
  • for event history calendar instruction, and.
  • .
• "Bullet" all other interviewer instructions with an 11-point bold blue diamond
  ().
• Single space within an instruction and double space between instructions;
• Place an online help indicator ( [F1]-Help ) above the question on the right margin, for questions with "question-by-question objectives" (QxQ's);
• Indent instructions that follow the question.
• Place any context-related information below the question-level help indicator on the right margin (for example, changing person-level information as the interviewer navigates a household roster or grid).
• Display instructions in the order associated with required interviewer tasks;
• Include an actual question in explicit interviewer checkpoints, displayed in 11-point Arial bold blue.
• Capitalize only key task-related action verbs (ASK, READ, ENTER, and PROBE), and only at the beginning of instructions.
• Keep instructions simple and concise.
  • Put long instructions or those not directly related to asking questions or entering responses into online help (question-by-question objectives).
• Conditional instructions start with the conditional phrase, not the action verb, and the action verb is not capitalized (e.g. conditional probes and data entry instructions).
• In probe instructions, place text to be read to the respondent in Arial black.
• Place references to respondent answers in quotation marks.

Examples of Formatted Questions

Following are examples of Web survey questions (in English and Arabic) of figures that can be included in instrument design guidelines to convey what questions formatted to specifications would look like. It is best to provide examples for all languages in which the survey will be implemented, and for all question types.

Appendix B

Following are examples taken or adapted from the European Social Survey (ESS) Round 4.

Instrument Overview

	Q#	Topics
Core	A1 — A10	Media; social trust
Core	B1 — B40	Politics, including: political interest, efficacy, trust, electoral and other forms of participation, party allegiance, socio-political orientations
Core	C1 — C36	Subjective well-being, social exclusion; religion; perceived discrimination; national and ethnic identity
Rotating module	D1-D55	Timing of life; the life course; timing of key life events, attitudes to ideal age, youngest age and oldest age of life events, planning for retirement
Rotating module	E1-E55	Personal and social well-being, helping others, feelings in the last week, life satisfaction, satisfaction with work.
Core	F1 — F73	Socio-demographic profile, including: household composition, sex, age, type of area, education & occupation of respondent, partner, parents, union membership, income, marital status
Supplementary	Section G	Human values scale
Supplementary	Section H	Test questions
Interviewer questionnaire	Section I	Interviewer self-completion questions

Missing Value Definitions

Not applicable: 6, 66, 666 etc.	respondent has been routed away from the question
Refusals: 7, 77, 777 etc.	respondent has explicitly refused
Don't know: 8, 88, 888 etc.	respondent has explicitly said "don't know"
No answer: 9, 99, 999 etc.	Missing data not elsewhere explained

Common Questions across Implementations

1. Interviewer checkpoints (with masks [?? /?? /??] and response format instructions [ (dd/mm/yy) ]
   
2. Survey question (variable A1, show card instruction (CARD 1), navigation instructions (e.g., if response is 04, ASK A2)
   
3. Show card (CARD 1, used for survey questions A1 through A6)
   
4. Data dictionary [variable A1; variable name TVTOT; variable label TV WATCHING, TOTAL TIME ON AVERAGE WEEKDAY; two-digit integer format with zero decimal places; universe (A1: Ask all); response options and codes; and skip instructions]:
   
5. Scale questions in grid (variables B30 through B33)
   
6. Show card (CARD 12, used for questions B30 through B33)
   
7. Data dictionary for scale questions in grid [variables B30 through B33; variable names GINCDIF, FREEHMS, PRTYBAN, SCNSENV; variable labels (e.g., GOVERNMENT SHOULD REDUCE DIFFERENCES IN INCOME LEVELS); single-digit integer; universe; response options and codes]:
   

   B30	GINCDIF	GOVERNMENT SHOULD REDUCE DIFFERENCES IN INCOME LEVELS	F1.0	1 Agree strongly 2 Agree 3 Neither agree nor disagree 4 Disagree 5 Disagree strongly 7 Refusal 8 Don't know 9 No answer	B30-B33: Ask All B30-B33: Same format,values and categories
   B31	FREEHIS	GAYS AND LESBIANS FREE TO LIVE LIFE AS THEY WISH
   B32	PRTYBAN	BAN POLITICAL PARTIES THAT WISH OVERTHROW DEMOCRACY
   B33	SCNSENV	MODERN SCIENCE CAN BE RELIED ON TO SOLVE ENVIRONMENTAL PROBLEMS

Country-Specific Questions

1. ESS highlights country-specific questions in gray in the questionnaire specifications, for example, variabes B11 and B12:
   
2. Data dictionary (variables B11 and B12; variable names VOTE and PRTVTxx; variable labels; one- and two-digit integer formats; response options and codes; universes; and skip instructions):
   

   B11	VOTE	VOTED LAST NATIONAL ELECTION	F1.0	1 Yes 2 No 3 Not eligible to vote 7 Refusal 8 Don’t know 9 No answer	B11: Ask all Ask B12
   					Ask B12 if B11=1
   B12	PRTVTxx	PARTY VOTED FOR IN LAST NATIONAL ELECTION [COUNTRY]	F2.0	"66" Not applicable "77" Refusal "88" Don't Know "99" No answer	B12: Country-specific question, see section E.1.1

Glossary

Behavior coding

Systematic coding of the interviewer-respondent interaction in order to identify problems that arise during the question-answer process.

Codebook

A document that provides question-level metadata that are matched to variables in a dataset. Metadata include the elements of a data dictionary, as well as basic study documentation, question text, universes (the characteristics of respondents who were asked the question), the number of respondents who answered the question, and response frequencies or statistics.

Context effects

The impact of question context, such as the order or layout of questions, on survey responses.

Data dictionary

Question or variable-level metadata, including variable names, labels, and data types.

Hard consistency check

A warning that there is a discrepancy between the current response and a prior response; the interviewer or respondent cannot continue until the check is resolved.

Interface design

Aspects of computer-assisted survey design focused on the interviewer’s or respondent’s experience and interaction with the computer and instrument.

Item nonresponse

The lack of information on individual data items for a sample element where other data items were successfully obtained.

Measurement error

Survey error (variance or bias) due the measurement process; that is, error introduced by the survey instrument, the interviewer, or the respondent.

Metadata

Data that describes other data. The term encompasses a broad spectrum of information about the survey, from study title to sample design, details such as interviewer briefing notes, contextual data and/or information such as legal regulations, customs, and economic indicators.

Nonresponse error

Error (variance or bias) that is introduced when not all sample members participate in the survey (unit nonresponse) or not all survey items are answered (item nonreponse) by a sample member.

Question-by-question objectives

Text associated with some questions in interviewer-administered surveys that provides information on the objectives of the questions.

Soft consistency check

A warning that there is a discrepancy between the current response and a prior response; the interviewer or respondent may ignore the check and continue the survey.

Source instrument

The original instrument from which other (target) instruments are translated or adapted as necessary.

Target population

The finite population for which the survey sponsor wants to make inferences using the sample statistics.

Unit nonresponse

A sample case that has little or no information because the individual declined the invitation to participate in the survey. Also known as a nonrespondent.

Universe statement

A description of the group of respondents to which the survey item applies (e.g., "Female, ≥ 45, Now Working").

Usability evaluation

Evaluation of a computer-assisted survey instrument to assess the impact of design on interviewer or respondent performance. Methods of evaluation include review by usability experts and observation of users working with the computer and survey instrument.

References

[1] Aykin, N. (ed.). (2005). Usability and Internationalization of Information Technology. Mahwah, NJ: Lawrence Erlbaum Associations.

[2] Aykin, N., & Milewski, A.E. (2005). Practical Issues and Guidelines for International Information Display. In Aykin, N. (ed.), Usability and Internationalization of Information Technology. Mahwah, NJ: Lawrence Erlbaum Associations.

[3] Clemmensen, T., &. Goyal, S. (2005). Cross-Cultural Usability Testing—The Relationship between Evaluator and Test User. Working Paper no. 06-2005, Department of Informatics, Copenhagen Business School.

[4] Couper, M.P. (1999). The Application of Cognitive Science to Computer-Assisted Interview. In Sirken et al. (eds.), Cognition and Survey Research. New York: John Wiley & Sons, Inc.

[5] Couper, M.P. (2001). Web Surveys: The Questionnaire Design Challenge. Proceedings of the 53rd Session of the ISI.

[6] Couper, M.P., Traugott, M.W., & Lamias, M.J. (2001). Web Survey Design and Administration. Public Opinion Quarterly 65, 230-253.

[7] Couper, M.P. (2002). New Technologies and Survey Data Collection: Challenges and Opportunities. Paper presented at the International Conference on Improving Surveys, Copenhagen.

[8] de Leeuw, E., Hox, J., & Kef, S. (2003). Computer-Assisted Self-Interviewing Tailored for Special Populations and Topics. Field Methods 15, 233-251.

[9] Dillman, D. A. (2007). Mail and Internet Surveys: The Tailored Design Method. New York: Wiley.

[10] Eurostat. (2003). Definition of Quality in Statistics. Retrieved Sept. 25, 2008 from http://epp.eurostat.ec.europa.eu/pls/portal/docs/ page/pgp_ds_quality/tab47141301/definition_2.pdf

[11] Hansen, S.E., and Couper, M.P. (2004). Usability Testing as a Means of Evaluating Computer Assisted Survey Instruments. In Presser et al. (eds.), Methods for Testing and Evaluating Survey Questionnaires. New York: John Wiley & Sons.

[12] Heerwegh, D. (2003). Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey. Social Science Computer Review 21(3), 360-373.

[13] Hert, C.A. (2001). Studies of Metadata Creation and Usage. Proceedings of the Federal Committee on Statistical Methodology. Washington, D.C.: U.S. Department of Commerce.

[14] Jagne, J., & Smith-Atakan, A.S.G. (2006). Cross-cultural interface design strategy. Universal Access Information Society 5, 209-305.

[15] Kondratova, I., & Goldfarb, I. (2007). Color Your Website: Use of Colors on the Web. Usability and internationalization: Global and local user interfaces 4560, 123-132.

[16] Murphy, E.D., Nichols, E.M., Anderson, A.E., Harley, M.D., & Pressley, K.D. Building Usability into Electronic Data-Collection Forms for Economic Censuses and Surveys.

[17] Russo, P., & Boor, S. (1993). How Fluent is Your Interface?: Designing for International Users. Proceedings of the INTERACT ’93 and CHI '93 conference on human factors in computing systems. New York: Association for Computing Machinery.

[18] Smith, T. W. (1995). "Little Things Matter. Little things matter: A sampler of how differences in questionnaire format can affect survey responses." Proceedings of the American Statistical Association, Section on Survey Research Methods, 1046-1051. Alexandria, VA: American Statistical Association.

[19] Smith, T. W. (2005). The Laws of Studying Societal Change. GSS Social Change Report 50. NORC/University of Chicago.

[20] Survey Research Center. (2007). SRC Blaise Standards. Ann Arbor: The University of Michigan.

Return to top

Previous chapter | Next chapter | Home

© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to publish any of this material in any form.

View Chapters

• Introduction
• Study Structure
• Tenders, Bids, & Contracts
• Ethical Considerations
• Sample Design
• Questionnaire Design
• Translation
•      Translation Assessment
•      Language Harmonization
•      Translators
•      Translation Budgeting
•      Translation Scheduling
•      Translation Tools
• Adaptation
• Survey Instrument Design
• Pretesting
• Interviewer Recruitment & Training
• Data Collection
• Harmonization of Data
• Data Processing
• Dissemination
• Assessing Quality
• Glossary
• Bibliography