• Home
• |
• Contact Us
• |
• User Guide

Study, Organizational, and Operational Structure Guidelines

I. Study, Organizational, and Operational Structure

Webpage last modified: 2008-Oct-29

Introduction

The following guidelines outline the various organizational and operational aspects that should be considered in the structural design of a cross-national project or any project involving multiple languages or cultures. In particular, the focus is a cross-cultural survey that is implemented only once or the first wave of a longitudinal cross-cultural study, although most of the guidelines are also applicable to additional waves of a longitudinal study.

Cross-cultural studies are organized in many different ways, and each method has its advantages and disadvantages. Key determinants that will affect the structure of such studies include the source(s) and flow of funding, the availability of human and technical resources, both centrally and at local levels, the best way (mode) of contacting and interviewing respondents, and the research infrastructure. These factors may vary from country to country, and all will influence how the overall study is organized, designed, and later implemented. Although increased spending does not necessarily improve survey quality, there is a minimum cost involved in assuring quality. The inverse relationship between survey error and survey cost generally drives the target level of quality of the survey (final product).

One organizing structure that has been used often in large-scale, cross-cultural studies, and that will be referenced throughout these guidelines, is some form of a coordinating center. The coordinating center generally includes people from different countries, institutions, and affiliations, and assumes varying levels of organizational responsibility or guidance for some or all of the aspects of the study. Examples of different coordinating centers can be found in information on the websites of survey programs such as the International Social Survey Programme, European Social Survey, Afrobarometer, and World Mental Health Initiative.

Guidelines

Goal: To establish the study's overall structure and locus of control at all levels and across all aspects of the study's design and implementation, and to communicate this structure and the procedural consequences of the structure to each participating country and survey organization.

1. Identify and specify the chosen organizational structure for a study at the supranational, national, and, as necessary, subnational level.
   Rationale

   In order to manage a complex undertaking efficiently and effectively, roles and responsibilities must be clearly delineated and communicated; a framework and structure of responsibilities and tasks should be created that facilitates their execution. This can be accomplished through a central coordinating body that teams with local expertise in each participating country. The coordinating center should first determine its own structure and identity to then best set the organizational standards for the distributed locations.

   Procedural steps

   • Become familiar with the climate and culture of all locations being studied in order to establish the most appropriate organizational structure.
     • Recent media material can depict the pertinent issues of these societies.
   • Assess the substantive, methodological, and contractual expertise needed both centrally and for all locations.
   • Identify the impact that structural aspects of the planned organization have on control, responsibility, and communication at the central and local country levels:
     • Determine reporting and fiduciary responsibilities to funding sources
     • Consider the creation of a number of domain-specific working groups comprised of qualified participants from the countries and the central organization, including external experts as needed. Examples could be groups for sampling, questionnaire design, interviewer training, and quality control.
     • Specify leadership, authority, and responsibility across all domains and levels.
     • Balance central control and implementation of decisions about standardization with appropriate local input.
     • Balance central and local participation in deciding how to address general and individual adaptation in processes, methods, and substantive content.
     • Determine control of the country-specific and cross-national data, ensuring confidentiality, including analysis and publication of the data (see Dissemination of Survey and Statistical Data).
   • In addition to working groups, consider the creation of a lead team to oversee the entire survey implementation, as well as individual task teams who supervise different stages of the process:
     • Define responsibilities for each team.
     • Arrange regular meetings (or conference calls) of working group and team leaders to discuss study progress.
     • Arrange for expert consultation on substantive and methodological aspects during the planning and execution of all stages of the research process
   • Develop a communication flowchart (i.e., who talks to whom about what) with one reference person and one back-up person at each point in the flowchart [9].
   • Evaluate the feasibility of implementing the survey structure given the populations, governments, and politics of the countries being studied. Identify accommodations that may need to be at the local level due to, for example, illiterate populations, oppressive governments, and/or neglected infrastructures.
   Lessons learned

   • Organizational structures for cross-cultural surveys differ along a number of important dimensions, and these shape how and when decisions are made. For example, multiple sources of funding are typically needed to provide enough support to coordinate a large-scale, cross-cultural survey project; furthermore, each country may be funded separately. Funding sources may have requirements that complicate reporting structures within the research project and/or conflict with the goals of the overall cross-cultural project. The points at issue may relate to a wide variety of features, from data availability to the content of questionnaires.
   • The organizational structure should not be so complex that it unnecessarily prolongs decision-making and task completion. Yet, inadequate definition of organizational structures may result in haphazard decision-making and task completion, potentially leading to duplication of effort and/or resulting in unnecessary conflict. The study organizers should keep these conflicting concerns in mind while making their decisions.
2. Clearly define and then implement specifications for every phase of the survey, keeping the study's research goals in mind at all times.
   Rationale

   Study specifications ensure that critical aspects of the survey process are defined, and can then be measured and controlled. They simultaneously identify required or expected quality standards and help ensure comparability across countries. The specifications should therefore be detailed and measurable with clearly delineated deliverables at each phase of the study. The study specifications form the basis of the country-level contracts (see Tenders, Bids and Contracts).

   Procedural steps

   • Clearly state the study's goals and objectives, ensuring that central and local study objectives do not conflict [2] [3].
   • Taking into account the overall study goals and weighing the cost/error tradeoffs involved, detail the study design, specifications, and requirements [2]. See the following chapters for procedural steps regarding recommendations for specifications on the topics:
     • Ethical Considerations in Surveys
     • Sample Design
     • Questionnaire Design
     • Translation
     • Adaptation
     • Survey Instrument Design
     • Pretesting
     • Interviewer Recruitment, Selection, and Training
     • Data Collection
     • Harmonization of Survey and Statistical Data
     • Data Processing And Statistical Adjustment
     • Dissemination of Survey and Statistical Data
   • Create a study timeline, production milestones, and deliverables with due dates [7]. Keep in mind local considerations which may affect the study's due dates.
     • Require interim products.
     • Require frequent reporting.
     • Require deliverables with unique identifiers for interviewers and sample elements .
     • Decide what actions to take, as necessary, to rectify delays or delinquencies.
     • Determine sanctions/penalties for a participant location's failure to meet study specifications.
   • If and where possible, include methodological experiments in the study design and specifications that can inform and improve future surveys [6].
   • Establish a backup plan to ensure the completion of a high-quality survey in case countries are unable to meet the study specifications.
   Lessons learned

   • In any given study, some specifications will be more important than others. Top-priority specifications should be determined and communicated to all participants before a study is initiated.
   • Organizing clearly-defined specifications and a system of checks and balances will help maintain the highest standards throughout all levels and domains of a cross-cultural study.
   • Difficult decisions may need to be made if the specifications are not met. For example, if a country/location does not meet a specified response rate goal or does not provide specified information on interviewer contact efforts, a decision needs to be made about whether the data from that location should be excluded from the final data set.
   • A failure to communicate overall study goals may lead to local decisions that threaten comparability across countries/locations. For example, a country may remove particular items from the questionnaire in order to reduce the cost of the interview without realizing that those items are necessary to measure an important survey construct. Conversely, a country may add items to the questionnaire in order to study a locally-relevant topic without realizing that those items may affect the management, processing, and quality of the data.
   • The coordinating center should be aware that questionnaire design, translation, adaptation, and pretesting often are more time-consuming and difficult than anticipated. For example, the European Social Survey (ESS) took roughly five years to design because there were unexpected challenges such as first needing to adapt or entirely rewrite core question items, instead of taking them directly from existing surveys (as planned) [4].
3. Define quality standards at all stages of the survey process [8] [9].
   Rationale

   Setting quality standards is critical to ensuring the same level of methodological rigor across countries or locations [5]. Local adaptations will be necessary and appropriate for some aspects of implementation of the project, but any adaptation in the procedure or instrument should be thoroughly discussed and evaluated beforehand. Frequent measurement and reporting to a coordinating center, along with sufficient methodological support, should allow for timely intervention if problems do arise.

   Procedural steps

   • Acquaint survey organizers with important quality control literature, especially those people designing and specifying the study structure.
   • Using the study's specifications, set realistic, measurable quality standards that reflect the following: relevance, accuracy, timeliness, accessibility, comparability, coherence, and completeness, as described in the quality assurance literature [2] (see Assessing Quality for Cross-Cultural Surveys).
     • Ensure all survey questions are relevant to the study objectives.
     • Maximize comparability across both time and space. For example, document the different versions of a question implemented at different times and/or in different places (or languages), and make this information available to data users.
     • If possible, assess accuracy by looking at the differences between the study estimates and any available "true" or gold standard values (see explanation of mean square error in relation to quality in Data Collection).
     • Base statistical outcomes on common definitions, classifications, and methodological standards (see Data Processing and Statistical Adjustment).
     • Disseminate the results in a timely manner (see Dissemination of Survey and Statistical Data).
     • Make the results widely accessible and assist data users in understanding them (see Dissemination of Survey and Statistical Data).
   • Form a team in each country/location that regularly meets to discuss the quality of the local survey. The team should have or should be provided with methodological expertise needed for this task. The team should document and report any developments to the coordinating center [1] [2].
   • Quickly address any deviations from expectations that may occur (see Tenders, Bids, and Contracts) [2].
   • Monitor costs in order to avoid overruns:
     • Create a cost-monitoring instrument and checklist.
     • Ensure sufficient funds are allocated to budget quality assessment and documentation activities.
     • Assess risk and determine contingencies for each phase of implementation—weighing cost and errors.
   • Implement a certification process or signing-off procedure for each stage in order to check and document that the study design and specification standards are being followed. Invoke sanctions, as specified in the contract, if the certification is not fulfilled (see Tenders, Bids, and Contracts).
   • Use a Plan-Do-Check-Act cycle (PDCA) by first determining the study's quality standards, then implementing them throughout the research process, assessing quality indicators at each stage, and finally making appropriate changes to repeat the cycle of PDCA [2].
   • Incorporate methodological research where possible. This will inform long-term quality improvement [6] [8].
   • See subsequent chapters for specific recommendations on quality control and quality assurance:
     • Ethical Considerations in Surveys
     • Sample Design
     • Questionnaire Design
     • Translation
     • Adaptation
     • Survey Instrument Design
     • Pretesting
     • Interviewer Recruitment, Selection, and Training
     • Data Collection
     • Harmonization of Survey and Statistical Data
     • Data Processing And Statistical Adjustment
     • Dissemination of Survey and Statistical Data

   Return to top of guideline

   Lessons learned

   • Variations in country-level research infrastructure, research traditions, and methodological rigor need to be thoroughly investigated and understood when setting quality standards. Some countries will need more assistance in meeting some standards, and this should be taken into account early in the planning process.
4. Document every stage of the survey lifecycle.
   Rationale

   Only if quality is continually measured and documented can an analysis assist in pin-pointing problems. Even if sources of error are not recognized while the study is underway, such documentation can be used to improve specifications and procedures for future studies.

   Procedural steps

   • Record any modifications made either centrally or locally to the study protocol.
   • Create a clear, concise description of all survey implementation procedures (i.e., keep descriptions user friendly). Ensure the procedures are as transparent as possible, with sufficient detail, to ensure that they could, theoretically, be replicated. Examples of topics which need to be covered include:
     • Sampling design and implementation (see Sample Design).
     • Questionnaire development (see Questionnaire Design).
     • Changes as a result of pretesting (see Pretesting).
     • Interviewer characteristics (see Interviewer Recruitment, Selection, and Training).
     • Use of incentives (see Data Collection).
     • Mode of data collection (see Data Collection).
     • Production schedule (see Data Collection).
     • Sample coversheets (see Data Collection).
     • Respondent selection and initial contact procedures (see Data Collection).
     • Supervisory structure (see Data Collection).
     • Observations (recordings, scripted mock interviews, etc.) (see Data Collection).
     • Bio measures (see Data Collection).
     • Interviewer falsification (see Data Collection).
     • Reluctance handling protocols (see Data Collection).
     • Recontacts and reinterviews (see Data Collection).
     • Final non-interviews (see Data Collection).
     • Data entry and data capture procedures (see Data Processing and Statistical Adjustment).
     • Data editing checks (see Data Processing and Statistical Adjustment).
     • Data coding rules (see Data Processing and Statistical Adjustment).
     • Encountered errors and their resolution (see Data Processing and Statistical Adjustment).
     • Final response rate with final results for every released sample element (see Data Processing and Statistical Adjustment).
     • Quality control outcomes to demonstrate that expected levels were reached.
     • Copies of all study materials (training manuals, questionnaires, etc.).
   • For specific recommendations on documentation, see the chapters below:
     • Ethical Considerations in Surveys
     • Sample Design
     • Questionnaire Design
     • Translation
     • Adaptation
     • Survey Instrument Design
     • Pretesting
     • Interviewer Recruitment, Selection, and Training
     • Data Collection
     • Harmonization of Survey and Statistical Data
     • Data Processing And Statistical Adjustment
     • Dissemination of Survey and Statistical Data
   Lessons learned

   • Documentation should accompany the survey implementation phases. On the one hand, as described here, documentation is used to detect problems in time to address them. In addition, if documentation is left to the end of the survey, details will be lost or forgotten.
   • Documentation requirements and formats should be standardized across all countries, in order to be able to measure quality and comparability. Any changes countries make to their protocols and procedures must be carefully documented, since these could explain potential differences in the data, either over the course of the study (within a country) or across variables (between countries).

© 2008 The authors of the Guidelines hold the copyright. Please contact us if you wish to publish any of this material in any form.

View Chapters

• Introduction
• Study Structure
• Tenders, Bids, & Contracts
• Ethical Considerations
• Sample Design
• Questionnaire Design
• Translation
•      Translation Assessment
•      Language Harmonization
•      Translators
•      Translation Budgeting
•      Translation Scheduling
•      Translation Tools
• Adaptation
• Survey Instrument Design
• Pretesting
• Interviewer Recruitment & Training
• Data Collection
• Harmonization of Data
• Data Processing
• Dissemination
• Assessing Quality
• Glossary
• Bibliography