A survey can be well administered and still produce unreliable findings. The quality of survey evidence depends on decisions made long before enumerators begin interviewing respondents.
Surveys are widely used to understand communities, assess programme performance, measure outcomes and inform policy. They can generate valuable evidence, but collecting a large volume of responses does not automatically produce trustworthy results.
A survey may achieve its target sample size and complete fieldwork on schedule yet fail to answer the research question. The sample may not represent the intended population. Questions may measure something different from what researchers intended. Respondents may misunderstand questions, enumerators may interpret instructions differently, or field teams may apply inconsistent procedures.
These problems are difficult to correct after data collection. Some can be reduced through cleaning and analysis, but others undermine the validity of the findings from the outset.
Good survey design begins before fieldwork. It requires a clear research purpose, an appropriate sampling strategy, meaningful measurement, careful piloting and a practical system for monitoring data quality.
1. Start with the decision, not the questionnaire
Before developing survey questions, researchers should establish what the survey is intended to find out and how the findings will be used.
A broad topic such as youth employment, access to healthcare or agricultural productivity is not yet a research question. Researchers need to identify the specific problem, population, outcomes and decisions that the evidence should inform.
For example, a programme may want to understand whether its business training is helping participating enterprises improve their performance. A questionnaire that simply asks participants whether they enjoyed the training may provide useful feedback, but it cannot establish whether business performance changed.
A stronger design would identify the relevant outcomes, such as changes in sales, business survival, employment or record-keeping practices, and determine what information is needed to assess them.
This process helps researchers distinguish between information that is interesting and information that is necessary.
Before drafting a questionnaire, ask:
- What problem are we investigating?
- Who needs to be studied?
- What outcomes, behaviours or conditions must be measured?
- What period should respondents consider?
- What decisions will the findings inform?
These questions establish the foundation for the sampling strategy, questionnaire and analysis plan.
2. Choose a sampling strategy that fits the population
Sampling determines whose experiences and characteristics are represented in the survey.
When it is not feasible to collect data from an entire population, researchers select a subset of individuals, households, enterprises, institutions or other units. The method used to select that subset affects what conclusions can reasonably be drawn.
Probability sampling methods, including simple random, stratified and cluster sampling, give eligible units a known probability of selection under the design. When implemented appropriately, they support statistical inference to the population covered by the sampling frame.
Non-probability methods, such as purposive or convenience sampling, can be useful for exploratory research, qualitative inquiry or populations that are difficult to reach. However, findings from these samples should not automatically be treated as statistically representative of a wider population.
Sample size is only one part of the decision. Researchers must also consider:
- Target population: Who should the findings describe?
- Sampling frame: What list or mechanism will identify eligible units?
- Selection method: How will participants or units be selected?
- Coverage: Which eligible groups may be excluded?
- Non-response: Who may be missed because they cannot be contacted or decline to participate?
- Precision: How much uncertainty is acceptable for the intended estimates?
A large sample cannot compensate for systematic exclusion of important groups. For example, a household survey conducted only among people who are available during working hours may underrepresent employed adults.
Where a survey uses stratification, clustering, unequal selection probabilities or other complex design features, these should be reflected in the analysis. Ignoring the survey design can produce misleading standard errors, confidence intervals and population estimates.
3. Translate concepts into meaningful measures
Many important research concepts cannot be observed directly.
Poverty, empowerment, resilience, service quality, institutional trust and programme effectiveness require researchers to decide what observable information will represent them.
This process is called operationalisation: translating an abstract concept into measurable variables or indicators.
Consider the concept of access to healthcare. Researchers might measure distance to a facility, travel time, affordability, availability of services, waiting time or whether needed care was received. Each indicator captures a different dimension.
Choosing only one measure may create an incomplete picture.
The measurement strategy should therefore make explicit:
- What exactly is being measured?
- How is the concept defined?
- Which indicator or question represents it?
- What response options are appropriate?
- What reference period applies?
- How will the resulting variable be interpreted?
Questions should be specific enough to reduce ambiguity. Asking whether a household has “good access” to healthcare invites respondents to apply different interpretations. Asking about travel time, costs and use of services produces more clearly defined information.
Researchers should also distinguish between a measure and the broader concept it is intended to represent. A single indicator may provide evidence about one dimension of a concept without capturing the entire construct.
4. Design questions respondents can answer accurately
Questionnaire wording affects the information collected.
Questions may introduce bias when they are leading, ambiguous, double-barrelled, overly technical or dependent on difficult recall. Response options can also shape results when they omit common experiences or force respondents into categories that do not fit.
Consider the question:
“Do you agree that the programme has significantly improved your household’s economic wellbeing?”
This wording combines a positive assumption about the programme with a broad concept that respondents may interpret differently.
A more neutral approach would ask about specific changes over a defined period. For instance, researchers might ask whether household income sources changed, whether the respondent started or expanded an economic activity, or how the household’s ability to meet selected expenses changed.
The most appropriate questions depend on the research objective. There is no universally superior questionnaire.
Good questionnaire design also considers:
- Logical question order and clear transitions.
- Response options that are mutually exclusive where appropriate and sufficiently comprehensive.
- Skip patterns that prevent irrelevant questions from being asked.
- Clear definitions for terms respondents may interpret differently.
- Recall periods suited to the information being requested.
- Appropriate handling of sensitive questions.
- Translation that preserves meaning rather than simply replacing words.
Questions should be reviewed not only by researchers but, where possible, by people familiar with the population and context being studied.
5. Pilot the survey before full deployment
A questionnaire that looks clear to its designer may be confusing to respondents.
Piloting allows researchers to identify problems before they affect the full dataset. It should test more than whether the questionnaire can be completed from beginning to end.
A useful pilot examines whether respondents understand questions as intended, whether response options fit their experiences, whether the interview takes a reasonable amount of time, and whether digital forms behave correctly.
For electronic questionnaires, testing should include validation rules, skip logic, calculations, required fields, repeat groups, preloaded information and data submission. Where relevant, researchers should also test offline functionality, device compatibility and the handling of incomplete interviews.
Piloting can reveal questions that respondents routinely misunderstand, answer inconsistently or cannot answer at all. It can also expose unrealistic fieldwork assumptions.
The pilot findings should lead to documented revisions. If major changes are made to the questionnaire or measurement approach, the revised instrument may need further testing.
A pilot is not merely a rehearsal for enumerators; it is a test of the entire survey design.
6. Prepare field teams to collect consistent data
Even a carefully designed instrument can produce inconsistent data if field teams interpret or apply it differently.
Enumerator training should cover the purpose of the survey, definitions of key concepts, respondent selection, informed consent, confidentiality, question delivery, probing rules and procedures for handling unusual situations.
Training should include practical exercises and supervised practice. Enumerators need to understand not only what the questionnaire says but also why particular procedures matter.
For example, if household selection rules are ignored whenever a selected respondent is unavailable, enumerators may begin interviewing more accessible households instead. This can introduce selection bias.
Field protocols should explain how to handle non-response, callbacks, refusals, incomplete interviews and deviations from the sampling plan. Supervisors should be able to identify when an issue requires correction, escalation or documentation.
Consistency does not mean treating every situation mechanically. It means applying agreed procedures while recording legitimate exceptions transparently.
7. Build data quality checks into fieldwork
Data quality assurance should operate throughout the survey, not only after the data have been exported for analysis.
A practical system combines preventive controls, field monitoring and post-collection checks.
Preventive controls include clear instructions, well-designed response options, validation rules and appropriate restrictions on data entry. Monitoring may identify unusual interview durations, missing information, unexpected response patterns, duplicate records or inconsistencies between related answers.
These signals require investigation. An unusually short interview, for example, may indicate an incomplete interview, a technical issue or a legitimate case that needs contextual explanation. It should not automatically be classified as misconduct.
Supervisors should review findings promptly so that problems can be corrected while fieldwork is still underway. Any changes to survey procedures or instruments should be documented and communicated consistently.
After collection, researchers should check completeness, consistency, valid ranges, duplicate records, missingness and deviations from the intended sample. They should also maintain clear records of cleaning decisions so that exclusions, recoding and corrections are traceable.
Importantly, automated checks cannot establish every aspect of data quality. A value can be within the permitted range and still be inaccurate. Data quality therefore depends on both technical controls and sound research judgement.
8. Match the analysis to the survey design
The final stage is ensuring that analysis reflects how the data were collected and what the measurements can support.
Researchers should examine missing data, response patterns, sample composition and potential sources of bias before interpreting results. They should distinguish descriptive findings from estimates of relationships and from claims about causality.
For example, a survey showing that programme participants have higher incomes than non-participants does not, by itself, establish that the programme caused the difference. Participants and non-participants may differ in other ways that influence income.
Similarly, a survey of programme participants may describe their experiences accurately while providing insufficient evidence about outcomes for the wider population.
Where probability sampling is used, weighting and variance estimation may be necessary to account for the design and non-response. Where non-probability sampling is used, the limitations of population generalisation should be stated clearly.
Researchers should also report the relevant denominators, explain missing information and communicate uncertainty. A precise-looking percentage is not necessarily a reliable estimate if the underlying sample or measurement is weak.
The purpose of analysis is not to make the data appear stronger than they are. It is to determine what conclusions the evidence can reasonably support.
A practical checklist before launching a survey
Before fieldwork begins, research teams should be able to answer the following questions:
- Is the research problem clear, and are the intended decisions identified?
- Is the target population explicitly defined?
- Is the sampling strategy appropriate for the research objectives?
- Are key concepts translated into valid, clearly defined measures?
- Are the questions neutral, understandable and suitable for respondents?
- Has the questionnaire been piloted and revised?
- Have enumerators and supervisors been trained on consistent procedures?
- Are data quality checks and escalation procedures in place?
- Does the analysis plan reflect the sampling design and measurement limitations?
- Are consent, confidentiality, data protection and ethical requirements addressed?
If several of these questions remain unanswered, the survey may not be ready for full deployment.
Conclusion: Reliable evidence is designed, not simply collected
Survey quality is determined by a connected series of decisions. The research problem guides what needs to be measured; the sampling strategy determines whose information is captured; the questionnaire shapes the responses; field procedures influence consistency; and data quality controls help establish whether the resulting dataset is fit for analysis.
Weaknesses at any of these stages can undermine the usefulness of the findings. Collecting more responses will not necessarily resolve those weaknesses.
For organisations using research to guide programmes, resource allocation or public policy, the priority should be to design surveys around the decisions they need to inform and to protect evidence quality at every stage.
At DAREL Consulting, we connect research design, evaluation, data quality and analytical methods to help organisations generate evidence that is credible, interpretable and useful for action.
The strongest survey is not simply the one that collects the most data. It is the one that produces the right information, from the right sources, using methods that support defensible conclusions.
