Introduction
Evidence can inform better decisions, but only when we understand what it establishes, how it was generated and where its limitations lie.
Organisations rely on research reports, evaluation findings, administrative records, surveys and statistical analyses to guide decisions. These sources can reveal important patterns, challenge assumptions and identify opportunities for improvement.
Yet evidence is not automatically reliable simply because it appears in a published report, comes from a recognised institution or is supported by statistical analysis.
A programme evaluation may report positive outcomes without establishing whether the intervention caused them. A survey may produce precise estimates while excluding important groups. Two studies may reach different conclusions because they examine different populations, use different methods or investigate different aspects of the same problem.
When such findings inform programme design, resource allocation or public policy, the ability to assess evidence critically becomes essential.
Critical evidence review is not about dismissing research or searching for weaknesses in every study. It is about understanding what the available information can reasonably establish, how confidently conclusions can be drawn and what remains uncertain.
1. Start with the claim, not the report
Before assessing a study, identify the specific claim it is being used to support.
A report may contain dozens of findings, but only some may be relevant to the decision under consideration. Separating the central claim from the supporting information helps focus the review.
Consider a development organisation evaluating a youth employment programme.
The programme report states that 70% of participants obtained employment after completing training. This may be useful evidence of participants’ employment status at follow-up.
However, several questions remain:
- What proportion of participants was successfully followed up?
- How was employment defined?
- Were participants employed before entering the programme?
- How long after training was employment measured?
- Were the jobs sustained, adequately paid or relevant to the skills acquired?
- Would similar participants have obtained employment without the programme?
Each question concerns a different aspect of the claim.
If the purpose is to understand the employment status of participants, the reported proportion may be informative, provided the measurement and follow-up procedures are sound.
If the purpose is to establish whether the programme improved employment outcomes, additional evidence is needed to distinguish the programme’s contribution from other explanations.
The first task in evidence review is therefore to clarify precisely what is being claimed and what kind of evidence would be needed to support it.
2. Examine where the evidence comes from
The source of evidence matters because it influences how information was collected, analysed and presented.
Relevant sources include peer-reviewed research, official statistics, administrative records, programme evaluations, systematic reviews and independent assessments. Each can provide valuable information, but each also has limitations.
Peer-reviewed research has undergone scholarly assessment, but publication does not guarantee that every method, result or interpretation is beyond question.
Official statistics may benefit from established collection procedures and broad coverage, yet they can still be affected by measurement limitations, reporting delays or changes in definitions.
Programme reports may contain detailed operational information that external researchers cannot access, but their findings should be examined in light of the evaluation design, reporting incentives and transparency of the analysis.
Administrative records can offer extensive information about service use or transactions, but they generally represent what a system records rather than everything that occurs in the population.
When reviewing a source, consider:
- Who produced the evidence and for what purpose?
- What population, location and period does it cover?
- Where did the data originate?
- Are the methods and assumptions documented?
- Are the findings and limitations reported transparently?
- Can important results be checked against other credible sources?
These questions help establish whether the evidence is appropriate for the intended use. The source should be assessed on its merits rather than accepted or rejected solely because of its institutional identity.
3. Assess whether the method supports the conclusion
A central principle of evidence review is that the strength of a conclusion should match the research design.
Different methods answer different questions. A descriptive survey can estimate how common a characteristic is within the population it adequately represents. Qualitative interviews can explore experiences, perceptions and possible mechanisms. Statistical models can examine relationships between variables. Experimental and suitable quasi-experimental designs can help estimate causal effects under appropriate assumptions.
Problems arise when a method is used to support a conclusion it cannot adequately establish.
For example, a survey finds that farmers who receive agricultural extension services have higher crop yields than farmers who do not.
This is evidence of an association. It does not automatically establish that extension services caused the difference.
Farmers who receive extension services may differ from other farmers in access to credit, land quality, education, market access or willingness to adopt new practices. These factors may also influence yields.
A credible causal assessment must consider alternative explanations and use an appropriate design and analytical strategy to address them.
Similarly, interviews with programme participants may provide valuable insight into why a service is useful or difficult to access. However, participants’ experiences alone may not establish how common those experiences are across the entire target population.
The relevant question is not whether a method is sophisticated or familiar. It is whether the method is appropriate for the claim being made.
4. Look closely at measurement and data quality
Even a well-designed study can produce weak evidence if its key concepts are measured poorly.
Consider a programme intended to improve household economic wellbeing. If success is measured only through programme participation, the indicator may capture access to an intervention rather than a change in household wellbeing.
If employment is measured using a single question without clarifying the reference period or employment status, estimates may not be comparable across respondents or studies.
When reviewing evidence, examine how the principal variables were defined and measured.
Important questions include:
- Are the concepts clearly defined?
- Do the indicators adequately represent those concepts?
- Are the data collection procedures documented?
- Is missing information substantial or unevenly distributed?
- Could self-reporting, recall or recording errors affect the findings?
- Are the measurement procedures comparable across groups and periods?
Sample size also deserves careful attention, but it should not be treated as a guarantee of quality.
A larger sample can improve statistical precision under suitable conditions. It cannot automatically correct systematic measurement error, selection bias or an indicator that fails to capture the concept of interest.
A smaller, carefully designed study may provide stronger evidence for a particular question than a much larger dataset with serious limitations.
5. Distinguish statistical significance from practical importance
Statistical results are often used to communicate whether an observed relationship is likely to reflect more than random variation under a specified statistical model.
However, statistical significance is not the same as practical importance.
A study involving a very large sample may identify a small difference as statistically significant even when that difference has limited operational relevance.
Conversely, a study with a small sample may produce an important estimated effect but insufficient precision to distinguish it clearly from zero.
Evidence review should therefore consider the size and direction of an estimated effect, its uncertainty, the quality of the underlying data and the implications for the decision being made.
For example, an evaluation may find that an intervention increases average monthly earnings by a small amount. Whether this matters depends on the size of the increase relative to household needs, the cost of the intervention, how the benefits are distributed and whether the change is sustained.
Confidence intervals and other uncertainty measures can help readers understand how precise an estimate is. They should be interpreted alongside the study design and substantive context rather than as standalone indicators of evidence quality.
The objective is to understand the magnitude and relevance of a finding, not merely whether it crosses a statistical threshold.
6. Compare findings across studies without forcing agreement
Evidence becomes more informative when findings from different sources are examined together.
A single study may provide a detailed account of one population or setting. Other studies may help establish whether similar patterns occur elsewhere, under different conditions or over longer periods.
However, comparing evidence requires more than counting how many studies report positive or negative results.
Studies may differ in their research questions, populations, intervention designs, outcome definitions, follow-up periods and analytical methods. These differences can explain why findings do not always agree.
Suppose one evaluation finds that agricultural training improves farmers’ adoption of recommended practices, while another finds little change in adoption.
The results may differ because the programmes operated in different contexts, the participants faced different resource constraints or the interventions offered different levels of support.
The appropriate response is to investigate these differences rather than immediately conclude that one study must be wrong.
A structured review should consider:
Consistency: Do the findings point in similar directions, and how strong is that agreement?
Context: Were the studies conducted in comparable institutional, economic and social environments?
Methodological quality: Are some findings supported by stronger designs or more reliable measurements?
Explanatory differences: Can variation in implementation, population characteristics or research methods account for the results?
Coverage: Which populations, outcomes or settings remain insufficiently studied?
Systematic reviews and meta-analyses can help synthesise a body of research when appropriate methods are used. However, combining studies does not eliminate weaknesses in the original evidence, and a pooled estimate may be misleading when important differences between studies are ignored.
The goal is to understand the overall body of evidence, including genuine disagreement and unresolved questions.
7. Recognise uncertainty and the limits of what can be concluded
Good evidence review makes uncertainty visible.
Some studies have limited sample sizes. Others lack suitable comparison groups, rely on incomplete records or measure outcomes over periods too short to capture meaningful change.
These limitations do not necessarily make the evidence useless. They affect the conclusions it can support.
A descriptive evaluation may establish that programme participants experienced improved outcomes over time, while leaving uncertainty about how much of the improvement resulted from the intervention.
Qualitative research may identify plausible explanations for low participation without establishing the prevalence of each barrier across a wider population.
Administrative data may reveal geographic differences in service utilisation but provide limited information about people who never accessed the service.
A useful evidence review distinguishes three things:
- What the evidence establishes: Findings directly supported by the available data and methods.
- What the evidence suggests: Plausible interpretations that are consistent with the findings but not definitively established.
- What remains unknown: Questions that require additional information or a stronger research design.
This distinction improves reporting and helps decision-makers avoid treating preliminary findings as definitive conclusions.
It also identifies where further research would be most useful.
8. Match the strength of the evidence to the decision
Not every decision requires the same level of evidence.
An organisation deciding how to improve the scheduling of its training sessions may reasonably draw on participant feedback, attendance records and implementation experience.
A government deciding whether to commit substantial public resources to expanding an intervention may need stronger evidence about effectiveness, costs, distributional consequences and implementation feasibility.
The consequences of a mistaken decision matter. So do the cost of obtaining additional evidence, the urgency of the decision and the possibility of revising the decision as new information becomes available.
A practical approach is to assess:
- How consequential is the decision?
- How strong is the current evidence?
- What uncertainties could materially change the decision?
- Would additional analysis or data collection reduce those uncertainties?
- What are the costs of delaying action?
- Can the decision be implemented incrementally and reviewed as evidence accumulates?
This approach avoids two common mistakes: acting with unjustified confidence when the evidence is weak, and delaying every decision until uncertainty has been eliminated.
Evidence-informed decision-making is not about waiting for perfect information. It is about making the best defensible decision with the evidence available while recognising its limitations.
9. A practical checklist for critical evidence review
Before relying on a report, dataset or evaluation, researchers and decision-makers can use the following checklist.
- Claim: What exactly does the evidence claim to establish?
- Relevance: Does it address the problem and decision under consideration?
- Source: Who produced the evidence, and are the methods transparent?
- Design: Is the research design appropriate for the question?
- Measurement: Are the main concepts and outcomes defined and measured credibly?
- Data quality: Are sampling, missing information and potential errors adequately addressed?
- Analysis: Are the analytical methods appropriate, and are important assumptions explained?
- Uncertainty: Are limitations and alternative explanations acknowledged?
- Consistency: How do the findings compare with other relevant evidence?
- Use: What conclusions and actions are justified by the evidence, and what further information is needed?
The checklist is not a substitute for specialist appraisal where technical expertise is required. It provides a structured starting point for asking better questions and identifying issues that may affect the reliability of a finding.
Conclusion: Evidence is valuable when its limits are understood
Research can support better decisions by revealing patterns, testing assumptions and helping organisations understand complex problems. But its value depends on more than the presence of data, the reputation of a source or the sophistication of an analytical technique.
Critical evidence review requires examining the claim being made, the methods used, the quality of the measurements, the uncertainty surrounding the findings and the extent to which conclusions are supported by the available information.
It also requires considering the wider body of evidence and the practical consequences of acting on a finding.
For researchers, evaluators, programme leaders and policymakers, the objective is not to find evidence that confirms an existing position. It is to develop the clearest possible understanding of what is known, what remains uncertain and what can reasonably be done next.
At DAREL Consulting, we connect research, evaluation, data analytics, economics and policy to help organisations assess evidence critically and translate it into informed decisions. Because good decisions require not only evidence, but also a clear understanding of what that evidence actually tells us.
