Home/Insights/Methodology reference/Impact study methodology
Impact study methodology
Impact evaluation asks what would have happened otherwise. Without a credible counterfactual, it measures change and calls it effect.
The central problem
Impact is the difference between the outcome observed and the outcome that would have occurred without the intervention. The second is never observed, so every impact evaluation is an argument about how it was estimated.
A before-and-after comparison is not an impact evaluation. It confuses change over time with the effect of the intervention, and will attribute to a programme anything else that happened in the same period.
Constructing the counterfactual
Randomised assignment where it is possible. Otherwise the quasi-experimental toolkit: difference-in-differences against a comparison group, regression discontinuity around an eligibility threshold, interrupted time series with a well-established pre-trend, or a synthetic control constructed from a weighted combination of unexposed units.
Each rests on an assumption that should be stated and, where possible, tested.
Attribution and contribution
Where a counterfactual cannot be constructed, the honest claim is contribution rather than attribution: the intervention plausibly contributed, alongside other factors, and the analysis sets out why.
Theory-based evaluation — a stated theory of change, with the causal chain from inputs through outputs to outcomes made explicit and each link examined — is the recognised approach. It is weaker than a counterfactual design and considerably stronger than asserting attribution without one.
Practical requirements
Baseline data, collected before the intervention. Outcomes defined in advance, because impact evaluations are vulnerable to outcome switching when the primary measure disappoints. An adequate follow-up period, since effects on outcomes such as mortality or employment take years to appear. Attention to spillover, where the comparison group is affected too, which biases the estimate toward zero.
Cost should be reported alongside effect. An impact evaluation that establishes benefit without cost gives a decision-maker half of what they need.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Not an impact evaluation: before-and-after measurement with no comparison.
- State: the counterfactual and the assumption it depends on.
- Claim contribution, not attribution, where no counterfactual exists.
Reporting standards
- Difference-in-differences and synthetic control methodology literature
- TREND Statement, for non-randomised intervention evaluation
Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.
Related