Home/Insights/Methodology reference/Quasi-experimental design: disadvantages
Quasi-experimental design: disadvantages
Every limitation reduces to one thing: the groups were not formed at random, so any difference between them may be the reason for the result.
Confounding that cannot be removed
Adjustment, matching and propensity methods handle confounders that were measured. Unmeasured and unmeasurable confounders remain, and no statistical technique recovers them.
Propensity score methods are frequently described in a way that implies otherwise. They balance the covariates entered into the model and nothing else.
Selection, and why it is hard to see
Who received the intervention was determined by something, and that something is often related to the outcome. Confounding by indication is the clinical case; in policy evaluation, areas that adopt an intervention first typically differ from those that adopt it late.
Threats to internal validity
History. A co-occurring change produces the same pattern as an effect.
Maturation. The outcome was already trending.
Regression to the mean. Intervening after an extreme value guarantees apparent improvement.
Testing and instrumentation. Measurement itself changes, or the definition of the outcome changes mid-series — a coding revision will look exactly like an intervention effect.
Analytical fragility
Results often depend on model specification, on the comparison group chosen and on the time window. Where the estimate moves substantially under reasonable alternative choices, that instability is itself a finding and should be reported rather than resolved by selecting the preferred specification.
Sensitivity analysis, and quantitative bias analysis showing how strong an unmeasured confounder would need to be, are the honest responses.
Consequence for evidence synthesis
Systematic reviews commonly rate these designs at higher risk of bias, and GRADE starts observational evidence at low certainty. A quasi-experimental result therefore carries less weight in a guideline, irrespective of how large it is.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Never state: that adjustment or propensity scoring controls for confounding.
- Report: sensitivity to model specification and comparison group.
- Expect: lower certainty ratings in synthesis, and say so in the discussion.
Reporting standards
- ROBINS-I, for risk of bias assessment
- GRADE Working Group, on certainty of observational evidence
Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.
Related