Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one

Home/Insights/Methodology reference/Quasi-experimental design: disadvantages

Quasi-experimental design: disadvantages

Every limitation reduces to one thing: the groups were not formed at random, so any difference between them may be the reason for the result.

Experimental and quasi-experimental

Confounding that cannot be removed

Adjustment, matching and propensity methods handle confounders that were measured. Unmeasured and unmeasurable confounders remain, and no statistical technique recovers them.

Propensity score methods are frequently described in a way that implies otherwise. They balance the covariates entered into the model and nothing else.

Selection, and why it is hard to see

Who received the intervention was determined by something, and that something is often related to the outcome. Confounding by indication is the clinical case; in policy evaluation, areas that adopt an intervention first typically differ from those that adopt it late.

Threats to internal validity

History. A co-occurring change produces the same pattern as an effect.

Maturation. The outcome was already trending.

Regression to the mean. Intervening after an extreme value guarantees apparent improvement.

Testing and instrumentation. Measurement itself changes, or the definition of the outcome changes mid-series — a coding revision will look exactly like an intervention effect.

Analytical fragility

Results often depend on model specification, on the comparison group chosen and on the time window. Where the estimate moves substantially under reasonable alternative choices, that instability is itself a finding and should be reported rather than resolved by selecting the preferred specification.

Sensitivity analysis, and quantitative bias analysis showing how strong an unmeasured confounder would need to be, are the honest responses.

Consequence for evidence synthesis

Systematic reviews commonly rate these designs at higher risk of bias, and GRADE starts observational evidence at low certainty. A quasi-experimental result therefore carries less weight in a guideline, irrespective of how large it is.

In practice

Fix, flag or query.

The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.

  • Never state: that adjustment or propensity scoring controls for confounding.
  • Report: sensitivity to model specification and comparison group.
  • Expect: lower certainty ratings in synthesis, and say so in the discussion.

Reporting standards

  • ROBINS-I, for risk of bias assessment
  • GRADE Working Group, on certainty of observational evidence

Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.