Home/Insights/Methodology reference/Comparative study methodology
Comparative study methodology
Two or more groups compared without random assignment. Everything depends on how unlike the groups were to begin with.
The design
Comparative studies set groups against each other — treated against untreated, exposed against unexposed, one service model against another — where the grouping was determined by something other than randomisation.
They are common because randomisation is often impossible: you cannot randomise people to a risk factor, an occupation or a country.
Baseline comparability is the whole argument
If the groups differ systematically at baseline, any outcome difference may reflect that rather than the exposure. Confounding by indication is the clinical version: sicker patients receive the more aggressive treatment, which then appears to produce worse outcomes.
Report baseline characteristics by group. Note that a significance test on baseline differences is uninformative in a non-randomised study — the groups genuinely differ, and what matters is the size of the imbalance in variables related to the outcome, not whether it reaches a threshold.
What can be done about it
Matching on measured confounders, with the matching variables and tolerances stated.
Regression adjustment, which assumes the model form is right.
Propensity scores — matching, weighting or stratifying on the modelled probability of exposure. These balance measured covariates only; they do nothing for unmeasured ones, a point frequently lost in write-ups.
Instrumental variables, where a valid instrument exists, which is rare and requires defending.
None of these converts an observational comparison into a trial. A sensitivity analysis showing how strong an unmeasured confounder would need to be to explain the result is more honest than a claim of adjustment.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Report: baseline characteristics by group, with the magnitude of imbalance.
- Do not: significance-test baseline differences in a non-randomised comparison.
- State plainly: propensity methods balance measured covariates only.
Reporting standards
- STROBE Statement, for observational comparisons
- ROBINS-I, for risk of bias in non-randomised intervention studies
Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.
Related