Home/Insights/Evidence synthesis/Evaluating risk of bias in systematic reviews of interventions
Evaluating risk of bias in systematic reviews of interventions
Risk of bias is judged per outcome, with a named tool matched to the design. A generic "quality score" is not an assessment.
Why scales were abandoned
Early reviews used summary quality scores — adding points for randomisation, blinding, follow-up and so on. These were shown to be unreliable: different scales applied to the same trials produced different and sometimes opposite conclusions, because the weighting of items was arbitrary.
Current practice assesses specific domains of bias and reaches a judgement for each, without summing them into a number.
The tools, and when each applies
RoB 2 for randomised trials. Domains: the randomisation process; deviations from intended interventions; missing outcome data; measurement of the outcome; selection of the reported result. Each is rated low risk, some concerns, or high risk, with an overall judgement following stated rules.
ROBINS-I for non-randomised studies of interventions. It adds pre-intervention domains — confounding and selection of participants — and is judged against a hypothetical target trial, which is what makes it comparable with randomised evidence.
QUADAS-2 for diagnostic accuracy studies, assessing both risk of bias and applicability across patient selection, index test, reference standard, and flow and timing.
ROBIS for assessing risk of bias in a systematic review itself, which is what an overview of reviews needs.
The two things most often done wrong
Assessing per study rather than per outcome. Blinding may be irrelevant to mortality and critical to a patient-reported outcome. One judgement for the whole study conceals that.
Treating the assessment as decorative. A risk of bias table that appears in the results and is never mentioned again has not been used. It should drive sensitivity analysis — does the conclusion hold if high-risk studies are excluded? — and feed the GRADE rating.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Never: use a summary quality score.
- Assess per outcome, with the tool matched to the design.
- Use the result: sensitivity analysis excluding high-risk studies.
Sources
- Cochrane RoB 2, ROBINS-I and ROBIS tools
- QUADAS-2, for diagnostic accuracy studies
- GRADE Working Group, on risk of bias as a certainty domain
Sources are given so each claim can be checked at its origin. Where a tool’s current maintenance status could not be confirmed, the page says to check it rather than asserting it.
Related