Which Risk of Bias Tool? A Decision Tree for RoB 2, ROBINS-I, Newcastle–Ottawa, QUADAS-3 and JBI Tools

Risk of bias assessment in a systematic review judges whether design or conduct could distort a result. Before choosing a risk of bias tool, define population, intervention or exposure, comparator, outcome, follow-up and the estimand: the specific effect the review aims to estimate.

The question type decides the framework. An intervention question asks whether a treatment changes an outcome; an exposure question asks about the causal effect of an exposure; a prognostic-factor question examines association with a future outcome; a prediction question asks how accurately a model estimates individual risk; a diagnostic question asks how well a test identifies a condition against a reference standard. So the same cohort study may need ROBINS-I for a medicine’s causal effect, QUIPS for a biomarker’s prognostic value or PROBAST+AI for a prediction model. In a trial, the effect of assignment can differ from the effect of adhering to treatment, and RoB 2 must match the effect and analysis you intend [1].

Step 1: What is the target of inference?

• Causal intervention effect: RoB 2 for randomized trials [1]; ROBINS-I for non-randomized intervention studies [2].
• Causal exposure effect: ROBINS-E for non-randomized exposure studies [7].
• Diagnostic accuracy: QUADAS-3 for new assessments [3], or QUADAS-2 where an existing protocol justifies it [4].
• Prognostic-factor association: QUIPS [8].
• Prediction-model assessment (development or evaluation): PROBAST+AI [5].
• Other eligible questions: a design-specific JBI critical appraisal checklist [10] or the Newcastle–Ottawa Scale [9], where their scope fits.

Step 2: Define the assessment unit: the outcome, comparison and particular result or estimate being judged.

Step 3: Fix the version and rules: prespecify the tool version, judgment rules and how disagreements are resolved. If a review answers several questions, give each evidence stream its own framework.

A popular checklist is not automatically appropriate, and more detailed appraisal cannot fix a mismatch between tool and question.

Randomized trials. RoB 2 covers five domains: randomization, deviations from intended interventions, missing outcome data, outcome measurement and selection of the reported result [1]. Judgments apply to a result, not a publication; use the cluster and crossover variants where relevant. See our RoB 2 explainer.

ROBINS-I or ROBINS-E? ROBINS-I is for intervention effects, ROBINS-E for causal exposure effects. Both demand explicit handling of confounding, so prespecify important confounders and the intended comparison. The original ROBINS-I dates from 2016 [2]; Version 2, first released in 2024 and revised in November 2025, adds judgment algorithms and addresses immortal-time bias, but remains a draft [11]. Record which version you used and never mix guidance across versions.

QUADAS-2 or QUADAS-3? QUADAS-3, published in 2026, assesses four domains (Participants, Index Test, Target Condition and Analysis), with applicability judged for the first three, across six phases from defining the review question to the overall judgment [3]. Its key change is estimate-level assessment: a study reporting sensitivity and specificity at several thresholds may carry different risk of bias for each estimate. QUADAS-2 [4] remains valid for existing protocols that justify it; document any switch.

QUIPS or PROBAST+AI? QUIPS judges prognostic-factor associations [8]. PROBAST+AI separates model development from evaluation of predictive performance and covers regression and machine-learning prediction models [5].

NOS or JBI? NOS can suit selected cohort and case-control reviews, but a total star score hides domain-level weaknesses [9]; see our NOS guide. JBI tools give design-specific checklists [10]. Neither replaces a framework built for causal effects, diagnostic accuracy or prediction.

• Scope: does the tool match the question and design?
• Assessment unit: is the judgment about a study, result, estimate or model?
• Version: is the version appropriate and recorded?
• Evidence: is each judgment backed by reported information and stated reasoning?
• Synthesis: is the use of judgments in the analysis prespecified?

Poor reporting does not prove bias, though it may prevent a confident judgment. Applicability, whether the evidence matches your question, is a separate issue from internal risk of bias.

Risk-of-bias judgments should shape interpretation, not trigger automatic numerical corrections. Suppose a pooled risk ratio of 0.80 (95% CI 0.72–0.89) becomes 0.91 (95% CI 0.80–1.04) when high-risk studies are excluded. That shows sensitivity to study selection; it does not prove the excluded studies were biased or that 0.91 is the true effect. Examine study weights, confounding control and outcome definitions. A prespecified sensitivity analysis tests robustness but cannot identify the mechanism or direction of bias. Carry the judgments into the certainty of evidence alongside inconsistency, indirectness, imprecision and publication bias [6].