Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Risk of Bias Assessment

Understanding Risk of Bias in Systematic Reviews and Meta-Analysis: RoB 2, ROBINS-I and GRADE

Cochrane replaced its original risk-of-bias tool, cited more than 40,000 times, after empirical studies found only moderate agreement between reviewers using it. The 2019 revision, RoB 2, assesses bias one outcome at a time rather than rating an entire trial as a single unit.

Pubrica AcademyWritten by a systematic review methodologist Reviewed bya research data analyst Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources10 peer-reviewed

Risk of bias, in a systematic review or meta-analysis, is a structured judgment of how likely a specific study's result is to be systematically distorted away from the true effect by flaws in its design or conduct, such as inadequate randomization, unblinded outcome assessment or selective reporting of results. It is distinct from a study's overall “quality” in a general sense, because a well-conducted study can still carry a specific, identifiable risk of bias in one domain while being strong in others, and a review should assess and report each of those domains rather than collapsing them into a single undifferentiated quality score. The Cochrane Collaboration's tools for this task are the most widely used in biomedicine, and the 2019 revision of its principal tool for randomized trials, RoB 2, changed both the structure of the assessment and the standard applied to it.

Learning objectives

  • Why Cochrane revised its original risk-of-bias tool after nearly a decade of use
  • The five fixed domains assessed by RoB 2 and the three judgments available for each
  • Why risk of bias is assessed for each outcome rather than once for an entire study

Common misconceptions

  • “A single study either has a low risk of bias overall or it does not.” RoB 2 assesses risk of bias separately for each reported result (one outcome, one comparison) through five fixed domains, each judged independently as low risk, some concerns or high risk; a single study can therefore carry different risk-of-bias judgments for the different outcomes it reports.

Reasons for the replacement of the original Cochrane tool

Cochrane's original risk-of-bias tool, published in 2011, had accumulated more than 40,000 citations and nearly a decade of routine use across systematic reviews before it was revised3. Despite that widespread adoption, empirical studies of the original tool found only moderate reliability between different reviewers' risk-of-bias judgments on the same studies, with “unclear” ratings common and few trials rated as low risk across any domain, a pattern that prompted the Cochrane Methods group to begin a structured revision process1. The revision proceeded from an August 2015 planning meeting through pilot testing in 2016, with a full draft posted in October 2016, followed by three further years of training and field feedback before the final RoB 2 tool was published in the BMJ in 20191. Jørgensen and colleagues, in a separate review of how the original tool was used and discussed, documented a substantial body of published commentary indicating inconsistent application across both Cochrane and non-Cochrane reviews, which reinforced the view that the original tool's instructions permitted excessive divergence in judgment between reviewers assessing the same evidence6.

The five domains assessed by RoB 2

RoB 2 structures its assessment around fixed bias domains covering the randomization process, deviations from the intended intervention, missing outcome data, measurement of the outcome, and selection of the reported result, each addressed through a series of signalling questions that map, via a specified algorithm, onto one of three possible domain-level judgments: low risk of bias, some concerns, or high risk of bias1. A study's overall risk-of-bias judgment for a given result is generally determined by its worst-performing domain, although several domains rated “some concerns” can together justify an overall “high risk” judgment, which prevents the overall rating from averaging away a serious problem in one specific area1. Structurally, RoB 2 replaced the original tool's terms “selection bias” and “performance bias” with cause-based domain labels, removed the “unclear” option so that reviewers must reach one of the three substantive judgments, and explicitly distinguishes the effect of assignment to an intervention (intention-to-treat) from the effect of adhering to it (per-protocol), treating these as two different questions that a trial might address1. It also ceased to treat every unblinded trial as automatically at high risk, assessing instead whether the absence of blinding plausibly affected the specific outcome and domain in question1. A separate validation study by Minozzi and colleagues applying RoB 2 in practice found that interrater reliability remained a substantial challenge under the revised tool, which cautions against assuming that any risk-of-bias tool, including the current one, removes the need for well-trained, calibrated assessors7.

Assessment of bias in non-randomized studies: ROBINS-I

Because randomization is the design feature that most directly protects against confounding, non-randomized studies of interventions are subject to a structurally different and generally larger set of bias risks; for this reason, Cochrane maintains a separate tool, ROBINS-I (Risk Of Bias In Non-randomized Studies of Interventions), for this study type rather than adapting RoB 2 to cover it2. ROBINS-I assesses bias arising from confounding and from the selection of participants into the study, in addition to the domains it shares conceptually with RoB 2, such as deviations from intended interventions, missing data, outcome measurement, and selective reporting2. Its judgment categories also use a different scale from RoB 2's three-tier system, ranging from low through moderate and serious to critical risk of bias. This reflects the fact that non-randomized evidence routinely carries bias risks that a randomized trial does not, and that collapsing those risks onto the three categories used for randomized trials would understate the difference in how the two types of evidence should be interpreted2. A systematic review drawing on both randomized and non-randomized evidence should apply RoB 2 and ROBINS-I to the respective study types separately and should not present their resulting judgments as directly equivalent on a single combined scale. Both tools should also be distinguished from ROBIS, a separate instrument designed to assess risk of bias in the systematic review itself (its search, selection and synthesis methods) rather than in any individual study the review includes. A review can score well on ROBIS while still containing individual studies rated at high risk on RoB 2 or ROBINS-I, because the two assessments address different questions about where bias may enter the overall body of evidence5.

From risk of bias to certainty of the evidence

A risk-of-bias judgment on an individual study is an input into, and not equivalent to, the overall certainty rating that a review assigns to a body of evidence for a given outcome. The GRADE (Grading of Recommendations Assessment, Development and Evaluation) framework, developed by the GRADE Working Group, uses risk of bias across the contributing studies as one of several domains, alongside inconsistency, indirectness, imprecision and publication bias, that can lower the certainty rating of a body of evidence from high to moderate, low or very low4. An empirical study by Savović and colleagues, examining how reported design characteristics related to intervention effect estimates across hundreds of trials, found that inadequate randomization and lack of blinding were associated with measurably different, typically exaggerated, effect estimates compared with trials that handled those domains well. This finding provides the direct empirical justification for incorporating risk of bias into a GRADE certainty rating rather than treating it as a procedural formality with no bearing on how a result should be interpreted8. PRISMA 2020 in turn requires that a review report both the risk-of-bias assessment method used and, separately, the method used to assess certainty in the body of evidence, treating these as two distinct, both-mandatory reporting items9.

Common errors in the application of these tools

Minozzi and colleagues' validation study of RoB 2 in real-world application did not merely report a low interrater reliability figure; it documented specific, recurring sources of disagreement between assessors, including differing interpretations of the wording of the signalling questions and inconsistent judgments about how far a given deviation from ideal practice in a domain should move the overall rating from “some concerns” to “high risk”7. A recurring error identified by the study and related commentary is that assessors rely on a study's reported methods section rather than checking the full trial registration, protocol or statistical analysis plan where one is available. This matters because RoB 2's signalling questions on domains such as selective reporting are designed to be answered by comparing what a trial pre-registered with what it ultimately reported, not by reading the published methods section in isolation and accepting its description of what was intended at face value17. A second common error is applying RoB 2's three-tier judgment scale to a non-randomized study, or ROBINS-I's four-tier scale to a randomized trial, rather than confirming a study's actual design before selecting the tool. This elementary sequencing error occurs often enough in practice that Cochrane's guidance explicitly identifies confirmation of study design as the first step before any domain-level assessment begins210.

A third documented error is treating risk-of-bias assessment as a single pass that is never revisited, rather than as a judgment to be reconsidered if new information about a study, such as a later-published amendment to the trial registration or a correction notice, emerges during the review. Because RoB 2 assessments are intended to be made per outcome and per result rather than once per study, a review team that assesses risk of bias only for a trial's primary outcome and then applies that judgment uncritically to all of the trial's secondary outcomes is taking precisely the outcome-level shortcut that the tool's structure was designed to prevent1. Training assessors on these recurring failure points (checking pre-registration documents rather than relying on the methods section alone, confirming study design before selecting a tool, and reassessing at the level of the individual outcome rather than the whole study) may address a meaningful share of the interrater disagreement that Minozzi and colleagues documented, although their study indicates that disagreement is unlikely to be eliminated entirely under current tools.

Key points

Key points

Risk of bias is assessed for each outcome through fixed domains rather than as a single overall label for an entire study, and the tool used (RoB 2 for randomized trials, ROBINS-I for non-randomized studies) must match the design of the study being assessed.

Practical application

Practical application

  1. The risk-of-bias tool should be matched to the study design rather than the review topic. RoB 2 should be used for randomized trials and ROBINS-I for non-randomized studies, and their resulting judgments should be kept on separate, non-equivalent scales when both appear in the same review.
  2. Risk of bias should be assessed for each outcome rather than once per study. A separate domain-by-domain assessment should be conducted for each outcome and comparison that a study reports, since the same study can carry different risk-of-bias judgments for different results.
  3. Risk-of-bias judgments should inform a GRADE certainty rating rather than ending at the study level. Both the risk-of-bias method and the certainty-rating method should be reported explicitly, as PRISMA 2020 requires, so that readers can see how study-level bias was translated into the review’s overall confidence in its findings.

Frequently asked questions

They are different, related tools: RoB 2 judges risk of bias in an individual study's specific result, while GRADE uses that judgment, combined with inconsistency, indirectness, imprecision, and publication-bias assessments across all contributing studies, to rate the overall certainty of a body of evidence for an outcome.

The revision process found that 'unclear' was used too often and too inconsistently across reviewers to be informative, so RoB 2 requires assessors to use the available signalling questions and algorithm to reach one of three substantive judgments instead.

No. RoB 2 specifically moved away from treating lack of blinding as automatically high risk, instead requiring reviewers to assess whether the absence of blinding plausibly affected the specific outcome and domain being judged.

Yes, and this is a deliberate feature of RoB 2's design: because selective reporting and other biases can affect specific outcomes differently, the tool assesses risk of bias separately for each result a study reports rather than issuing one blanket judgment.

No. A validation study applying RoB 2 in practice found interrater reliability remained a genuine challenge even with the revised tool, which is why well-trained, calibrated assessors and documented disagreement-resolution procedures remain necessary.

References

  1. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. read 10 October 2026. View on The BMJ
  2. Sterne JAC, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. read 10 October 2026. View on The BMJ
  3. Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. read 10 October 2026. View on The BMJ
  4. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. read 10 October 2026. View on The BMJ
  5. Whiting P, Savović J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology. 2016;69:225-234. read 10 October 2026. View on Journal of Clinical Epidemiology
  6. Jørgensen L, Paludan-Müller AS, Laursen DRT, et al. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials: overview of published comments and analysis of user practice in Cochrane and non-Cochrane reviews. Systematic Reviews. 2016;5:80. read 10 October 2026. View on Systematic Reviews
  7. Minozzi S, Cinquini M, Gianola S, Gonzalez-Lorenzo M, Banzi R. The revised Cochrane risk of bias tool for randomized trials (RoB 2) showed low interrater reliability and challenges in its application. Journal of Clinical Epidemiology. 2020;126:37-44. read 10 October 2026. View on Journal of Clinical Epidemiology
  8. Savović J, Jones H, Altman D, et al. Influence of reported study design characteristics on intervention effect estimates from randomized controlled trials. Annals of Internal Medicine. 2012;157(6):429-438. read 10 October 2026. View on Annals of Internal Medicine
  9. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on The BMJ
  10. Higgins JPT, Thomas J, Chandler J, et al (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.4, Chapter 8: Assessing risk of bias in a randomized trial. Cochrane, 2023. read 10 October 2026. View on Cochrane Training

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Understanding Risk of Bias in Systematic Reviews and Meta-Analysis: RoB 2, ROBINS-I and GRADE. Pubrica. https://pubrica.com/academy/risk-of-bias-in-systematic-review-and-meta-analysis/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Risk-of-Bias Assessment Setup Checklist

These items should be confirmed before risk of bias is assessed in a systematic review.

[ ] Confirmed RoB 2 is used for randomized trials and ROBINS-I for non-randomized studies, not mixed on one scale
[ ] Set up the assessment to run per outcome/comparison, not once per study
[ ] Trained and calibrated at least two independent assessors on the chosen tool
[ ] Pre-specified how domain-level disagreements will be resolved
[ ] Linked risk-of-bias judgments explicitly to the GRADE certainty-of-evidence rating
[ ] Reported both the risk-of-bias method and the certainty-rating method per PRISMA 2020

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s biostatistics and evidence synthesis team applies RoB 2 and ROBINS-I with calibrated, independent assessors and links the resulting judgments to GRADE certainty ratings, so that a review’s risk-of-bias assessment withstands journal and reviewer scrutiny.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Pubrica can review a risk-of-bias assessment to confirm that it applies the appropriate tool to the appropriate outcome.

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.