Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Writing · Appraising Systematic Reviews

Important Tips for Critically Appraising Systematic Reviews and Meta-Analyses Using AMSTAR 2 and ROBIS

AMSTAR 2, the most widely used critical-appraisal tool for systematic reviews, identifies seven specific weaknesses as ‘critical’ flaws capable of undermining the conclusions of a review on their own. It thereby provides reviewers and readers with a verifiable means of distinguishing a rigorous synthesis from one that is only superficially thorough. This article sets out those domains and the additional checks provided by ROBIS.

Pubrica AcademyWritten by a systematic review methodologist Reviewed bya research writing editor Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources10 peer-reviewed

Critical appraisal of a systematic review requires examination of its methods, not merely its conclusions, because the narrative summary of a review may appear authoritative irrespective of how rigorously the underlying search, selection and analysis were conducted. Appraisal of a systematic review or meta-analysis by a peer reviewer, editor or informed reader requires consideration of a small number of specific, verifiable methodological questions, most of which are now codified in standardized appraisal tools rather than left to individual judgment.

Learning objectives

  • The seven ‘critical domains’ used by AMSTAR 2 to identify reviews whose conclusions cannot be trusted
  • What ROBIS adds to AMSTAR 2, specifically with regard to bias in the review process itself
  • The practical checks that identify the most common correctable flaws in a submitted systematic review

Common misconceptions

  • “A systematic review that includes a large number of studies is necessarily more reliable than one that includes few.” AMSTAR 2 treats the comprehensiveness and documentation of the literature search, and the handling of risk of bias in the included studies, as critical domains independent of the number of studies ultimately included, so that a review of many poorly vetted studies may score worse than a review of fewer, carefully assessed ones.

The seven critical domains of AMSTAR 2

Beverley Shea and colleagues published AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews) in BMJ in 2017 as an updated 16-item critical-appraisal tool for systematic reviews that include randomized or non-randomized studies of healthcare interventions1. The tool designates seven of its sixteen items as critical domains: registration of a protocol before the review began, an adequate literature search, justification for the exclusion of individual studies, an acceptable method for assessing risk of bias in included studies, appropriate statistical methods for meta-analysis, consideration of risk of bias when interpreting results, and, where a meta-analysis was performed, an assessment of the likelihood of publication bias1. A weakness in even one critical domain is sufficient for AMSTAR 2 to rate the overall review as critically low confidence, irrespective of its performance on the remaining nine items; this feature distinguishes the scoring of AMSTAR 2 from a simple additive checklist1.

Additional checks provided by ROBIS

The ROBIS (Risk Of Bias In Systematic reviews) tool of Penny Whiting and colleagues, published in the Journal of Clinical Epidemiology in 2016, was developed specifically to assess risk of bias in the systematic review process itself, as distinct from the broader focus of AMSTAR 2 on overall methodological quality2. ROBIS comprises four domains (study eligibility criteria; identification and selection of studies; data collection and study appraisal; and synthesis and findings), each of which is assessed for the risk that the conduct of the review introduced bias into its conclusions, followed by a final judgment on the overall risk of bias in the review2. A reviewer applying ROBIS asks whether the authors of the review could, even inadvertently, have selected or synthesized evidence in a manner that favored a particular conclusion; this question is distinct from confirming that each methodological step was documented2.

Checks that identify the most common problems

In practice, a small number of checks identify most of the correctable flaws that reviewers encounter. First, the appraiser should confirm that the review followed a registered protocol (commonly in PROSPERO, the International Prospective Register of Systematic Reviews) and check whether the published methods correspond to those pre-registered, since unexplained deviations are a documented indicator of selective reporting3. Second, the search strategy itself should be verified: the PRISMA 2020 statement requires reporting of the full search strategy for at least one database in a form that could be rerun, and a review that summarizes its search only in prose cannot be adequately appraised for comprehensiveness4. Third, the appraiser should check that risk of bias in each included study was assessed with an appropriate, named tool (such as the Cochrane RoB 2 tool for randomized trials) rather than a generic or unstated method, and that this assessment informed how results were weighted or interpreted rather than being reported in a separate table unconnected to the conclusions of the review5. Finally, where a meta-analysis is included, the appraiser should check for an assessment of publication bias, commonly by means of a funnel plot or Egger's test; the original 1997 BMJ paper by Matthias Egger and colleagues that established this test warned that small-study effects and missing unpublished results can both produce funnel-plot asymmetry that mimics a genuine treatment effect if not interpreted carefully6.

Common errors made by appraisers

Sound appraisal of a systematic review requires avoidance of several specific errors that recur even among experienced reviewers and editors. The first is to treat the conclusion statement of a review as a substitute for direct examination of its methods; a confidently worded abstract conclusion carries no more weight than the rigor of the underlying search, selection and bias-assessment process, and AMSTAR 2 exists precisely because that rigor cannot be reliably inferred from the wording of the conclusion1. The second is to score a review additively rather than according to the critical-domain logic of AMSTAR 2; summing points across all sixteen items can produce an apparently respectable overall score for a review that has a single disqualifying weakness in one of the seven critical domains, which is the scoring error that the distinction between critical and non-critical items is designed to prevent1. The third is to end the appraisal at the search and selection stage without checking whether risk-of-bias findings were carried through into the interpretation of the results, a gap that ROBIS is designed to identify in its fourth domain (synthesis and findings) and that a reviewer who checks AMSTAR 2 items in isolation, rather than considering how they relate to one another, may miss2. A short standing worksheet based on the seven critical domains of AMSTAR 2 and the four process domains of ROBIS, applied consistently to every review appraised rather than reconstructed from memory on each occasion, is the most reliable documented safeguard against all three of these recurring errors.

Consistency between the discussion and the results

An important and often under-weighted step in appraising a systematic review is the direct comparison of its discussion and conclusion sections with its own reported results, rather than an assumption that the two necessarily align because they appear in the same document. The AMSTAR 2 critical domain on interpreting results in light of risk of bias exists because a review may report a candid, well-conducted risk-of-bias assessment in its results section and then disregard it when drawing conclusions, describing a pooled effect in terms such as clear or convincing that the underlying bias assessment does not support1. A careful appraiser checks whether the certainty language used in the abstract and conclusion (terms such as “demonstrates”, “proves” or “confirms”, as opposed to more qualified terms such as “suggests” or “is consistent with”) matches the certainty established by the review's own GRADE (Grading of Recommendations Assessment, Development and Evaluation) or risk-of-bias assessment, since such a mismatch is one of the more common ways in which the findings of a technically well-conducted review are overstated in precisely the passages that most readers, and most secondary sources citing the review, will read. This check requires only a few minutes once the risk-of-bias and heterogeneity findings have been extracted, and it addresses a category of problem distinct from those covered by the structural checklist items of AMSTAR 2 and ROBIS, both of which are concerned primarily with the methods of the review rather than the rhetorical precision of its conclusions.

The abstract as the subject of a separate final check

The abstract of a systematic review is, in practice, the only section that most of its eventual readers (clinicians scanning search results, journalists and researchers citing it at second hand) will read, which justifies appraising it as a distinct final step even after the full review has been checked against AMSTAR 2 and ROBIS. The PRISMA 2020 statement includes a dedicated abstract checklist because of this disproportionate influence, requiring that the abstract report the objectives of the review, data sources, eligibility criteria, a summary of the risk-of-bias assessment, the number of included studies and participants, the main results with their precision, and any important limitations, rather than the headline finding alone3. The final step for an appraiser is therefore to read the abstract in isolation, as though it were the only part of the paper available, and to consider whether a reader relying on it alone would form a materially different impression of the certainty of the evidence than a reader of the full risk-of-bias and heterogeneity findings; a meaningful discrepancy between these impressions is a finding that warrants reporting in its own right, independent of how well the underlying methods otherwise scored.

Key points

Key points

Appraisal of a systematic review involves checking a small set of specific methodological items, namely protocol registration, comprehensiveness of the search, risk-of-bias assessment and assessment of publication bias, because a weakness in any one of these can undermine the conclusions of the review regardless of the number of studies included.

Practical application

Practical application

  1. Every systematic review under appraisal should first be assessed against the seven critical domains of AMSTAR 2. These domains should be checked before any other item, since a critical weakness in a single domain determines the overall confidence rating irrespective of other strengths.
  2. The registered protocol should be compared with the published methods. The PROSPERO (or equivalent) registration should be consulted and compared with what was reported; unexplained discrepancies are a documented indicator of selective reporting.
  3. Appraisers should check whether risk-of-bias findings informed the conclusions of the review. A risk-of-bias table that is reported but never referred to in the interpretation of pooled results is a common and correctable deficiency.

Frequently asked questions

AMSTAR 2 assesses the overall methodological quality of a systematic review across 16 items, seven of them critical; ROBIS focuses specifically on risk of bias introduced by the review's own conduct across four domains, a narrower and more process-focused lens.

Because each of the seven critical domains (protocol registration, search adequacy, exclusion justification, risk-of-bias assessment, statistical method appropriateness, interpretation accounting for bias, and publication-bias assessment) addresses a way in which the core conclusions of the review could be fundamentally unreliable, and non-critical items cannot offset such a weakness.

The appraiser should confirm that the full, rerunnable search strategy for at least one database is reported, as PRISMA 2020 requires; a prose summary alone is insufficient to judge comprehensiveness.

No. The original paper by Egger and colleagues cautions that funnel-plot asymmetry can also result from genuine small-study effects unrelated to publication bias; the test therefore identifies a pattern that warrants investigation rather than proving bias on its own.

Yes. A rigorous appraisal checks not only whether risk of bias was assessed but also whether that assessment was used to weight or qualify the interpretation of the pooled results.

References

  1. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. read 10 October 2026. View on BMJ
  2. Whiting P, Savović J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-234. read 10 October 2026. View on Journal of Clinical Epidemiology
  3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on BMJ
  4. Sterne JAC, Savović J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. read 10 October 2026. View on BMJ
  5. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634. read 10 October 2026. View on BMJ
  6. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. read 10 October 2026. View on BMJ
  7. Whiting P, Savović J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225-234. read 10 October 2026. View on Journal of Clinical Epidemiology
  8. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on BMJ
  9. Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on Cochrane Training
  10. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634. read 10 October 2026. View on BMJ

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Important Tips for Critically Appraising Systematic Reviews and Meta-Analyses Using AMSTAR 2 and ROBIS. Pubrica. https://pubrica.com/academy/study-guide/research-writing/guide-reviewing-systematic-reviews-meta-analyses/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Systematic Review Appraisal Worksheet

For use in the critical appraisal of a systematic review or meta-analysis.

[ ] Confirmed a registered protocol exists and matches the published methods
[ ] Verified the full search strategy for at least one database is reported and rerunnable
[ ] Checked that exclusion of individual studies was justified and documented
[ ] Confirmed risk of bias in included studies was assessed with a named, appropriate tool
[ ] Checked whether risk-of-bias findings informed the interpretation of pooled results
[ ] If a meta-analysis is included, checked for a publication-bias assessment (funnel plot/Egger's test)
[ ] Scored the review using AMSTAR 2's seven critical domains before any other item

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s systematic review support team applies AMSTAR 2 and ROBIS directly to draft and submitted reviews, identifying weaknesses in critical domains before they are raised by a journal reviewer.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Systematic reviews assessed against the seven critical domains of AMSTAR 2

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.