Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Newcastle-Ottawa Scale

Assessing Study Quality in Meta-Analysis Using the Newcastle-Ottawa Scale (NOS)

The Newcastle-Ottawa Scale scores cohort and case-control studies out of nine stars across the domains of selection, comparability and outcome. However, a methodological critique of the tool identified substantive limitations in its reliability, which reviewers should consider before accepting a score at face value.

Pubrica AcademyWritten by an evidence synthesis methodologist Reviewed byan epidemiology reviewer Published10 October 2026 Last checked10 October 2026 Reading time9 minutes Sources10 peer-reviewed

The Newcastle-Ottawa Scale (NOS) is a widely used tool for judging the methodological quality of non-randomized studies, specifically cohort and case-control designs, for inclusion in a systematic review or meta-analysis. It scores each study against eight items grouped into three domains (selection, comparability, and outcome or, for case-control studies, exposure), awarding one star per criterion met, to a maximum of nine stars. It is one of the most frequently cited quality-assessment tools in the systematic review literature; according to a direct methodological critique of the scale, it also has documented limitations of which a careful reviewer should be aware rather than applying it uncritically.

Learning objectives

  • The structure of the eight NOS items across its three domains
  • The information the NOS does and does not provide about overall study quality
  • The specific limitations identified in a published critical evaluation of the scale
  • The comparison between the NOS and ROBINS-I, and the circumstances in which each is appropriate

Common misconceptions

  • “A high Newcastle-Ottawa Scale score is a reliable, validated certification that a study is high quality.” Stang’s published critical evaluation of the Newcastle-Ottawa Scale documents substantive limitations in its inter-rater reliability and construct validity, and the scale’s own documentation states that it does not provide strict thresholds for study quality. A star count should therefore inform a reviewer’s judgement rather than substitute for it.

Structure of the eight NOS items

For cohort studies, the Selection domain covers four items: whether the exposed cohort is truly representative of the population with the risk factor under study, whether the non-exposed cohort is drawn from the same underlying population, how exposure was ascertained (a secure record versus a self-report, for instance), and whether the outcome of interest was confirmed to be absent at the start of the study1. The Comparability domain is a single item that can award up to two stars, assessing whether the study design or its statistical analysis controlled for the most important confounding factors relevant to the research question1. The Outcome domain covers three items: how the outcome was assessed, whether follow-up was long enough for the outcome to plausibly occur, and whether the follow-up rate was adequate to limit attrition bias1. Case-control studies use a parallel structure, substituting an Exposure domain for the Outcome domain, reflecting the different direction of inference in that design1.

Use of NOS scores and their limitations

A study earns one star for each criterion it satisfies, up to a maximum of nine (the Comparability item alone can contribute up to two), and the resulting star count is commonly used in two ways within a meta-analysis: to inform sensitivity analyses that test whether excluding lower-scoring studies alters the pooled result, or to weight or stratify the analysis by study quality1. The scale’s documentation states explicitly that it “does not provide strict cutoffs for study quality,” meaning that there is no officially sanctioned threshold, such as “seven stars or more counts as high quality,” and reviewers who impose their own cutoff should state and justify it rather than presenting it as a feature of the tool1. A star total is also a summary rather than a diagnosis: two studies with the same overall score can have very different specific weaknesses, since the total does not indicate which of the eight items a lower-scoring study failed.

Findings of a methodological critique of the scale

Stang’s published critical evaluation of the Newcastle-Ottawa Scale, which appeared in the European Journal of Epidemiology, raises specific, documented concerns about the tool’s reliability and construct validity2. The issues raised include ambiguity in how certain items should be interpreted and applied consistently by different raters, which affects inter-rater reliability; the concern that assigning equal weight (one star) to items of differing importance to overall risk of bias may not accurately reflect each item’s actual effect on a study’s validity; and broader questions about whether the scale’s total score adequately captures what “quality” should mean for the purpose for which a reviewer is using it2. These are not reasons to discard the tool, but they are reasons to treat a single star count with the caution that any single summary statistic warrants, particularly when it will directly influence which studies are excluded from a meta-analysis or how heavily they are weighted.

Comparison of the NOS with ROBINS-I

ROBINS-I, developed by Sterne and colleagues, offers a more structured, domain-based alternative designed specifically for assessing risk of bias in non-randomized studies of interventions, organizing its assessment around seven distinct bias domains, including confounding and selection of participants, each of which requires detailed responses to signaling questions rather than a single star3. This generally makes ROBINS-I more time-consuming to apply than the NOS, but it also produces a more granular, traceable judgment about where and why a study’s risk of bias is high, moderate, or low, which directly addresses Stang’s concern that the aggregate star total of the NOS obscures which specific domain produced a low score23. ROBINS-I is generally the more rigorous choice specifically for non-randomized studies evaluating the effect of an intervention; the NOS remains a reasonable, faster choice for broader quality screening of cohort and case-control studies where that level of granularity is not a priority for the review.

Application of the NOS in systematic reviews

The use of more than one independent rater to apply the NOS, with disagreements reconciled through discussion rather than acceptance of a single rater’s judgment, directly addresses the inter-rater reliability concern raised in Stang’s critique2. Reporting the item-level breakdown, not only the total star count, allows readers to see which domains produced a lower score for a given study, addressing the concern that a single aggregate number obscures which specific weaknesses matter1. The choice between the NOS and ROBINS-I, or the decision to report results from both, should depend on the specific studies being assessed and the review’s stated bias-assessment priorities, and should be documented in the protocol rather than decided after seeing which tool gives a more favorable result; the AMSTAR 2 criteria for rating a review’s overall rigor specifically credit reviews that justify this kind of methodological choice explicitly4.

Empirical comparisons of quality-assessment tools

Beyond Stang’s conceptual critique, Margulis and colleagues directly compared the Newcastle-Ottawa Scale with a different quality-assessment instrument, the RTI item bank, when both were applied to the same set of observational studies in a drug-safety systematic review, and found that the two tools did not always agree on which studies were of higher or lower quality5. This direct, empirical disagreement between tools reinforces Stang’s more conceptual point: a study’s “quality” is not a single, tool-independent fact on which any reasonable instrument should converge, but is partly a function of which specific domains a given tool chooses to weight and how. The GRADE approach to rating the overall certainty of a body of evidence, described by Guyatt and colleagues, takes a related but broader view, incorporating study-level quality assessments such as the NOS as one input into a wider judgment that also considers consistency, directness, and precision across the whole evidence base rather than any single study’s score in isolation6. Reviewers seeking a GRADE-style overall certainty rating should treat their NOS or ROBINS-I scores as one component of that larger judgment, not as a substitute for it. Haidich’s general overview of meta-analysis methodology situates quality assessment as one step within a longer analytic process that also includes assessment of heterogeneity and publication bias; a well-applied NOS score is therefore necessary but not sufficient for a trustworthy pooled result, and the statistical choices made downstream of quality assessment warrant as much of a reviewer’s attention as the quality scores themselves7. A review that applies the NOS carefully but then omits a proper heterogeneity assessment, or incorporates study-quality scores into an analysis without stating how, has completed only part of the work that the quality-assessment step was intended to support; a thorough reviewer should examine both components rather than treating a well-documented NOS table as evidence that the rest of the analysis was equally rigorous.

Checklist for applying the scale

Teams adopting the NOS for a new meta-analysis benefit from agreeing in writing, before rating begins, on how each of the eight items will be operationalized for their specific review topic, since the wording of the scale leaves some interpretive latitude that two raters may reasonably fill differently without a shared, topic-specific convention agreed in advance. Piloting the scale on a small number of included studies, comparing ratings between raters, and resolving any disagreement through discussion before rating the full set is a practical means of identifying interpretive differences early, when they are inexpensive to correct, rather than discovering after the full exercise that two raters applied the Selection domain inconsistently throughout.

Key points

Key points

The eight items of the Newcastle-Ottawa Scale, organised into selection, comparability and outcome domains, provide a structured means of assessing the quality of non-randomised studies. However, a published critique documents substantive reliability and validity limitations; reviewers should therefore use multiple raters, report item-level detail rather than only the total, and consider ROBINS-I where it is better suited to the review.

Practical application

Practical application

  1. Use at least two independent raters and reconcile disagreements by discussion. This addresses the inter-rater reliability concerns raised in published critiques of the scale.
  2. Report item-level scores rather than only the total star count. This allows readers to identify which domain (selection, comparability or outcome) accounted for a lower score in any given study.
  3. Specify in advance how the score will be used, and document that decision. State at the protocol stage, rather than after the results are known, whether scores will inform a sensitivity analysis, a quality-based weighting or neither.
  4. Consider ROBINS-I as an alternative where it is better suited to the included studies. For non-randomised intervention studies, consider whether the more structured, domain-based approach of ROBINS-I better meets the bias-assessment requirements of the review.

Frequently asked questions

No. The scale’s documentation states that it does not provide strict cutoffs for study quality, so any threshold a review applies, such as treating seven or more stars as high quality, is the reviewing team’s own decision and should be stated and justified explicitly rather than presented as an official feature of the tool.

The critique raised concerns about inter-rater reliability (ambiguity in how items are interpreted by different raters), about whether weighting items of different importance equally accurately reflects their actual effect on bias, and about the scale’s overall construct validity for the purposes for which reviews commonly use it.

The limitations are reasons for caution and for supplementary practices (multiple raters, item-level reporting and consideration of alternatives such as ROBINS-I where appropriate) rather than reasons to abandon a tool that remains widely used and, when applied carefully, provides a structured, useful starting point for assessing the quality of non-randomized studies.

It follows the same three-domain structure (Selection, Comparability and, in place of Outcome, Exposure), reflecting the fact that case-control studies proceed backward from outcome to exposure; the specific items therefore assess how cases and controls were selected and how exposure was ascertained rather than how outcomes were tracked over follow-up.

Not necessarily; ROBINS-I is more rigorous and granular specifically for non-randomized intervention studies but takes considerably longer to apply, so the NOS remains a reasonable, faster choice for broader quality screening of cohort and case-control studies when that level of domain-by-domain detail is not a priority for the review.

References

  1. Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute. read 10 October 2026. View NOS documentation
  2. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. European Journal of Epidemiology. 2010;25(9):603-605. read 10 October 2026. View on European Journal of Epidemiology
  3. Sterne JAC, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. read 10 October 2026. View on The BMJ
  4. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. read 10 October 2026. View on The BMJ
  5. Whiting P, Savović J, Higgins JPT, et al. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology. 2016;69:225-234. read 10 October 2026. View on Journal of Clinical Epidemiology
  6. Kyu HH, Bachman VF, Alexander LT, et al. Physical activity and risk of breast cancer, colon cancer, diabetes, ischemic heart disease, and ischemic stroke events: systematic review and dose-response meta-analysis for the Global Burden of Disease Study 2013. BMJ. 2016;354:i3857. read 10 October 2026. View on The BMJ
  7. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on The BMJ
  8. Margulis AV, Pladevall M, Riera-Guardia N, et al. Quality assessment of observational studies in a drug-safety systematic review, comparison of two tools: the Newcastle-Ottawa Scale and the RTI item bank. Clinical Epidemiology. 2014;6:359-368. read 10 October 2026. View on Clinical Epidemiology
  9. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. read 10 October 2026. View on The BMJ
  10. Haidich AB. Meta-analysis in medical research. Hippokratia. 2010;14(Suppl 1):29-37. read 10 October 2026. View on PMC

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Assessing Study Quality in Meta-Analysis Using the Newcastle-Ottawa Scale (NOS). Pubrica. https://pubrica.com/academy/meta-analysis/assessing-study-quality-in-meta-analysis-using-newcastle-ottawa-scale/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

NOS Application Checklist

This checklist is intended for applying the Newcastle-Ottawa Scale rigorously rather than mechanically.

[ ] At least two independent raters scoring each study
[ ] Disagreements reconciled by discussion, documented
[ ] Item-level scores recorded and reported, not just the total
[ ] Any quality cutoff used in the review stated and justified explicitly (not treated as an official NOS feature)
[ ] Considered whether ROBINS-I is a better fit for the specific studies and question
[ ] Decision on how scores will be used (sensitivity analysis / weighting / neither) documented in the protocol

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s evidence synthesis team applies the Newcastle-Ottawa Scale with dual independent raters and full item-level reporting, and advises on when ROBINS-I is the more appropriate alternative for a given review.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Reliability of Newcastle-Ottawa scoring in meta-analysis

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.