Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · QUADAS-2 Limitations

Limitations of the QUADAS-2 Tool: Comparative Designs, Reporting Dependence and Assessor Agreement

QUADAS-2 was not designed to assess studies that compare two or more diagnostic tests head-to-head. This gap was sufficiently significant that its developers published a separate tool, QUADAS-C, in 2021 specifically to cover comparative diagnostic accuracy designs.

Pubrica AcademyWritten by a diagnostic accuracy research writer Reviewed bya systematic review methodologist Published10 October 2026 Last checked10 October 2026 Reading time7 minutes Sources10 peer-reviewed

QUADAS-2 is the standard tool for assessing risk of bias and applicability concerns in diagnostic accuracy studies, but it has specific, documented limitations that affect how and when it should be used. It was designed for studies assessing a single index test against a reference standard, not for studies directly comparing two or more tests; it depends heavily on the completeness of reporting in the primary study, which is frequently inadequate; and the consistent application of its signalling questions requires a level of methodological judgment that, in practice, produces measurable disagreement between independent assessors. Each of these limitations is documented in the methodological literature, and each has prompted either a supplementary tool or explicit guidance on how to compensate for the gap.

Learning objectives

  • The reasons QUADAS-2 does not cover studies directly comparing two or more diagnostic tests, and the tool that addresses this gap
  • The ways in which incomplete primary-study reporting limits the conclusions of QUADAS-2 assessments
  • The points at which assessor judgment introduces measurable inconsistency into QUADAS-2 ratings

Common misconceptions

  • “QUADAS-2 is a universal, one-size-fits-all tool suitable for any diagnostic accuracy study design.” The methodological lineage of QUADAS-2 shows that this is not the case: the developers identified that the tool was not built to assess comparative diagnostic accuracy studies, in which two or more tests are evaluated head-to-head in the same study population, and in 2021 published a distinct tool, QUADAS-C, specifically to cover that design, a gap the original tool was never intended to close.

Unsuitability of QUADAS-2 for comparative diagnostic accuracy studies

The four domains of QUADAS-2 (patient selection, index test, reference standard, and flow and timing) were designed around a study structure in which a single index test is evaluated against a reference standard in one patient population1. Many diagnostic accuracy questions, however, are addressed by comparative studies, in which two or more index tests are evaluated head-to-head in the same patients specifically to determine which test performs better. The domains and signalling questions of QUADAS-2 were not built to assess this design, since they do not address biases specific to within-study test comparisons, such as whether each test was interpreted blind to the results of the other2. This gap was sufficiently important that several of the original developers of QUADAS-2 published a dedicated successor tool, QUADAS-C, in 2021, adding comparison-specific signalling questions covering issues such as whether the comparison was randomized and whether the results of both tests were interpreted independently2. Reviewers who assess comparative diagnostic accuracy studies with the base QUADAS-2 tool rather than QUADAS-C risk overlooking bias specific to the comparative design, a limitation documented by the development lineage of the tool itself rather than by external critique alone2.

Dependence on the completeness of primary-study reporting

The signalling questions of QUADAS-2 can be answered only as confidently as the reporting of the primary study allows, and systematic audits of the diagnostic accuracy literature, both before and after the introduction of STARD, have repeatedly found that many published studies omit information relevant to several QUADAS-2 domains, including the exact method of patient recruitment, whether assessors were blinded to other test results, and the precise interval between index test and reference standard3. The methodological paper by Wade, Corbett and Eastwood on applying a modified QUADAS tool in practice describes this problem from the reviewer’s perspective: when primary studies did not report key methodological details, reviewers frequently had to default to “unclear” risk-of-bias judgments rather than confident low- or high-risk ratings, which limits the discriminating power of the resulting risk-of-bias summary across a body of evidence in which many studies are judged “unclear” on the same domain4. This is a limitation of the evidence base QUADAS-2 is applied to as much as of the tool itself, but it directly constrains what conclusions a QUADAS-2 assessment can support34.

Assessor disagreement arising from signalling-question judgments

Even where a primary study is reported completely, answering the signalling questions of QUADAS-2 and translating those answers into a domain-level risk-of-bias judgment still require methodological judgment, and independent assessors applying the same tool to the same study do not always reach the same conclusion1. The original QUADAS-2 development paper reports this directly: during the piloting and refinement of the tool, the development group found that some signalling questions required further clarification or rewording specifically because early users interpreted and applied them inconsistently, prompting revisions before the final version was published1. For this reason, the QUADAS-2 guidance recommends piloting the tool on a sample of included studies and using two independent assessors with a documented disagreement-resolution process, rather than relying on the judgment of a single assessor, an acknowledgment that the reliability of the tool depends partly on how it is implemented and not solely on its written criteria1.

Application of QUADAS-2 within its limitations

None of these limitations renders QUADAS-2 unsuitable for its intended purpose, the assessment of single-index-test diagnostic accuracy studies, for which it remains the standard, well-validated tool recommended by the Cochrane Handbook for Diagnostic Test Accuracy reviews1. The appropriate response to its documented limitations is to use the correct tool for the study design (QUADAS-C for comparative studies), to pilot the tool and use multiple independent assessors with a documented resolution process for disagreements, and to report “unclear” judgments transparently as a reflection of incomplete primary-study reporting rather than treating every domain as confidently rated when the underlying evidence does not support that confidence24.

Rationale for the absence of a single summary quality score in QUADAS-2

An early and influential limitation identified in diagnostic accuracy quality assessment generally, predating QUADAS-2 itself, was the practice of collapsing the quality assessment of a study into a single overall numeric score. The 2005 methodological paper by Whiting, Harbord and Kleijnen argued directly against this practice, showing that different scoring systems applied to the same set of studies could rank those studies very differently, and that a single combined score obscures which specific domain (patient selection, blinding or another) is driving the risk of bias of a study, information that a reader requires far more than an undifferentiated number5. This critique directly shaped the design of QUADAS-2: unlike some earlier quality-assessment approaches, QUADAS-2 deliberately reports separate domain-level judgments rather than summing them into one score, precisely to avoid the problem that Whiting and colleagues identified1. The underlying concern about bias in diagnostic accuracy studies that QUADAS-2 was eventually built to assess is itself long documented: the foundational 1999 study by Lijmer and colleagues found empirically that diagnostic accuracy estimates varied systematically with specific design features, including case-control designs and differential application of the reference standard, providing much of the original empirical basis for treating these as distinct, separately assessable domains of bias rather than a single undifferentiated notion of “study quality”6. The methodological overview by Deeks of systematic reviews of diagnostic test evaluations presents this domain-based approach as the settled position of the field, correcting a limitation of the earlier, cruder quality-scoring methods that QUADAS and subsequently QUADAS-2 were specifically built to address, even though QUADAS-2 itself carries the narrower, more specific limitations described above7. The later empirical analysis by Rutjes and colleagues of bias and variation across a large sample of diagnostic accuracy studies reinforces this point from a different perspective, finding that several of the specific design features that the domains of QUADAS-2 are built to detect (non-consecutive patient recruitment, differential application of the reference standard and lack of blinding) were each independently associated with inflated accuracy estimates; this finding forms part of the accumulated evidence base that justified assessing these features separately rather than combining them into one score8. The survey by Willis and Quigley of methodological practice in diagnostic accuracy research found that, even years after QUADAS and its successors became available, uptake of recommended methodological safeguards, including blinding and appropriate patient selection, remained inconsistent across the published literature. This constitutes a further practical limitation on what any risk-of-bias tool can achieve: QUADAS-2 can only report what a primary study did and cannot compel primary researchers to adopt the safeguards its domains are built to assess9. The chapter on assessing methodological quality in the Cochrane Handbook for Diagnostic Test Accuracy reviews acknowledges this limitation explicitly, recommending that reviewers treat a body of evidence dominated by high or unclear risk-of-bias ratings as a reason to qualify the overall conclusions of the review, rather than as a problem that QUADAS-2 itself can resolve through more careful application10. None of these limitations is a reason to avoid QUADAS-2; rather, they indicate that it should be applied with its documented boundaries in view: the comparative variant should be chosen when the study design requires it, unclear judgments should be expected and reported transparently where primary studies under-report their methods, and assessor disagreement should be resolved through a documented process rather than overridden by the confidence of a single reviewer.

Key points

Key points

QUADAS-2 was designed for diagnostic accuracy studies of a single index test, not for comparative designs, for which QUADAS-C should be used. Its judgments are limited by the completeness with which the primary study reports its own methods, and its signalling questions still require assessor judgment that may produce disagreement even when applied carefully.

Practical application

Practical application

  1. The study design should be confirmed as one that QUADAS-2 was built to assess. QUADAS-C, rather than QUADAS-2, should be used for studies that directly compare two or more index tests head-to-head.
  2. The tool should be piloted, and two independent assessors should be used with a documented resolution process. This approach directly addresses the documented inconsistency in the interpretation and application of signalling questions.
  3. “Unclear” judgments should be reported transparently, with discussion of what they reflect. The discussion section of the review should state whether a high proportion of “unclear” ratings reflects gaps in the reporting of the underlying literature rather than ambiguity in the review team’s own assessment process.

Frequently asked questions

Not reliably, because it was not designed to capture biases specific to comparative designs. QUADAS-C, published in 2021 by several of the same developers, was created specifically to assess comparative diagnostic accuracy studies.

Because the signalling questions of the tool can be answered only as confidently as the reporting of the primary study allows; incompletely reported studies, a common and documented problem in the diagnostic literature, necessitate more “unclear” judgments.

Not necessarily. The development process of the tool found that early users interpreted some signalling questions inconsistently, which is why the use of two independent assessors with a documented disagreement-resolution process is standard recommended practice.

No; it is a companion tool for a specific study design (comparative diagnostic accuracy studies), and QUADAS-2 remains the appropriate tool for single-index-test studies.

No. A reviewer can record only what is reported; for this reason, STARD-compliant reporting at the primary-study stage directly affects how confidently a later QUADAS-2 assessment can be made.

References

  1. Whiting PF, Rutjes AWS, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536. read 10 October 2026. View on Annals of Internal Medicine
  2. Yang B, Mallett S, Takwoingi Y, et al. QUADAS-C: A Tool for Assessing Risk of Bias in Comparative Diagnostic Accuracy Studies. Annals of Internal Medicine. 2021;174(11):1592-1599. read 10 October 2026. View on Annals of Internal Medicine
  3. Bossuyt PM, Reitsma JB, Bruns DE, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. read 10 October 2026. View on BMJ
  4. Wade R, Corbett M, Eastwood A. Quality assessment of comparative diagnostic accuracy studies: our experience using a modified version of the QUADAS tool for systematic reviews. BMC Medical Research Methodology. 2013;13:150. read 10 October 2026. View on BMC Medical Research Methodology
  5. Whiting P, Harbord R, Kleijnen J. No role for quality scores in systematic reviews of diagnostic accuracy studies. BMC Medical Research Methodology. 2005;5:19. read 10 October 2026. View on BMC Medical Research Methodology
  6. Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061-1066. read 10 October 2026. View on JAMA
  7. Deeks JJ. Systematic reviews of evaluations of diagnostic and screening tests. BMJ. 2001;323(7305):157-162. read 10 October 2026. View on BMJ
  8. Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174(4):469-476. read 10 October 2026. View on CMAJ
  9. Willis BH, Quigley M. Uptake of newer methodological developments and its influence on the perceived quality of diagnostic test accuracy studies. BMC Medical Research Methodology. 2011;11:27. read 10 October 2026. View on BMC Medical Research Methodology
  10. Reitsma JB, Rutjes AWS, Whiting P, Yang B, Leeflang MM, Deeks JJ. Chapter 9: Assessing methodological quality. In: Deeks JJ, Bossuyt PM, Leeflang MM, Takwoingi Y, eds. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy, version 2.0. Cochrane, 2023. read 10 October 2026. View the Cochrane DTA Handbook

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Limitations of the QUADAS-2 Tool: Comparative Designs, Reporting Dependence and Assessor Agreement. Pubrica. https://pubrica.com/academy/meta-analysis/what-are-the-limitations-of-the-quadas-2-tool/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

QUADAS-2 Applicability Pre-Check

It should be confirmed that QUADAS-2 is the appropriate tool before the risk-of-bias assessment begins.

[ ] Confirmed the included studies are single-index-test designs, not head-to-head comparisons (use QUADAS-C if comparative)
[ ] Piloted the tool on a sample of included studies before full assessment
[ ] Assigned two independent assessors and documented the disagreement-resolution process
[ ] Flagged domains likely to be "unclear" due to known gaps in primary-study reporting
[ ] Planned how "unclear" ratings will be discussed and interpreted in the review's findings

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s systematic review team selects the risk-of-bias tool appropriate to the study design, QUADAS-2 or QUADAS-C, and documents assessor agreement transparently.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Appropriateness of QUADAS-2 for the included studies

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.