Contents
On this page
- Reviewer reliability as a measurable question
- Findings of the cross-disciplinary evidence
- Focused specialties and diffuse general fields
- Grant review and manuscript review
- Implications for interpreting a single review decision
- Effects of reviewer blinding
- A field-specific test: near-chance agreement in clinical neuroscience
- Reliability of grant panel scoring
Academy · Research Publication · Peer Review Reliability Across Fields
The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement
Journal peer review is often assumed to operate uniformly across disciplines. However, decades of meta-research on reviewer agreement indicate that reliability differs substantially between narrowly focused specialties and broad, diffuse fields.
Two reviewers assigned to the same manuscript frequently disagree, and the extent of their agreement depends substantially on the field. The most influential cross-disciplinary analysis of this question, Daryl Cicchetti’s 1991 review in Behavioral and Brain Sciences, found that reviewer reliability is consistently low across science in general, but that the pattern of disagreement differs in a specific, replicable way between narrow specialties and broad, heterogeneous fields. This distinction is relevant to the weight that any single peer review decision should carry, and to how editors, funders and authors should interpret a single rejection or acceptance.
Learning objectives
- Why the cross-disciplinary analysis by Cicchetti found that grant reviewers agree more about what is unworthy of support than about what has genuine scientific value
- The documented difference between ‘general and diffuse’ fields and ‘specific and focused’ fields in the decisions on which reviewers agree, namely rejection or acceptance
- The implications of this evidence for the degree of confidence that a single review decision warrants, in any discipline
Common misconceptions
- “The reliability of peer review is broadly similar across scientific disciplines.” The cross-disciplinary review by Cicchetti found that patterns of reviewer agreement differ systematically: in general, diffuse fields, reviewers agree much more on what to reject than on what to accept, whereas in specific, focused fields the correlation between referee recommendations and editorial decisions runs in the opposite direction.
- “Low agreement between reviewers indicates that peer review does not function.” The evidence indicates that agreement is low in a structured manner: reviewers in both grant and manuscript review tend to agree much more about what is clearly unworthy than about what is of high value, which is a different and more informative finding than random disagreement.
- “A single rejection reliably indicates that a paper or grant lacks merit.” Because referee agreement on acceptance is consistently weaker than agreement on rejection in broad fields, a single negative decision provides less diagnostic information about merit than is often assumed.
Reviewer reliability as a measurable question
Peer review reliability is not a philosophical question but a measurable statistical property, typically assessed by comparing the recommendations or scores that two or more independent reviewers give to the same manuscript or grant application. Daryl V. Cicchetti's 1991 paper in Behavioral and Brain Sciences, “The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation,” remains the most frequently cited synthesis of this evidence base; it critically re-examined reliability findings reported across multiple fields and prompted open peer commentary from editors and methodologists in the same issue1. A later, widely read overview by former BMJ editor Richard Smith similarly concluded that peer review remains a flawed, inconsistent filter at the centre of both science and journal publishing, rather than the objective quality check it is often assumed to be10.
Findings of the cross-disciplinary evidence
Cicchetti's central finding was that peer review reliability is consistently reported as low across the fields examined1. However, “low” did not mean uniform or random: the pattern of disagreement followed the structure of the field under review rather than chance.
For grant submissions specifically, the review found that referees agree considerably more about which proposals are unworthy of support than about which proposals have genuine scientific value1, which implies that a negative signal from a grant panel is more dependable than a positive one, an asymmetry that is somewhat counterintuitive and relevant to both funders and applicants.
Focused specialties and diffuse general fields
The most actionable distinction in the analysis is between two categories of field1:
- General and diffuse fields — Cicchetti's examples include cross-disciplinary physics, general medicine, cultural anthropology and social psychology. In these fields, reviewers agree considerably more about what to reject than about what to accept.
- Specific and focused fields — examples include nuclear physics, medical specialties, physical anthropology and behavioral neuroscience. In these fields the pattern is reversed: rejection rates vary widely between reviewers, but the correlation between referee recommendations and the editor's final decision is high, suggesting that agreement is stronger on acceptance than on rejection.
In summary, the broader and more heterogeneous the subject matter of a field, the more its reviewers converge on rejecting weak work but diverge on recognising strong work, whereas narrow specialties show the opposite asymmetry.
Grant review and manuscript review
The analysis treats grant review and manuscript review as related but distinct reliability problems, and its central conclusion (consistently low reported reliability, with judgments of “unworthy work” more stable than judgments of “valuable work”) applies to grant panels as much as to journal referees1. Cicchetti's paper also proposed specific recommendations for improving reliability and quality in both systems, and noted that further empirical work was particularly needed in the physical sciences, where the evidence base at the time was less extensive than in biomedicine or the social sciences1.
Implications for interpreting a single review decision
The practical implication for authors, applicants and editors is that a single review verdict should be weighted according to both its direction and the breadth of the field. In a broad, diffuse field, a rejection carries more diagnostic weight than an acceptance; the reverse weighting applies in narrow specialties. This is also the principal empirical argument for multiple independent reviewers and for structural reforms, such as pre-registration of study design before results are known, discussed elsewhere in the literature on review reform.
Effects of reviewer blinding
A separate strand of reliability research has tested a specific intervention, masking the identity of a manuscript's authors from reviewers, to determine whether it improves agreement or quality. The results are more modest than might be expected. A 1990 randomized trial at a general internal medicine journal found that blinding reviewers to author identity significantly improved editors' ratings of review quality, but authors reported no corresponding improvement in thoroughness, constructiveness or fairness2. A separate 1998 randomized trial of the same intervention found that blinding had no measurable effect on whether a paper was ultimately accepted or rejected, which is arguably the most important outcome3. A companion 1998 trial published in JAMA by the PEER Investigators likewise found that masking author identity produced no detectable improvement in the quality of reviewers' critiques9, and a related 1998 JAMA trial on blinding and unmasking found no difference between blinded and unblinded review in reviewers' or editors' assessments of review quality4. Across these trials, 27–46% of reviewers correctly identified the authors despite blinding, through self-referencing or familiarity with the work, which probably dilutes any effect that blinding could have3. A subsequent 1999 BMJ trial of the opposite intervention, fully open review with reviewers' names disclosed to authors, likewise found no meaningful difference in review quality or recommendations compared with standard anonymous review5. Taken together, this body of work suggests that neither blinding nor disclosure of reviewer identity, by itself, meaningfully changes reviewer reliability, and that the underlying problem lies elsewhere.
A field-specific test: near-chance agreement in clinical neuroscience
One of the clearest single-field demonstrations of low reliability comes from a 2000 study in Brain that directly tested whether reviewer agreement in clinical neuroscience exceeded that expected from chance alone. Examining reviewer recommendations (accept, revise or reject) at a neuroscience journal, the authors calculated a kappa statistic of only 0.08 for one journal studied, a value indicating agreement barely above chance6. The authors concluded unequivocally that agreement between reviewers in clinical neuroscience was little greater than would be expected by chance alone6. Clinical neuroscience lies closer to Cicchetti's “specific and focused” category than to a diffuse field such as general medicine, so this near-chance result illustrates how far reliability can deteriorate even within a narrow specialty, depending on the specific journal and reviewer pool.
Reliability of grant panel scoring
Grant review has its own dedicated reliability literature, distinct from that on journal manuscript review. A 2008 synthesis in American Psychologist examining the peer-review process for competitive grant applications concluded that reliability, validity and freedom from bias in grant scoring are all empirically testable properties of a review system, and that measured reliability in grant panels has frequently been found to be only modest, broadly consistent with the manuscript-review findings described above, rather than indicating a separate, more dependable process7. A 2010 meta-analysis aggregating reliability coefficients across a large number of published peer-review reliability studies, spanning both grant and manuscript review, similarly concluded that the overall reliability of peer assessments is best characterized as low to moderate, and that reliability varies meaningfully by field and by the specific rating dimension examined, a finding that echoes Cicchetti's original cross-disciplinary pattern two decades later with a larger evidence base8.
Key points
Key points
The reliability of peer review is low across disciplines, but not randomly so: broad, diffuse fields show strong reviewer agreement on rejecting weak work and weak agreement on recognising strong work, whereas narrow specialties show the reverse pattern.
Practical application
Practical application
- A single decision should be weighted according to the breadth of the field as well as its direction. In a broad or interdisciplinary field, a single rejection should be regarded as more diagnostically reliable than a single acceptance; in a narrow specialty, acceptance should be weighted more heavily.
- A second independent assessment should be sought before a rejected proposal is abandoned. Because agreement on acceptance is the weaker signal in diffuse fields, a second independent opinion is particularly valuable before concluding that a rejected manuscript or grant lacks merit.
- Reviewer disagreement should be reported transparently in editorial decisions. Where feasible, divergent reviewer recommendations should be disclosed to authors rather than presented as a single averaged decision, so that authors can judge the extent to which a specific criticism should inform revision.
Frequently asked questions
No. Cross-disciplinary analysis found that reviewer agreement differs systematically between broad, diffuse fields and narrow, focused specialties, not only in degree but also in the type of judgment (acceptance or rejection) that is more consistent.
The evidence indicates structured rather than random disagreement: reviewers converge reliably on identifying unworthy work even where they diverge on recognizing the most valuable work, which supports refining review rather than abandoning it.
Cicchetti's analysis specifically identified the physical sciences as requiring further empirical study of peer review reliability relative to the biomedical and social-science fields.
References
- Cicchetti, D. V. (1991). "The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation." Behavioral and Brain Sciences, 14(1), 119–135. read 10 October 2026. View on Cambridge Core
- McNutt RA, Evans AT, Fletcher RH, Fletcher SW. The effects of blinding on the quality of peer review: a randomized trial. JAMA. 1990;263(10):1371-1376. read 10 October 2026. View on PubMed
- Fisher M, Friedman SB, Strauss B. The effects of blinding on acceptance of research papers by peer review. JAMA. 1994;272(2):143-146. read 10 October 2026. View on JAMA Network
- van Rooyen S, Godlee F, Evans S, Black N, Smith R. Effect of blinding and unmasking on the quality of peer review: a randomized trial. JAMA. 1998;280(3):234-237. read 10 October 2026. View on PubMed Central
- van Rooyen S, Godlee F, Evans S, Smith R, Black N. Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. BMJ. 1999;318(7175):23-27. read 10 October 2026. View source
- Rothwell PM, Martyn CN. Reproducibility of peer review in clinical neuroscience: is agreement between reviewers any greater than would be expected by chance alone? Brain. 2000;123(9):1964-1969. read 10 October 2026. View source
- Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. American Psychologist. 2008;63(3):160-168. read 10 October 2026. View source
- Bornmann L, Mutz R, Daniel HD. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE. 2010;5(12):e14331. read 10 October 2026. View source
- Justice AC, Cho MK, Winker MA, Berlin JA, Rennie D, for the PEER Investigators. Does masking author identity improve peer review quality? A randomized controlled trial. JAMA. 1998;280(3):240-242. read 10 October 2026. View on PubMed Central
- Smith R. Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine. 2006;99(4):178-182. read 10 October 2026. View on PubMed
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement. Pubrica. https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/
Pubrica Academy. “The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement.” Pubrica, 10 Oct. 2026, pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/.
Pubrica Academy (2026) The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement. Pubrica. Available at: https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/ (Accessed: 15 October 2026).
Pubrica Academy. “The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement.” Pubrica. October 10, 2026. https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/
Pubrica Academy, ‘The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/> accessed 15 October 2026.
Pubrica Academy. The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/
Pubrica Academy, “The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/
@misc{/la_peer_review_reliability_across_disciplines_evidence_2026,
author = {{Pubrica Academy}},
title = {The Reliability of Peer Review Across Disciplines: Evidence on Reviewer Agreement},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/concepts-definitions/the-reliability-of-scientific-grant-review-for-manuscript-a-cross-disciplinary-investigation/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Interpreting a split or single review verdict
This checklist should be used when determining how much weight to give a single peer-review outcome.
[ ] Identify whether the field is broad/diffuse or narrow/focused [ ] If rejected in a broad field, treat the verdict as higher-confidence; seek a second opinion before treating an acceptance as high-confidence [ ] If rejected in a narrow specialty, seek a second independent read before abandoning the work [ ] Request or review reviewer-by-reviewer scores rather than only an averaged verdict where available [ ] Document specific, falsifiable critiques separately from general impressions of quality
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s assessments of manuscript and grant readiness for review draw on documented patterns of reviewer agreement to identify which criticisms are likely to recur across independent reviewers and which reflect the individual view of a single reviewer.