Contents
On this page
- Direct observation of funding council panels
- Interaction between budget size and rating-scale granularity
- Influence of the ranking method on outcomes
- Limited influence of written review guidelines
- Implications for applicants and funders
- Effect of reviewer assignment on outcomes
- Effect of panel discussion on reviewer disagreement
- Distinguishing bias from statistical noise
- Approaches to improving panel composition
Academy · Research Publication · Grant Panel Decision-Making Case Study
The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes
A detailed study of Research Council of Norway grant panels found that budget size, rating-scale granularity and ranking method had a greater influence on which proposals were funded than the panels’ written review guidelines.
Grant review guidelines describe the criteria a panel is expected to apply, such as significance, feasibility and investigator track record. However, Liv Langfeldt’s direct observational study of Research Council of Norway (RCN) panels found that the outcome of a funding round was shaped considerably more by three structural and often unacknowledged features of the process: the budget available to the panel, the granularity of its rating scale and the method used to rank and eliminate proposals. Drawing on rare direct access to panel meetings and the complete documentation of applications and reviews, the study provides one of the most concrete empirical accounts available of how a funding body’s decision process, and not only its stated criteria, determines which applicants are funded.
Learning objectives
- Why written review guidelines had limited measurable effect on the criteria that RCN panels emphasized
- How budget size and rating-scale granularity interact to determine whether a panel can fund pluralistic or innovative projects or only safe, established ones
- How different proposal-ranking methods systematically favor either uncontroversial ‘safe’ projects or individually championed ‘pluralistic’ projects
Common misconceptions
- “Review guidelines are the principal determinant of the criteria a panel applies.” Langfeldt’s direct observation of RCN panels found that written guidelines had limited effect on the criteria reviewers emphasized; in practice, budget restrictions, rating-scale design and ranking method were more influential.
- “A panel with a larger budget simply funds more of the proposals it would fund with a smaller budget.” The study found that ample budgets combined with coarse rating scales allowed panels to pursue deliberate priorities, such as supporting underfunded fields, subfield pluralism or unusually innovative projects, whereas tight budgets and fine-grained scales precluded this.
- “The method by which a panel ranks proposals is a minor procedural detail.” The study found that the ranking method could determine the outcome: methods that allowed each panel member to secure one preferred project supported pluralism but gave individual members disproportionate influence, whereas methods that eliminated any proposal not prioritized by a majority tended to fund only uncontroversial, safe projects.
Direct observation of funding council panels
Liv Langfeldt's study, "The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome," originally published in Social Studies of Science and reprinted as a chapter in the collection Peer Review in an Era of Evaluation, examined grant panels at the Research Council of Norway through a combination of direct observation of panel meetings and full access to the underlying applications and review documents1. The author notes that this level of access is rare in grant-review research, in which panel deliberations are typically confidential and studied only indirectly through outcome data or surveys1.
Interaction between budget size and rating-scale granularity
One of the clearest findings of the study is an interaction effect: ample budgets combined with coarse rating scales allowed panels to pursue policy priorities such as supporting weaker fields, subfield pluralism or unusually innovative projects, because many proposals received similar ratings and the panel could then choose among them on the basis of judgment1. Tight budgets combined with fine-grained rating scales produced the opposite effect: proposals were sharply differentiated by score, leaving little scope for anything other than funding the highest-rated, typically most established, research1.
Influence of the ranking method on outcomes
The study found that the specific method a panel used to move from individual ratings to a final funding list was not a neutral administrative step but systematically favored certain types of proposal1:
- Methods that allowed each panel member to secure at least one personally favored project tended to support pluralism across subfields, but gave individual panel members disproportionate and somewhat arbitrary influence over which specific projects were funded.
- Methods that eliminated any proposal not prioritized by a majority of the panel tended to converge on uncontroversial, “safe” projects, those that no member actively opposed, at the expense of more controversial or boundary-extending research.
Consequently, two panels applying identical review criteria to the same pool of applications could reach meaningfully different funding outcomes solely because they used different ranking procedures.
Limited influence of written review guidelines
A notably counterintuitive finding is that written review guidelines, the formal criteria documents that applicants are instructed to address, had limited measurable effect on the criteria that panels emphasized in practice1. The structural features of the process (budget, rating scale and ranking method) proved to be stronger determinants of outcome than the stated evaluation framework. Because there were no clear, enforced assessment norms independent of these structural choices, the study also found that outcomes could depend in part on which reviewers were assigned and on how their individual reviews were organized and aggregated1.
Implications for applicants and funders
For funders, the practical implication is that changing the resolution of a panel's rating scale or its ranking method is a more powerful means of shaping funding outcomes, toward either safe consensus or pluralistic risk-taking, than revising the written review guidelines. For applicants, the implication is more modest but still relevant: the fate of a proposal depends partly on structural features of the specific panel and funding round to which it is assigned, and not solely on its intrinsic merit relative to a fixed, universal standard. The study's own caveat should also be noted: the underlying RCN data are from the 1990s and the funding landscape has since evolved, although the structural dynamics documented (budget, scale and ranking interacting to shape outcomes) are argued to generalize beyond that period1.
Effect of reviewer assignment on outcomes
Langfeldt's structural findings are complemented by a growing body of quantitative evidence that the identity of the reviewers assigned to a proposal is itself a major source of variability in outcome, independent of ranking method or budget. A 2018 study in PNAS by Pier and colleagues had 43 experienced NIH reviewers independently score the same set of grant applications under realistic conditions and found essentially no agreement between different reviewers on the same proposals; each reviewer was internally consistent regarding the strengths and weaknesses identified, but reviewers differed widely in how they translated those observations into a numeric score3. A 2011 retrospective analysis in the BMJ, modeling the observed scoring variability among members of an Australian National Health and Medical Research Council grant panel, estimated that of 2,705 proposals reviewed, only 9% would always have been funded and 61% would never have been funded, irrespective of which panel members reviewed them; for the remaining 29% of proposals, including 59% of the grants actually funded, funding status depended on which reviewers sat on the panel that year8. A comparable statistical study of telescope-time proposal ranking at the European Southern Observatory found the same pattern outside biomedicine: individual referees agreed on which proposals belonged in the top quartile only about one-third of the time before panel discussion, rising to approximately 55% after full panel deliberation, which remains far short of consensus10.
Effect of panel discussion on reviewer disagreement
It might be assumed that group discussion during a panel meeting resolves individual scoring disagreement of the kind described above. A 2017 study in Research Evaluation, tracking how scores changed before and after discussion across four grant panels, found the opposite pattern at the between-panel level: reviewers within a given panel converged somewhat after discussing a proposal, but different panels did not converge with one another; indeed, the range of scores between panels widened after discussion, driven partly by “score calibration talk,” in which the framing of one reviewer shifted the views of the whole panel4. This provides a direct, mechanistic illustration of Langfeldt's broader claim: the internal discussion dynamics of a panel, and not only its written criteria, measurably shift outcomes, and they do not evidently shift them toward a single, panel-independent assessment.
Distinguishing bias from statistical noise
Two different threats to fairness, which this literature treats separately, should be distinguished: systematic bias related to an applicant's identity, and ordinary reviewer-to-reviewer variability unrelated to identity. A 2019 experimental study in Nature Human Behaviour, in which 412 reviewers scored real R01 proposals edited to imply different principal-investigator races and genders, found little or no systematic bias by race or gender in initial review scores; any advantage favoring white male applicants was smaller than the authors' own pre-specified threshold for practical importance, across more than 4,500 alternative statistical models5. Considered together with the Pier and Graves findings above, the evidence indicates that grant review outcomes are strongly shaped by random, reviewer-specific variability, the “luck of the draw” to which an older 1981 Science study of NSF peer review attributed approximately half of a proposal's funding outcome7. This is a problem distinct from identity-based bias, and one that Langfeldt's structural factors (budget, scale, ranking method) can amplify or attenuate independently of whether bias is present.
Approaches to improving panel composition
Sociologist Michèle Lamont's extended qualitative study of disciplinary peer-review panels similarly found that panel deliberation is strongly governed by unwritten disciplinary customs and by the specific individuals present, rather than solely by stated criteria, which supports Langfeldt's claim that panel procedure warrants as much scrutiny as panel criteria6. A 2023 critique of NIH study-section practice proposes a specific structural remedy: rather than revising scoring criteria, the author argues that NIH should change how Scientific Review Officers staff panels, by recruiting several senior co-chairs with directly matched expertise who are empowered to flag reviewer comments reflecting a misunderstanding of the science before those comments affect a priority score, rather than allowing the critique of a single inadequately informed reviewer to defeat an application unchallenged9. That proposal directly reflects Langfeldt's original finding: changing who reviews, and how disagreement is adjudicated procedurally, alters outcomes more reliably than rewriting the criteria document against which all proposals are nominally scored.
Key points
Key points
A direct observational study of Research Council of Norway grant panels found that budget size, rating-scale granularity and ranking method shaped funding outcomes more than written review guidelines did; identical criteria may therefore produce different results depending on panel procedure alone.
Practical application
Practical application
- Funders should audit ranking methods for directional bias. Before a panel’s ranking procedure is finalized, funders should assess whether it structurally favors uncontroversial consensus projects or pluralistic, individually championed ones, since the study found that this choice can determine the outcome independently of the review criteria.
- Funders should align rating-scale granularity with policy objectives. Coarser rating scales are appropriate when the objective is to preserve panel discretion for strategic priorities (underfunded fields, innovation), and finer scales when the objective is strict rank-ordering by merit alone.
- Applicants should investigate the likely procedure of the specific panel, not only the published criteria. Where panel composition, budget constraints or ranking method can be ascertained in advance, they should inform submission timing and strategy, since the study found that these factors can be as influential as the content of the proposal.
- Funders should periodically examine whether assessment norms are consistent irrespective of panel composition. Because the study found that the absence of clear assessment norms allowed outcomes to vary with the reviewers assigned, routine calibration exercises across panels may reduce this source of variability.
Frequently asked questions
The Research Council of Norway (RCN); the author had direct access to panel meetings and to the full documentation of applications and reviews, which the study describes as rare in this area of research.
They were found to have limited measurable effect on the criteria that panels emphasized, relative to the stronger effects of budget restrictions, rating-scale design and ranking method. This does not mean that guidelines are irrelevant, but that they were not the dominant factor observed.
The underlying RCN data are dated, as the author acknowledges, but the structural dynamics documented (budget and scale interacting with ranking method to shape outcomes) are presented as general insights into panel decision-making rather than findings specific to that period.
References
- Langfeldt, L. (2001). "The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome." Social Studies of Science, 31(6), 820–841. read 10 October 2026. View source
- Langfeldt, L. (2022). "The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome." In Peer Review in an Era of Evaluation: Understanding the Practice of Gatekeeping in Academia (pp. 297–326). Springer. read 10 October 2026. View on Springer
- Pier EL, Brauer M, Filut A, Kaatz A, Raclaw J, Nathan MJ, Ford CE, Carnes M. Low agreement among reviewers evaluating the same NIH grant applications. PNAS. 2018;115(12):2952-2957. read 10 October 2026. View source
- Pier EL, Raclaw J, Kaatz A, Brauer M, Carnes M, Nathan MJ, Ford CE. 'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review. Research Evaluation. 2017;26(1):1-14. read 10 October 2026. View source
- Forscher PS, Cox WTL, Brauer M, Devine PG. Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nature Human Behaviour. 2019;3(3):257-264. read 10 October 2026. View source
- Lamont M. How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press; 2009. read 10 October 2026. View source
- Cole S, Cole JR, Simon GA. Chance and consensus in peer review. Science. 1981;214(4523):881-886. read 10 October 2026. View discussion & citation
- Graves N, Barnett AG, Clarke P. Funding grant proposals for scientific research: retrospective analysis of scores by members of grant review panel. BMJ. 2011;343:d4797. read 10 October 2026. View source
- Marian AJ. What ails the NIH peer review study sections and how to fix the review process of the grant applications. Journal of Cardiovascular Aging. 2023;3(1):11. read 10 October 2026. View on PubMed Central
- Patat F. A statistical study on proposal ranking at ESO. Part I: the pre-meeting phase. arXiv:1805.06981. 2018. read 10 October 2026. View source
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes. Pubrica. https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/
Pubrica Academy. “The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes.” Pubrica, 10 Oct. 2026, pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/.
Pubrica Academy (2026) The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes. Pubrica. Available at: https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/ (Accessed: 15 October 2026).
Pubrica Academy. “The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes.” Pubrica. October 10, 2026. https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/
Pubrica Academy, ‘The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/> accessed 15 October 2026.
Pubrica Academy. The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/
Pubrica Academy, “The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/
@misc{/la_grant_review_panel_decision_making_case_study_2026,
author = {{Pubrica Academy}},
title = {The Restrictions and Processes of Grant Review Decision-Making and Their Effects on Funding Outcomes},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/concepts-definitions/the-restrictions-and-processes-of-grant-review-decision-making-as-well-as-their-effects-on-the-outcome/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Panel-procedure audit for grant funders
This checklist should be used to assess whether the structural procedures of a panel, and not only its stated criteria, are shaping funding outcomes in a predictable direction.
[ ] Document the rating scale's granularity (coarse vs. fine) and what it allows or forecloses [ ] Document the specific ranking/elimination method used to move from ratings to a funding list [ ] Assess whether the ranking method structurally favors consensus-safe or individually-championed proposals [ ] Compare stated review guidelines against which criteria panelists actually discuss in meetings [ ] Check whether funding outcomes vary meaningfully by which reviewers were assigned to a given round
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica advises research teams on how a target funder’s panel structure and ranking procedures are likely to influence review outcomes, assisting applicants in the timing and positioning of submissions.