Contents
Academy · Research Data, Analytics & AI · Problems with Generic Interpretation
Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability
A peer-reviewed methodological critique argues that field-specific effect-size benchmarks, and not only Cohen’s original generic scale, are hazardous shortcuts, because the distribution of effect sizes within any field is itself shaped by publication bias, study quality and measurement reliability rather than by a stable underlying standard of what constitutes a meaningful effect.
A generic effect-size interpretation applies a single fixed benchmark, such as Cohen’s 0.2/0.5/0.8 or a similar convention, to judge the magnitude of a result without reference to what the specific outcome measure, population or research design represents. The problems with this practice extend beyond the frequently repeated caution that Cohen’s thresholds were originally arbitrary. A 2021 methodological critique further argues that the commonly proposed remedy, substituting field-specific benchmarks derived from the distribution of effect sizes already published in a given area, carries serious and underappreciated problems of its own, because that published distribution is itself distorted by factors unrelated to the magnitude of a meaningful effect.
Learning objectives
- Why Cohen’s original generic benchmarks were never intended as universal measures of importance
- The argument against substituting field-specific effect-size distributions as a remedy
- The effect of reliability and measurement issues on the meaning of an SMD, independent of any benchmark
Common misconceptions
- “A field-specific benchmark, derived from the distribution of effects published in that research area, resolves the problem of Cohen’s generic thresholds.” A 2021 peer-reviewed methodological paper argues the opposite: because the distribution of published effect sizes within a field is shaped by publication bias, inconsistent study quality and variable measurement reliability, that distribution does not represent a stable, trustworthy standard of ‘typical’ or ‘important’ magnitude. Substituting a field-specific benchmark for Cohen’s generic one may therefore introduce a different, still misleading, standard rather than a more valid one.
Cohen’s own caveat regarding his generic thresholds
Cohen introduced his small/medium/large conventions explicitly as a fallback for researchers with no better basis for choosing a target effect size, describing the specific numerical values as a necessary but admittedly arbitrary convention rather than an empirically grounded standard of importance1. In practice, a fixed numerical threshold, once published, tends to be applied far beyond the narrow, cautious role its author intended, and comes to be treated as a settled standard of what constitutes a meaningful effect across outcomes, populations and measurement instruments quite different from the general behavioural-science context Cohen had in mind. Lakens’s primer on effect-size reporting documents this drift directly, noting that researchers frequently cite Cohen’s benchmarks as though they were a validated standard rather than the acknowledged convention Cohen himself described2.
Limitations of field-specific benchmarks as an alternative
The apparently natural remedy, replacing Cohen’s generic scale with a benchmark derived from the distribution of effect sizes already published in a specific field, carries a distinct problem of its own. A 2021 peer-reviewed critique argues that this approach implicitly treats the published distribution of effect sizes as a trustworthy proxy for the true distribution of meaningful effects, whereas that published distribution is shaped by publication bias towards larger, statistically significant findings, by inconsistent study quality across the included literature, and by the particular mix of populations and outcome measures that happened to be studied and published in that field3. A field-specific benchmark therefore answers the question “what is typical among published effects in this literature”, which differs from, and may be considerably more biased than, the question “what magnitude would be clinically or practically meaningful”; the paper’s authors argue that conflating the two produces an interpretive standard that merely exchanges one arbitrary benchmark for another that is less obviously arbitrary but not necessarily more valid3. Button and colleagues’ analysis of statistical power in neuroscience raises a related, compounding concern: fields with a documented history of small, underpowered studies tend to produce inflated published effect-size estimates, because only the larger, often overestimated effects from small samples reach statistical significance and are subsequently published; a field-specific benchmark built from that literature consequently inherits the same inflation4.
Distortion of effect-size values by measurement reliability
A further, independent problem is that the numerical value of a standardized mean difference (SMD) is mechanically sensitive to the reliability of the outcome measure used to calculate it: holding the true underlying effect constant, a less reliable (noisier) measurement instrument produces a larger pooled standard deviation in the denominator of the SMD, which mechanically reduces the resulting SMD regardless of whether the true clinical effect has changed2. Consequently, two studies of an identical underlying effect, one using a well-validated, reliable instrument and the other a noisier, less validated one, can produce materially different SMD values solely as an artefact of measurement quality. This undermines the premise that any single numerical threshold, generic or field-specific, can be applied consistently across studies without first establishing whether the underlying measures are comparable in reliability. The appropriate response to these compounding problems is the same in each case: the effect size should be reported with its confidence interval, the specific population and instrument from which it was derived should be described, and any judgment of importance should be grounded in a minimally important difference established directly for that outcome measure rather than in any borrowed numerical threshold, generic or field-specific.
Further distortion of comparisons by population and study-design differences
Beyond publication bias and measurement reliability, the magnitude of an SMD is also sensitive to characteristics of the study population that are unrelated to the clinical importance of the underlying intervention. A trial enrolling a more homogeneous population, for instance patients selected within a narrow age range or with a tightly defined disease severity, produces a smaller pooled standard deviation across both arms than a trial enrolling a broader, more heterogeneous population; because that pooled standard deviation forms the denominator of the SMD, the more homogeneous trial will tend to show a larger SMD for an identical underlying treatment effect2. This is a well-documented property of standardized effect sizes, sometimes termed the “restriction of range” problem. It implies that comparing SMDs across studies conducted in differently selected populations, which is routine in meta-analysis, can make interventions appear more or less effective than they are, solely as an artefact of each study’s enrolment criteria and independently of any benchmark used to label the result4.
Recommended practice for review authors
Taken together, these compounding problems (an admittedly arbitrary generic convention, a field-specific benchmark distorted by publication bias, an SMD denominator sensitive to measurement reliability, and a numerator-denominator relationship sensitive to population homogeneity) point towards the same practical response rather than towards abandoning effect-size reporting altogether. Review authors should report the effect size with its full confidence interval, describe the population, instrument and study designs that produced it in sufficient detail for a reader to judge comparability, and ground any statement of practical importance in a minimally important difference established directly for the specific outcome and population in question wherever one exists, rather than in a borrowed numerical threshold of any kind35. Where no such validated benchmark exists, the Handbook’s guidance to compare against directly similar prior studies, rather than against a generic or field-average convention, remains the most defensible available option, provided that the limitations of the comparison are stated explicitly. None of these problems argues for abandoning quantitative synthesis; rather, they argue for treating every borrowed number, whether a Cohen threshold, a field average or a prior trial’s effect size, as context whose provenance must be verified before it is used to judge a new result, in the same way that a cited claim requires its source to be checked before it is repeated. A reviewer evaluating a submitted meta-analysis can apply this provenance check directly: when a manuscript states that its effect size is “consistent with previous research in this area”, the reviewer may reasonably ask which prior studies the comparison draws on, whether those studies share a comparable population and measurement instrument, and whether the comparison was determined before or after the new study’s result was known, since a comparison chosen post hoc is more likely to have been selected because it supports the new finding. The same scrutiny should be applied to a manuscript’s claims about prior literature, not only to its numerical benchmarks: a statement that an effect is “broadly consistent with the existing evidence base” warrants the same request for specifics (which studies, which populations, which instruments) as a statement that borrows a bare numerical threshold, because both serve the same interpretive function of anchoring a new result against an external standard whose comparability is uncertain. Requiring a specific citation and a brief statement of comparability, rather than accepting a general reference to “the literature”, obliges the author to perform the comparability check that the sentence implicitly claims has already been carried out. Peer reviewers who consistently question such phrasing, and ask authors to name the comparison studies explicitly, may contribute more to the interpretive rigour of the published literature than almost any other single, repeatable intervention available to an individual reviewer within the normal scope of a standard review.
Key points
Key points
Generic effect-size benchmarks are problematic not only because Cohen’s original thresholds were an acknowledged convention, but also because the obvious alternative, a field-specific benchmark, inherits publication bias and measurement-reliability artifacts from the literature on which it is based. No borrowed numerical threshold is therefore a reliable substitute for a benchmark established directly for the specific outcome measure in question.
Practical application
Practical application
- Any borrowed benchmark, whether generic or field-specific, should be treated as provisional. Neither Cohen’s original thresholds nor a field-average benchmark reliably represents the magnitude that is clinically meaningful for a specific outcome.
- The reliability of the measurement instrument should be checked before SMDs are compared across studies. A less reliable instrument mechanically reduces the SMD independently of the true underlying effect; comparisons of SMDs across studies using different instruments should therefore state this caveat.
- Judgments of importance should be based on a minimally important difference for the specific outcome. Where such a difference has been established for the outcome measure, it should be used in place of any borrowed generic or field-specific threshold.
Frequently asked questions
Not necessarily. A 2021 methodological critique argues that field-specific benchmarks inherit publication bias and inconsistent study quality from the literature on which they are based, which can make them a different, but not more valid, standard than Cohen’s generic convention.
A less reliable (noisier) outcome measure increases the pooled standard deviation used in the denominator of the SMD, which mechanically reduces the SMD even when the true underlying effect is unchanged; cross-study comparisons of SMDs are therefore sensitive to instrument quality.
Studies with smaller or null effects are less likely to be published; the distribution of effect sizes visible in the published literature is therefore skewed towards larger values, and a benchmark built from that distribution inherits the same skew.
A minimally important difference established directly for the specific outcome measure in question, reported alongside the effect size and its confidence interval, rather than a generic or field-average threshold.
References
- Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum Associates; 1988. read 10 October 2026. View on routledge.com
- Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology. 2013;4:863. read 10 October 2026. View on Frontiers
- Bakker M, et al. Denouncing the use of field-specific effect size distributions to inform magnitude. PeerJ. 2021;9:e11383. read 10 October 2026. View on PeerJ
- Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365-376. read 10 October 2026. View on Nature Reviews Neuroscience
- Sullivan GM, Feinn R. Using Effect Size-or Why the P Value Is Not Enough. Journal of Graduate Medical Education. 2012;4(3):279-282. read 10 October 2026. View on JGME
- Hedges LV, Olkin I. Statistical Methods for Meta-Analysis. Academic Press; 1985. read 10 October 2026. View on elsevier.com
- Higgins JPT, Li T, Deeks JJ. Chapter 6: Choosing effect measures and computing estimates of effect. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
- Schünemann HJ, Higgins JPT, Vist GE, et al. Chapter 15: Interpreting results and drawing conclusions. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
- Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. read 10 October 2026. View on BMJ
- Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta-Analysis. John Wiley & Sons; 2009. read 10 October 2026. View on wiley.com
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability. Pubrica. https://pubrica.com/academy/interpretation-of-generic-effect-size/
Pubrica Academy. “Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability.” Pubrica, 10 Oct. 2026, pubrica.com/academy/interpretation-of-generic-effect-size/.
Pubrica Academy (2026) Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability. Pubrica. Available at: https://pubrica.com/academy/interpretation-of-generic-effect-size/ (Accessed: 15 October 2026).
Pubrica Academy. “Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability.” Pubrica. October 10, 2026. https://pubrica.com/academy/interpretation-of-generic-effect-size/
Pubrica Academy, ‘Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/interpretation-of-generic-effect-size/> accessed 15 October 2026.
Pubrica Academy. Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/interpretation-of-generic-effect-size/
Pubrica Academy, “Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/interpretation-of-generic-effect-size/
@misc{/la_generic_effect_size_interpretation_problems_2026,
author = {{Pubrica Academy}},
title = {Why Generic Effect Size Interpretations May Be Problematic: Benchmarks, Bias and Measurement Reliability},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/interpretation-of-generic-effect-size/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Effect-Size Benchmark Caution Note
This checklist should be applied before any benchmark is cited to judge the importance of an effect size.
[ ] Identified whether the benchmark being used is generic (Cohen) or field-specific [ ] Checked whether the field-specific benchmark's source literature is subject to publication bias [ ] Checked the reliability/validation status of the outcome measurement instrument used [ ] Searched for an established minimally important difference specific to this outcome [ ] Reported the effect size with its confidence interval rather than a bare benchmark comparison
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s biostatistics team bases effect-size interpretation on outcome-specific minimally important differences and identifies measurement-reliability or publication-bias concerns before a borrowed benchmark is applied.