Contents
Academy · Research Data, Analytics & AI · Common P-Value Misinterpretations
Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements
A widely cited 2016 methodological guide catalogues 25 distinct, documented misinterpretations of p-values and related statistics that remain common in published research. The most frequent is treating the p-value as the probability that the null hypothesis is true rather than as the probability of the data given that hypothesis.
A p-value misinterpretation is a documented, recurring error in the description or use of a p-value’s statistical meaning, as distinct from a disagreement about which threshold or method to apply. The 2016 guide to statistical misinterpretations by Greenland, Senn, Rothman, Carlin, Poole, Goodman and Altman catalogues 25 specific misreadings of p-values, confidence intervals and statistical power that remain common in published research. The errors it documents are instructive because each initially appears to be a reasonable paraphrase of what a p-value means, which explains why each continues to recur despite being incorrect.
Learning objectives
- The most common p-value misinterpretation identified in the methodological literature
- Several other specific misinterpretations from the 2016 catalogue by Greenland et al.
- The correct statement with which each misinterpretation should be replaced
Common misconceptions
- “A p-value of 0.03 means there is a 3% probability that the null hypothesis (no real effect) is true.” Greenland and colleagues identify this misreading as probably the most common and important of the 25 misinterpretations they catalogue: a p-value is the probability of the observed data, or data more extreme, given that the null hypothesis is true; it is not the probability that the null hypothesis is true given the data, which is a different and generally larger quantity that the p-value alone cannot provide.
The most common misinterpretation
The 2016 guide by Greenland and colleagues, published jointly across several journals to maximise its reach, identifies the conflation of “P(data | null hypothesis true)” with “P(null hypothesis true | data)” as probably the most common and consequential p-value misinterpretation in the scientific literature1. A p-value answers only the first of these two questions: it is calculated by assuming that the null hypothesis is true and then determining how likely the observed data, or data more extreme, would be under that assumption1. It cannot by itself answer the second question, namely the probability that the null hypothesis is true given the observed data, because that calculation requires information the p-value does not contain, specifically a prior probability that the null hypothesis was true before the data were collected1. Goodman’s earlier and widely cited “dirty dozen” catalogue of p-value misconceptions makes the same point as its first and foundational entry, indicating that this confusion has been independently identified as the primary misinterpretation by methodological authorities writing a decade apart2.
Other documented misinterpretations
Several of the other 25 misinterpretations catalogued by Greenland and colleagues recur in meta-analysis and systematic review contexts. One is the treatment of a non-significant p-value as proof that no effect exists, whereas a non-significant result is equally compatible with a real effect that the study lacked the statistical power to detect1. A second is the treatment of statistical significance as equivalent to clinical or practical importance, whereas, as discussed in relation to the effect of sample size on p-values, a sufficiently large sample can render a trivial effect statistically significant1. A third, relevant to the interpretation of a confidence interval, is the assumption that a calculated 95% confidence interval has a 95% probability of containing the true effect, whereas the correct frequentist interpretation concerns the long-run behaviour of the method across repeated samples and is not a probability statement about the particular interval calculated1. A fourth is the assumption that two studies’ effects necessarily differ because their point estimates differ, without examining whether the two confidence intervals indicate a statistically meaningful difference between them1.
Correct statements to replace each misinterpretation
For the core misreading, the correct statement is that the p-value gives the probability of the observed data (or data more extreme) under the assumption that the null hypothesis is true, not the probability that the null hypothesis itself is true1. For the “non-significant means no effect” error, the correct statement reports the plausible range of effects given by the confidence interval alongside the non-significant p-value, making clear whether a clinically important effect remains plausible despite the absence of statistical significance3. For the conflation of significance with importance, the correct practice, as Sullivan and Feinn argue directly, is to report the effect size and its confidence interval alongside the p-value rather than allowing the p-value to substitute for a judgment of importance it was not designed to make4. For the confidence-interval probability error, the correct statement describes a 95% confidence interval as a range calculated by a method that would contain the true value in 95% of repeated samples, not as a 95% probability statement about the specific interval calculated1. Replacing each misinterpretation with its correct statement usually requires only a single added clause (reporting the confidence interval, naming the effect size or stating the correct conditional direction) rather than a fundamentally different analysis, which partly explains why these errors are both common and readily corrected once identified.
Reasons for the persistence of well-documented misinterpretations
Greenland and colleagues note that most of the 25 misinterpretations they catalogue have been identified repeatedly in the statistical literature for decades, Goodman’s “dirty dozen” itself dating to 2008 and drawing on earlier critiques; this raises the separate question of why the errors persist so widely despite being well documented12. Part of the explanation is structural: standard statistics training often teaches the calculation of a p-value in detail while devoting comparatively little time to the careful wording required to state what the resulting number does and does not mean, so that correct calculation and correct verbal interpretation are taught asymmetrically. A further part of the explanation is that several of the misinterpretations resemble reasonable paraphrases; the statement “a p-value this low means there’s only a 3% chance this is a fluke,” for instance, appears intuitive but is incorrect in precisely the way Greenland and colleagues describe, which is why such errors frequently pass peer review and editorial checking1.
A practical procedure for identifying these errors in manuscripts
A practical check is to read every sentence in the results and discussion of a manuscript that mentions a p-value or statistical significance and to ask whether it could be rewritten using the phrase “probability of the null hypothesis being true” without changing its intended meaning; if it could, the sentence very probably commits the core misinterpretation that Greenland and colleagues identify as the most common1. A second check is to search the manuscript for the words “no effect,” “no difference” or “no association” and to confirm that each instance is supported by examination of the confidence interval, not solely by a non-significant p-value. A third is to search for any sentence that uses “significant” as a synonym for “important,” “large” or “meaningful” and to confirm that the effect size, not only the p-value, supports that characterisation. Applying these three targeted checks before submission identifies the most consequential misinterpretations described in this article far more reliably than a general proofreading pass, because each check targets the specific wording pattern that the misinterpretation tends to take. The checks are deliberately narrow and mechanical rather than a re-examination of every statistical claim, which makes them practical to apply consistently: a reviewer or co-author unfamiliar with the full catalogue of 25 misinterpretations can still apply them and detect the small number of errors responsible for most of the documented harm. Journals and review bodies that incorporate one or more of these checks into their editorial checklists, rather than relying on individual reviewers to detect such errors ad hoc, establish a more consistent second line of defence than reviewer judgment alone, since a checklist item is applied identically regardless of which reviewer is assigned to a manuscript. A statistics or methods editor assigned to flag these patterns before a manuscript reaches content reviewers, a role adopted by a growing number of journals, detects a meaningful proportion of these errors earlier than a subject-matter reviewer reading for scientific content would, and allows that reviewer to concentrate on the scientific substance of the work rather than dividing a single review between assessing the content and checking the statistical wording.
Key points
Key points
A documented set of p-value misinterpretations, led by the confusion of the probability of the data given the null hypothesis with the probability that the null hypothesis is true, recurs across published research. Each misinterpretation has a correctly stated replacement, which usually requires only that the confidence interval and effect size be reported alongside the p-value rather than a different analysis.
Practical application
Practical application
- Authors should state explicitly what the p-value does and does not indicate. It should be described as reflecting the probability of the data under the null hypothesis, never the probability that the null hypothesis is true.
- The confidence interval should be reported whenever a non-significant result is reported. The interval shows readers whether a clinically important effect remains plausible, rather than allowing “not significant” to be read as “no effect”.
- A p-value should never substitute for a judgment of importance based on effect size. The effect size and its confidence interval should be reported alongside any statement of significance, so that readers can judge magnitude independently of the p-value.
Frequently asked questions
Confusing the probability of the observed data given that the null hypothesis is true with the probability that the null hypothesis itself is true given the data; Greenland and colleagues identify this as probably the most common and important of the 25 misinterpretations they catalogue.
No, not under the standard frequentist interpretation. The correct statement is that the method used to calculate the interval would contain the true value in 95% of repeated samples; this is a statement about the long-run behaviour of the method, not a probability statement about the specific interval calculated.
No. A non-significant result is also compatible with a real effect that the study lacked the power to detect; examining whether the confidence interval still includes clinically important effects is necessary for the correct interpretation of a non-significant finding.
The 2016 paper by Greenland, Senn, Rothman, Carlin, Poole, Goodman and Altman in the European Journal of Epidemiology catalogues 25 specific misinterpretations of p-values, confidence intervals and power, each accompanied by the correct statement.
References
- Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. read 10 October 2026. View on Springer
- Goodman S. A dirty dozen: twelve p-value misconceptions. Seminars in Hematology. 2008;45(3):135-140. read 10 October 2026. View on Seminars in Hematology
- Schünemann HJ, Higgins JPT, Vist GE, et al. Chapter 15: Interpreting results and drawing conclusions. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
- Sullivan GM, Feinn R. Using Effect Size-or Why the P Value Is Not Enough. Journal of Graduate Medical Education. 2012;4(3):279-282. read 10 October 2026. View on JGME
- Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician. 2016;70(2):129-133. read 10 October 2026. View on Taylor & Francis Online
- Fisher RA. Statistical Methods for Research Workers. Oliver & Boyd; 1925. read 10 October 2026. View on Internet Archive
- Deeks JJ, Higgins JPT, Altman DG. Chapter 10: Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
- Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365-376. read 10 October 2026. View on Nature Reviews Neuroscience
- Benjamin DJ, Berger JO, Johannesson M, et al. Redefine statistical significance. Nature Human Behaviour. 2018;2:6-10. read 10 October 2026. View on Nature Human Behaviour
- Higgins JPT, Li T, Deeks JJ. Chapter 6: Choosing effect measures and computing estimates of effect. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements. Pubrica. https://pubrica.com/academy/misinterpretations-in-p-values/
Pubrica Academy. “Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements.” Pubrica, 10 Oct. 2026, pubrica.com/academy/misinterpretations-in-p-values/.
Pubrica Academy (2026) Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements. Pubrica. Available at: https://pubrica.com/academy/misinterpretations-in-p-values/ (Accessed: 15 October 2026).
Pubrica Academy. “Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements.” Pubrica. October 10, 2026. https://pubrica.com/academy/misinterpretations-in-p-values/
Pubrica Academy, ‘Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/misinterpretations-in-p-values/> accessed 15 October 2026.
Pubrica Academy. Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/misinterpretations-in-p-values/
Pubrica Academy, “Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/misinterpretations-in-p-values/
@misc{/la_common_p_value_misinterpretations_2026,
author = {{Pubrica Academy}},
title = {Common Misinterpretations of P Values: A Catalogue of Errors and Correct Statements},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/misinterpretations-in-p-values/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
P-Value Misinterpretation Self-Check
This checklist may be used to screen a draft for the most common p-value misstatements before submission.
[ ] Checked that no sentence states or implies the p-value is the probability the null hypothesis is true [ ] Checked that no non-significant result is described as proving 'no effect' without checking the confidence interval [ ] Checked that no significance statement is used as a stand-in for clinical/practical importance [ ] Checked that no confidence interval is described as a probability statement about that specific interval [ ] Checked that differing point estimates between studies are not called 'different effects' without comparing intervals
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s biostatistics and editorial team screens manuscripts and reviews for this catalogue of common p-value misstatements before submission and replaces each with its statistically correct equivalent.