Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Sample Size and P-Values

Sample Size and the Interpretation of P Values: Standard Error, Statistical Power and Effect Size

A sufficiently large sample makes a p-value sensitive to even a trivially small, clinically meaningless effect, whereas a small, underpowered study may equally fail to detect a real and clinically important effect. The same p-value threshold therefore carries very different evidentiary weight depending on the sample size on which it is based.

Pubrica AcademyWritten by a biostatistics writer Reviewed bya meta-analysis methodologist Published10 October 2026 Last checked10 October 2026 Reading time7 minutes Sources10 peer-reviewed

A p-value is calculated from a test statistic that incorporates both the size of the observed effect and the precision of its estimate, and that precision is directly determined by sample size: holding the true effect constant, a larger sample produces a smaller standard error, which moves the test statistic further from zero and lowers the p-value, even though the true effect is unchanged. This single mechanical fact has two distinct practical consequences, operating in opposite directions, both of which distort naive interpretation of p-values if sample size is ignored: a sufficiently large study can yield a very small p-value for an effect too small to matter clinically, and a sufficiently small, underpowered study can yield a large, non-significant p-value despite the presence of a real and clinically important effect.

Learning objectives

  • The mathematical relationship between sample size, standard error and the p-value
  • Documented evidence of how underpowered studies in a research field produce misleading and unreplicable significant findings
  • Why the same p-value threshold should be interpreted differently according to the sample size and statistical power on which it is based

Common misconceptions

  • “A statistically significant p-value from a small study is just as trustworthy as one from a large study.” Button and colleagues’ widely cited analysis of statistical power in neuroscience found a median statistical power across the field of only about 21%, indicating that a typical study in that literature had approximately a one-in-five chance of detecting a real effect where one existed. The analysis also documented that low-powered studies which cross the significance threshold tend to overestimate the true effect size substantially, a measurable distortion rather than a theoretical concern.

The test statistic underlying most p-value calculations, including the Z-statistic that RevMan (Review Manager) reports for a pooled meta-analysis effect, is computed by dividing the observed effect by its standard error, and the standard error is itself a function of sample size, decreasing as sample size increases1. Consequently, the same observed effect size will produce a smaller p-value as the sample size increases, purely as a mechanical consequence of the reduction in standard error and independently of whether the true effect is large enough to be of practical importance. Sullivan and Feinn's paper on effect size presents this point as the central reason for reporting effect size alongside, rather than instead of, a p-value: because sample size alone can drive a p-value below 0.05, the p-value cannot by itself distinguish a large, important effect from a small, unimportant one once the sample is sufficiently large2.

Statistical significance of trivial effects in large samples

Greenland and colleagues' guide to statistical misinterpretations identifies the confusion of “statistically significant” with “clinically or practically important” as one of the most common and consequential errors in applied statistics, because a sufficiently large sample can cause an arbitrarily small, substantively meaningless effect to cross the conventional 0.05 threshold3. In a meta-analysis, pooling a large total number of participants across many included studies can produce precisely this pattern: a pooled mean difference too small to matter to any individual patient may still be statistically significant solely because the pooled sample is large. This is one of the reasons the Cochrane Handbook asks review authors to interpret the magnitude of the pooled effect and its confidence interval, not merely whether the p-value crosses 0.054.

Distortion of significant findings in small samples

At the opposite extreme, Button and colleagues' 2013 analysis of statistical power across the neuroscience literature calculated a median statistical power of only about 21% across the studies surveyed, indicating that a typical study in that field had approximately a one-in-five chance of detecting a real effect where one existed5. Low power produces a specific, documented distortion among the studies that cross the significance threshold: because an underpowered study can reach significance only if its observed effect is unusually large relative to the true effect, the significant results that emerge from a low-powered literature are systematically inflated estimates of the true effect size, a phenomenon the authors describe in detail as part of the broader reliability problem created by small samples5. Accordingly, a significant p-value from a small, underpowered study warrants more caution, not less, regarding the accuracy of the reported effect size than the same p-value from a larger, well-powered study, which is the reverse of the informal weighting of significance by sample size that is often applied.

Implications for the power of pooled meta-analyses

Pooling studies in a meta-analysis increases the effective total sample size behind the pooled estimate, which is one of the principal statistical reasons for conducting a meta-analysis: it can detect a real effect that no single included study was individually powered to detect2. Valentine, Pigott, and Rothstein's primer on statistical power for meta-analysis emphasizes, however, that this benefit is not automatic, because the power of a meta-analysis depends jointly on the number of studies, the sample size of each study and the degree of heterogeneity between studies; heterogeneity in particular reduces power by introducing an additional source of variance that a fixed-effect model does not incorporate6. A meta-analysis pooling many small, heterogeneous studies therefore does not automatically attain the power of a single large, well-conducted trial of equivalent total sample size, and review authors should calculate, or at least qualitatively assess, the actual power of the pooled analysis rather than assume that combining studies has resolved the power limitations of each individual study.

A practical rule for interpreting p-values in relation to sample size

A simple, generalizable practice follows from the mechanics described above: a p-value should never be interpreted without simultaneously checking the sample size (or total pooled N) and the effect size from which it was calculated. A very small p-value from a very large sample should prompt an assessment of whether the effect size is large enough to matter, using a field-appropriate benchmark or an established minimally important difference, before it is treated as evidence of a clinically consequential effect3. A non-significant p-value from a small sample should prompt an assessment of the width of the confidence interval and the statistical power of the study, since a wide confidence interval spanning both a null and a clinically important effect indicates that the study was inconclusive, not that it found evidence of no effect5. Applying this cross-check consistently addresses both of the sample-size-driven distortions described in this article through a single, repeatable practice. The direction of the required caution, however, depends on which distortion applies: for a significant result from a large sample, practical importance should be checked, whereas for a non-significant result from a small sample, power and the confidence interval should be checked. Conflating the two checks, for example by scrutinizing the effect size of a large study while accepting the null result of a small study at face value, reintroduces the asymmetric treatment of sample size that this section seeks to correct. Incorporating both checks into a standard routine for reading results, rather than applying whichever comes to mind for a given study, prevents sample size from distorting a reader's conclusions in either direction, at the cost of examining two additional numbers, the sample size and the effect size, whenever a p-value is reported. Teaching this practice explicitly in a methods training session or journal club, rather than assuming that it is acquired implicitly through reading the literature, justifies the small additional time required, because the two distortions run in opposite directions and are each sufficiently counterintuitive that neither is reliably learned through exposure to the literature alone. A journal club that examines one example of each distortion side by side, a large trial with a very small but significant effect and a small trial with a large but non-significant effect, tends to consolidate both lessons more durably than either example alone, because the contrast makes the underlying mechanism, the effect of sample size on the standard error, concrete. Retaining both worked examples as a reusable teaching pair, with a brief written note of each example and its exact numbers, also allows the lesson to be revisited quickly whenever a new manuscript or review raises the same issue in a different form, without reconstructing either calculation from memory.

Key points

Key points

Sample size affects the p-value through the standard error: a large sample can render a trivial effect statistically significant, and a small, underpowered study that crosses the significance threshold is likely to overestimate the true effect size substantially. The same p-value therefore carries different evidentiary weight depending on the sample size on which it is based.

Practical application

Practical application

  1. A p-value should always be interpreted alongside the observed effect size and the sample size. A significant result from a very large sample may represent a trivial effect; the effect size itself should be examined before significance is treated as evidence of importance.
  2. The effect-size estimate from a significant result in a small, underpowered study should be treated with particular caution. Low-powered studies that cross the significance threshold tend to overestimate the true effect, a documented pattern rather than a hypothetical risk.
  3. Statistical power or sample size should be reported alongside significance results where feasible. This allows readers to judge how much weight a given significant or non-significant result should carry.

Frequently asked questions

Yes, if the sample size is large enough. The p-value calculation is mechanically sensitive to sample size through the standard error, so a trivial effect can cross the 0.05 threshold purely because the sample is large.

No. A small, underpowered study may simply lack the statistical power to detect a real effect that exists, so a non-significant result from such a study is weak evidence of 'no effect' rather than proof of it.

Because an underpowered study can only reach statistical significance when its observed effect happens to be unusually large relative to the study's precision, the subset of significant results from a low-powered literature is systematically inflated, as documented in Button and colleagues' analysis of neuroscience research.

The effect size itself, its confidence interval, and ideally the sample size or statistical power behind the estimate, since the p-value alone does not distinguish a large sample's trivial effect from a genuinely important one.

References

  1. Deeks JJ, Higgins JPT, Altman DG. Chapter 10: Analysing data and undertaking meta-analyses. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
  2. Sullivan GM, Feinn R. Using Effect Size-or Why the P Value Is Not Enough. Journal of Graduate Medical Education. 2012;4(3):279-282. read 10 October 2026. View on JGME
  3. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. read 10 October 2026. View on Springer
  4. Schünemann HJ, Higgins JPT, Vist GE, et al. Chapter 15: Interpreting results and drawing conclusions. Cochrane Handbook for Systematic Reviews of Interventions. read 10 October 2026. View on training.cochrane.org
  5. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14(5):365-376. read 10 October 2026. View on Nature Reviews Neuroscience
  6. Valentine JC, Pigott TD, Rothstein HR. How Many Studies Do You Need? A Primer on Statistical Power for Meta-Analysis. Journal of Educational and Behavioral Statistics. 2010;35(2):215-247. read 10 October 2026. View on SAGE Journals
  7. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560. read 10 October 2026. View on BMJ
  8. IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology. 2014;14:25. read 10 October 2026. View on BMC
  9. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum Associates; 1988. read 10 October 2026. View on routledge.com
  10. Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician. 2016;70(2):129-133. read 10 October 2026. View on Taylor & Francis Online

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Sample Size and the Interpretation of P Values: Standard Error, Statistical Power and Effect Size. Pubrica. https://pubrica.com/academy/sample-size-in-interpretation-of-p-values/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Sample-Size-Aware P-Value Reading Note

This checklist should be applied before a conclusion is drawn from any reported p-value.

[ ] Checked the sample size (or total pooled N) behind the reported p-value
[ ] Checked the actual effect size and confidence interval, not just the p-value
[ ] For a significant result from a large sample: assessed whether the effect size is practically meaningful
[ ] For a significant result from a small/underpowered study: treated the effect-size estimate with extra caution
[ ] For a non-significant result from a small study: checked whether low power, not absence of effect, explains it

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s biostatistics team interprets every reported p-value in relation to its underlying sample size and power, identifying where a significant or non-significant finding may be an artefact of sample size rather than a reliable signal.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Pubrica can assess whether a reported p-value reflects a meaningful effect or primarily the size of the sample.

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.