Effect Size Versus Statistical Significance: The Limits of the P Value

Statistical significance (typically p < 0.05) indicates a result is unlikely under the null hypothesis. Effect size is a standardised measure of the magnitude of a difference or relationship, independent of sample size. A large enough sample can make a trivial effect statistically significant, which is why effect size, not significance alone, should drive judgements of practical importance.

Statistical significance of trivial effects in large samples

Sullivan and Feinn illustrate the point directly: statistical significance is a function of both the size of an effect and the size of the sample, so as sample size grows, increasingly small and practically trivial differences become statistically significant1. A p-value therefore cannot, by itself, tell a reader whether an effect is large enough to matter in practice — only whether it is larger than chance would be expected to produce, at the given sample size.

The standardised effect-size measures in common use

Standardised effect-size measures express the magnitude of a difference or relationship independent of the original units and the sample size, letting results be compared across studies. Common examples include Cohen's d (the standardised mean difference between two groups), the correlation coefficient r, and odds ratios or risk ratios for binary outcomes1. Conventional benchmarks exist (for example, Cohen’s approximate guide of small/medium/large for d), but these are general heuristics, not substitutes for judging an effect’s size against what matters clinically in the specific field.

Effect sizes in current reporting guidance

Sullivan and Feinn argue, consistent with guidance from bodies such as the American Psychological Association, that reporting an effect size alongside the p-value is now expected practice, precisely because it lets a reader judge practical or clinical significance directly rather than inferring it — incorrectly — from statistical significance alone1. A results section that reports only p-values, without accompanying effect sizes and confidence intervals, leaves the reader unable to judge whether a “significant” finding is important in practice.

Key points

Practical application

Frequently asked questions

@misc{effect_size_vs_statistical_significance_2026,
  author       = {{Pubrica Academy}},
  title        = {Effect Size Versus Statistical Significance: The Limits of the P Value},
  howpublished = {Pubrica},
  year         = {2026},
  month        = {oct},
  url          = {https://pubrica.com/academy/concepts-definitions/effect-size-vs-statistical-significance/},
  note         = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}

[ ] Standardised effect size reported for every significant result
[ ] Effect-size measure matches the outcome type and study design
[ ] Minimum clinically important difference stated, ideally pre-specified
[ ] Large-sample significant results checked against effect size for practical importance
[ ] Confidence interval reported alongside the effect size