Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Statistical analysis · Significance and effect size

Statistical Significance and Effect Size: Definitions, Distinctions and Reporting

Statistical significance concerns how surprising the data would be under the null hypothesis; effect size concerns the magnitude of the effect. Either may be present without the other, which is why reporting standards require both.

Pubrica AcademyWritten by a research writing specialist Reviewed bya research methodology reviewer Published15 October 2026 Last checked15 October 2026 Reading time13 minutes Sources9 primary

Distinct questions addressed by each quantity

These two quantities are frequently discussed as though one were a refinement of the other. They are not: they answer different questions, and either may be present without the other.

Table 1. The two questions

Statistical significanceEffect size
The question it answersHow surprising would the observed data be if there were no effect?How big is the effect?
What it depends onThe effect, the sample size and the variability together.The effect and the variability, but not the sample size.
What a large sample does to itEven a trivial difference becomes significant.No change; the same quantity is estimated more precisely.
What it cannot tell youWhether the effect is large enough to matter.Whether the result could easily have arisen by chance.
Reported asA p-value, compared against a threshold specified in advance.A standardised or unstandardised estimate, with a confidence interval.

Definition of a p-value

Greenland and colleagues give the definition carefully, and the conditional clause at the end is the part that matters:

“The P value is then the probability that the chosen test statistic would have been at least as large as its observed value” — “if every model assumption were correct, including the test hypothesis.”1

So it is a statement about the data, computed under an assumption. It is not the probability that the null hypothesis is true — Greenland and colleagues list that reading first among the misinterpretations: “The P value is the probability that the test hypothesis is true;”1

The American Statistical Association's statement puts the same point as a principle: “P-values do not measure the probability that the studied hypothesis is true” “or the probability that the data were produced by random chance alone.”2

And on the specific confusion this page exists to resolve, the ASA is explicit that a p-value does not measure the size of an effect: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.”2

Definition of an effect size

An effect size is an estimate of how large the difference or association is. It comes in two kinds, and the distinction is practical.

Unstandardised effect sizes are in the original units: a mean difference of 4.2 mmHg, a risk difference of 6 percentage points, an odds ratio of 1.8. Where the outcome has units a reader understands, these are almost always the better choice, because they can be judged directly.

Standardised effect sizes are in units of standard deviation, which makes them comparable across outcomes measured on different scales. The Cochrane Handbook notes that a standardised mean difference is “reported in units of SD rather than in units of any of the measurement scales”9 — which is the point of it and also its limitation, since a reader cannot picture a standard deviation of your outcome without being told what one is.

On direction, one thing worth getting right: for a difference measure “a value of 0 represents no difference between the groups”9, and which side means benefit depends entirely on how you ordered the comparison. The Handbook also warns that “the SMD method does not correct for differences in the direction of the scale”9, so scales that run in opposite directions have to be aligned before they are combined.

Cohen’s d and the interpretation of its sign

Cohen’s d is the standardised difference between two means. It comes from Cohen’s work on statistical power4, and the formula is the difference between the two means over the pooled standard deviation — “the standard deviation calculated from both groups as one.”6

d = (M1 − M2) ÷ SDpooled

A claim requiring a stated premise rather than a correction

Cohen’s d is sometimes described as ranging from zero to infinity. The statement is not simply incorrect: Lakens gives the same range, treating d as a magnitude. What is missing is the premise. As computed, d is the difference between two means divided by a pooled standard deviation, so it carries a sign that records which group scored higher; a negative d is a real and interpretable result. The convention is to report the absolute value, and it is |d| that ranges from zero upward. Stating the range without identifying which quantity it describes is what makes the sentence misleading.

To be concrete about it. Because the numerator is a difference, d is signed: “If the value is negative, that simply means that the second mean” was the larger6. In practice “the negative sign is usually not included”6, and Lakens, reporting d as a magnitude, states that it “ranges from 0 to infinity”5

Both of those are correct about different things, which is why the sentence on the old version of this page was misleading rather than false. If you write that d runs from zero upward, write |d|, or say you are giving the absolute value. If you report a signed d, the sign tells your reader which group did better and is worth keeping.

Conventional benchmarks and their limitations

Table 2. Cohen’s conventional benchmarks for d

ValueConventional label
d = 0.2Small
d = 0.5Medium
d = 0.8Large

Attributed to Cohen 1988 by Lakens, who also gives the warning against applying them rigidly.5,4

Those three numbers are quoted everywhere, usually without the warning that comes with them. Lakens, who reports them, says in the same paper that “these values are arbitrary and should not be interpreted rigidly”5, and adds the observation that does most of the work: “Small effect sizes can have large consequences”5 A d of 0.1 on mortality is not trivial.

The stronger objection is empirical. Schäfer and Schwarz examined typical effect sizes across psychological sub-disciplines and study designs and found differences large enough that it is “impossible to apply any global benchmarks”7 — concluding that “neither of these recommendations is currently applicable”7 of Cohen’s own interpretive recommendations now applies.

It is worth noting what the Cochrane Handbook does here: in its chapter on effect measures it offers no small, medium or large thresholds for the standardised mean difference at all9. That silence is itself informative.

So what should you do instead? Lakens gives the answer: “the best way to interpret Cohen's d is to relate it to other effects in the literature.”5 An effect size means something when it sits next to the effects other studies of the same outcome have found.

A worked example

Two groups, a continuous outcome, equal group sizes. The arithmetic below reproduces from the numbers printed, which is the only property that makes a worked example worth printing.

Table 3. Illustrative figures. Not from any study.

InterventionControl
n6060
Mean score at 12 weeks48.644.1
Standard deviation9.28.8

With equal group sizes the pooled standard deviation is the root mean square of the two:

<code>SD_pooled = sqrt((9.2² + 8.8²) / 2)
          = sqrt((84.64 + 77.44) / 2)
          = sqrt(81.04)
          = 9.00

d = (48.6 − 44.1) / 9.00
  = 4.5 / 9.00
  = 0.50</code>

So d = 0.50, which falls on Cohen’s medium benchmark — with every caveat in the previous section attached. Note what the number does not tell you: whether a 4.5-point difference on this scale matters to a patient. That is a clinical judgement, and it needs the scale's minimal important difference, not a standardised effect size.

Verifying a worked example

Two features should be checked in any worked example, including this one. First, the standardised effect should recompute from the printed means and standard deviations; an example that reports d = 2.95 when its own figures imply approximately 1.3 is internally inconsistent. Second, the means should be what they claim to be: within-group change scores are not population means, and they have different standard deviations.

The rationale for reporting both quantities

Reporting standards ask for both, and the APA’s are the most specific about it. Its reporting standards for quantitative research require, in the abstract, that authors “Report findings, including effect sizes and confidence intervals or statistical significance levels.”8 and, in the analysis, “effect-size estimates and confidence intervals on estimates that correspond to each inferential test conducted”8 for each inferential test conducted.

The reason is the asymmetry in Table 1. A p-value with no effect size leaves a reader unable to judge whether the finding matters. An effect size with no interval leaves them unable to judge whether it could easily have arisen by chance. Altman and Bland made the point that covers the other half of this in a single page of the BMJ: “This term wrongly implies that the study has shown that there is no difference,” “whereas usually all that has been shown is an absence of evidence of a difference.” “These are quite different statements.”3

Common errors

Table 4. Faults, and what to do instead

The faultWhy it mattersInstead
Reading a p-value as the probability the null is trueIt is the probability of data at least as extreme as those observed if the null hypothesis and the rest of the model were true; the conditional runs in the opposite direction.The conditioning assumptions should be stated, and the effect estimate reported as the result.
Treating a non-significant result as showing no effectIt is consistent with no effect, with a small effect, and with a real effect the study was too small to detect.The confidence interval should be reported, as it distinguishes between these three possibilities.
Treating significance as importanceWith a sufficiently large sample, a difference too small to be clinically important will reach any chosen threshold.The effect size should be reported, together with an assessment of whether it is large enough to be important.
Quoting d without its sign or without stating that it is absoluted is signed as computed. The statement that it ranges from zero upward is true only of |d|, and stating the range without this premise conceals which group performed better.The signed value should be reported, or it should be stated explicitly that the absolute value is given.
Applying 0.2, 0.5 and 0.8 as fixed rulesTypical effects differ so much between fields and designs that global benchmarks cannot be applied.The effect should be compared with others in the relevant literature.
Reporting a p-value with no estimateA significance claim with no effect and no interval cannot be appraised, compared or pooled.The estimate and interval should be reported, followed by the p-value if required.

Practical application

  1. The effect size and its confidence interval should be reported for every comparison and presented before the p-value.
  2. Unstandardised effect sizes should be used where the outcome has units that readers understand, with standardisation reserved for cases requiring comparability across scales.
  3. The sign of d should be reported, or it should be stated explicitly that the absolute value is reported.
  4. The standardised effect should be verified to recompute from the reported means and standard deviations.
  5. The standard deviation used in the denominator — pooled, control-group or pre-test — should be stated, because the choice changes the value.
  6. Effects should be compared with others in the relevant literature rather than with the values 0.2, 0.5 and 0.8.
  7. A non-significant result should not be described as showing no effect; the interval should be reported and interpreted.

Key points

Statistical significance concerns how surprising the data would be under the null hypothesis; effect size concerns the magnitude of the effect. A p-value is the probability of data at least this extreme if the null and the whole model were true — not the probability the null is true, and not a measure of magnitude. Cohen’s d is the mean difference over the pooled standard deviation, and it is signed: the range from zero upward belongs to |d|. The 0.2, 0.5 and 0.8 benchmarks are conventions their own reporters call arbitrary, and typical effects differ enough between fields that global benchmarks do not apply. Both should be reported, with the estimate presented first.

They address different questions. Statistical significance concerns how surprising the data would be if there were no effect; effect size concerns the magnitude of the effect. A very small difference can be highly significant in a large sample, and a large difference can be non-significant in a small one. Neither substitutes for the other, which is why reporting standards require both.

Greenland and colleagues define it carefully: it is the probability of data at least as extreme as those observed, computed on the assumption that the null hypothesis and the rest of the statistical model are correct. The conditional clause is frequently omitted. It is not the probability that the null hypothesis is true, nor is it a measure of the size of the effect.

Yes. d is the difference between two means divided by the pooled standard deviation, so it carries a sign indicating which group scored higher, and a negative d is a valid and interpretable result. By convention the absolute value is reported, and it is |d| that ranges from zero upward. A statement that d ranges from zero to infinity is true of the absolute value but misleading without that premise.

This depends on the field and the outcome rather than on a table. Cohen’s 0.2, 0.5 and 0.8 are the conventional benchmarks, and Lakens, who reports them, also warns that they are arbitrary and should not be interpreted rigidly. Schäfer and Schwarz found differences between sub-disciplines large enough to render global benchmarks inapplicable. The more useful question is how the effect compares with others in the relevant literature.

No; preventing this error is the purpose of the distinction. Significance is a statement about how unusual the data are under the null hypothesis, and it depends heavily on sample size. A trial of fifty thousand people can return p < 0.001 for a difference on which no clinician would act. Importance is a judgement about the effect size in context, and it must be argued rather than inferred from a p-value.

Both should be reported, with the effect size first. The APA’s reporting standards ask for effect-size estimates and confidence intervals corresponding to each inferential test, and for effect sizes in the abstract alongside significance levels. The Academy page on the null hypothesis discusses in greater detail what the p-value does and does not permit an author to conclude.

References

  1. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31:337–350. doi:10.1007/s10654-016-0149-3 read 15 October 2026. View on European Journal of EpidemiologyThe reference work on how p-values are misread. The definitions and the two misinterpretations quoted on this page are taken from it verbatim.
  2. Wasserstein RL, Lazar NA. The ASA’s Statement on p-Values: Context, Process, and Purpose. The American Statistician. 2016;70(2):129–133. doi:10.1080/00031305.2016.1154108 read 15 October 2026. View on The American StatisticianThe American Statistical Association’s formal statement on p-values. Its numbered principles are quoted on this page as published.
  3. Altman DG, Bland JM. Statistics notes: Absence of evidence is not evidence of absence. BMJ. 1995;311(7003):485. doi:10.1136/bmj.311.7003.485 read 15 October 2026. View on The BMJOne page, and the clearest statement of what a non-significant result does and does not license you to say.
  4. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: L. Erlbaum Associates, 1988. read 15 October 2026. View on a library catalogue recordThe work in which d and its conventional benchmarks were set out. No copy or excerpt could be opened, so nothing is quoted from Cohen directly; the benchmarks below are cited to sources that attribute them to him.
  5. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology. 2013;4:863. doi:10.3389/fpsyg.2013.00863 read 15 October 2026. View on Frontiers in PsychologyAn open-access primer on calculating and reporting effect sizes. The source both for the statement that d ranges from zero upward and for the warning that Cohen's benchmarks are arbitrary and should not be applied rigidly.
  6. University of North Texas. Quantitative Analysis in Exercise and Sport Science, chapter on effect size. UNT Open Books. read 15 October 2026. View on UNT Open BooksAn open textbook giving the formula for d and stating plainly that a negative value records direction. Cited for the sign, which Lakens does not address.
  7. Schäfer T, Schwarz MA. The Meaningfulness of Effect Sizes in Psychological Research: Differences Between Sub-Disciplines and the Impact of Potential Biases. Frontiers in Psychology. 2019;10:813. doi:10.3389/fpsyg.2019.00813 read 15 October 2026. View on Frontiers in PsychologyFinds very large differences in typical effect sizes between sub-disciplines and study designs, and concludes that this makes global benchmarks inapplicable. The strongest available source against quoting small, medium and large as universal.
  8. American Psychological Association. Journal Article Reporting Standards for Quantitative Research (JARS–Quant), Table 1. APA Style. read 15 October 2026. View on APA StyleAn association-level reporting standard that names effect sizes specifically and pairs them with significance in the same requirement, which is what makes it the right citation for reporting both.
  9. Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane, 2024. read 15 October 2026. View on CochraneOffers no small, medium or large thresholds for the standardised mean difference at all, which is itself evidence that the benchmarks are not universal. Also the source for the direction of a difference measure being set by the order of the comparison.

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s biostatistics team specifies analyses and reports them to the applicable standard: effect estimates with intervals, the denominator stated, and conclusions confined to what the analysis supports. Where a reviewer has requested the addition of effect sizes to a manuscript that reported only p-values, this constitutes a contained piece of work.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.