Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Funnel Plots & Publication Bias

Understanding Funnel Plots: A Key Tool in Meta-Analysis and the Limits of Visual Interpretation

An empirical study in which researchers were asked to judge funnel-plot symmetry visually found that they could not reliably distinguish biased from unbiased plots by inspection. Current methodological guidance therefore pairs the plot with a formal statistical test rather than treating visual inspection as sufficient. This article examines the tests, their limitations and the interpretation of asymmetry.

Pubrica AcademyWritten by a meta-analysis methodologist Reviewed bya biostatistics reviewer Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources10 peer-reviewed

A funnel plot displays each study included in a meta-analysis as a point on a scatter plot, with the study's effect estimate on one axis and a measure of its precision, typically standard error or sample size, on the other. In the absence of bias or genuine heterogeneity, the resulting shape should resemble an inverted funnel: precise, large studies cluster tightly near the pooled effect, while smaller, less precise studies scatter more widely around it. The plot is used mainly to investigate small-study effects, including but not limited to the kind of publication bias where small studies with null or unfavorable results are less likely to be published than small studies with favorable ones.

Learning objectives

  • The original statistical test proposed to formalise what a funnel plot shows visually
  • Why controlled research found visual judgement of funnel-plot asymmetry to be unreliable
  • What funnel-plot asymmetry can and cannot indicate about the cause of small-study effects

Common misconceptions

  • “Visual inspection of a funnel plot is a reliable method of detecting publication bias in a meta-analysis.” Norma Terrin and colleagues’ empirical evaluation, in which researchers were shown real funnel plots and asked to judge their asymmetry, found that visual inspection performed poorly in distinguishing meta-analyses with documented publication bias from those without it, especially when the included studies had broadly similar precision, which is the condition under which asymmetry is most difficult to see even though bias may still be present.

The statistical test that formalised the funnel plot

The 1997 BMJ paper by Matthias Egger, George Davey Smith, Martin Schneider and Christoph Minder proposed what became the most widely used formal test for funnel-plot asymmetry, a linear regression of each study’s standardised effect estimate against its precision, in which a statistically significant slope indicates asymmetry that visual inspection might miss, or might perceive where none is statistically present1. The rank-correlation test of Colin Begg and Madhuchhanda Mazumdar, published in 1994, offered an alternative, non-parametric approach to the same question, and the two tests are often reported together in methodologically rigorous reviews4. A later methodological paper by Jonathan Sterne and Matthias Egger addressed how the choice of precision measure on the vertical axis (standard error, sample size or inverse variance) alters the shape and interpretability of the resulting plot, and recommended standard error as generally the most useful default axis2.

Unreliability of visual judgement of asymmetry

The empirical study by Norma Terrin, Christopher Schmid and Joseph Lau tested how well researchers could detect asymmetry by eye, using real meta-analyses with known or suspected publication bias, and found that visual judgement performed poorly, particularly when the included studies were similar in size and precision, the scenario in which a funnel plot tends to resemble a vertical column rather than a wide-based funnel, irrespective of whether bias is present7. Joseph Lau, John Ioannidis, Norma Terrin, Christopher Schmid and Ingram Olkin followed this with a widely cited paper, “The case of the misleading funnel plot,” which used real-world examples to show how the apparent shape of a funnel plot can suggest a conclusion about bias that more careful statistical analysis does not support, and vice versa8. The comparison of methods for detecting publication bias by Jaime Peters and colleagues similarly found that performance varied with the number of studies and the degree of heterogeneity in the data, supporting the conclusion that no single visual or statistical check is reliable across every scenario a meta-analyst may encounter9.

Asymmetry as distinct from proof of publication bias

The 2011 BMJ guidance paper by Jonathan Sterne and colleagues on interpreting funnel-plot asymmetry, developed on behalf of several major meta-analysis methods groups, made the point explicit: a funnel plot can be asymmetric for several distinct reasons, namely true heterogeneity correlated with study size, data irregularities, chance, or reporting biases including but not limited to classic publication bias, and the plot alone cannot distinguish among them3. The practical recommendation of the paper is that funnel-plot asymmetry should prompt further investigation of which of these explanations fits the specific data set, rather than being reported directly as evidence of publication bias, a distinction that is frequently blurred when meta-analysis results are summarised in discussion sections3. The chapter of the Cochrane Handbook on assessing risk of bias due to missing results reflects the same guidance, recommending asymmetry tests only when a meta-analysis includes at least ten studies, since the tests have low statistical power with fewer studies, rendering any conclusion from a smaller funnel plot, whether visual or statistical, unreliable in either direction5.

A refinement: contour-enhanced funnel plots

One response to the ambiguity between publication bias and other causes of asymmetry is the contour-enhanced funnel plot, which overlays shaded regions corresponding to conventional thresholds of statistical significance (such as p < .01, p < .05 and p < .1) on the standard funnel plot. The methodological paper by Jaime Peters and colleagues proposing this enhancement argued that if missing studies cluster in the non-significant, white area of the plot rather than being distributed more generally across the missing region, the pattern is more consistent with classic publication bias (in which null results remain unpublished) than with other explanations for asymmetry, such as true heterogeneity, which would not be expected to follow statistical-significance boundaries in the same way10. This refinement does not fully resolve the interpretive ambiguity described by Sterne and colleagues, since several of the same alternative explanations can still produce a similar pattern, but it provides reviewers with a more precise and more falsifiable visual tool than the plain funnel plot, and its use has become sufficiently common that several meta-analysis software packages now generate contour-enhanced versions by default alongside the standard plot.

Use of funnel plots within a complete analysis

Good practice, as reflected across this literature, is to present the funnel plot alongside a formal asymmetry test such as Egger’s or Begg’s, to state explicitly how many studies were included (and to withhold strong conclusions below the ten-study threshold), and to discuss plausible alternative explanations for any detected asymmetry rather than defaulting to an account based on publication bias35. The trim-and-fill method of Sander Duval and Richard Tweedie, a complementary technique that estimates and adjusts for the number and outcomes of studies that may be missing because of small-study effects, is sometimes reported alongside the plot as a sensitivity analysis, although it shares the limitation of assuming that the mechanism of missingness is publication bias rather than another cause of asymmetry6. It is also advisable to determine, before any of these checks are run, which outcome and which subset of studies the funnel plot covers, since a meta-analysis reporting several outcomes across the same set of trials should generally produce a separate funnel plot for each outcome rather than one combined plot, because small-study effects and reporting patterns can differ substantially between a trial’s primary and secondary endpoints. A trial team under pressure to report a positive result is, for instance, more likely to report a favourable secondary outcome selectively than to alter its registered primary outcome, a pattern that a single funnel plot averaged across outcome types would obscure. Reviewers applying the Cochrane risk-of-bias framework for missing results are advised to document, for each outcome separately, how many of the included trials reported that outcome, since trials that measured but did not report a given outcome represent a distinct and separately important category of missing evidence that a funnel plot constructed only from trials reporting numerical results cannot capture, irrespective of how symmetric or asymmetric that plot appears5. This point warrants explicit statement because it is easily overlooked: a funnel plot can, by construction, describe only the studies and outcomes that were reported with usable data, so a reviewer reading a symmetric, apparently reassuring plot should still ask separately how the authors of the review searched for and accounted for trials that measured the outcome of interest but never reported it, since that failure leaves no visible trace on the plot, however carefully it is constructed or interpreted. Accordingly, before the shape of any funnel plot is interpreted, the methods section of the review should be checked for the thoroughness of its search for unpublished and non-indexed studies, including trial registries and conference abstracts, since a thorough search narrows the pool of plausible missing studies regardless of the appearance of the resulting plot. PRISMA 2020 requires reviews to report their full search strategy, including any trial registries or grey-literature sources searched, so that a reader evaluating the conclusions drawn from a funnel plot can assess the quality of this search directly rather than accepting the apparent symmetry or asymmetry of the plot without scrutiny.

Key points

Key points

A funnel plot is a starting point for investigating small-study effects rather than a conclusive test: it should be accompanied by a formal asymmetry test such as Egger’s or Begg’s, applied only when at least ten studies are included, and any asymmetry detected should be treated as a prompt to investigate possible causes, including but not limited to publication bias, rather than as direct proof of it.

Practical application

Practical application

  1. A funnel plot should always be accompanied by a formal statistical test. Egger’s regression test or Begg’s rank-correlation test should be reported alongside the plot rather than reliance being placed on visual inspection alone, since empirical research has found visual judgement to be unreliable.
  2. The number of included studies should be checked before conclusions are drawn. In accordance with Cochrane Handbook guidance, asymmetry tests, and the funnel plot itself, should be regarded as unreliable when fewer than ten studies are included.
  3. Where asymmetry appears, its cause should be investigated rather than assumed. Heterogeneity, data irregularities and chance should be considered as explanations alongside publication bias before detected asymmetry is attributed to any single cause in the discussion section.

Frequently asked questions

On its own, it demonstrates only that the smaller and larger studies in the meta-analysis show different average effects; it does not distinguish between several possible causes, including true heterogeneity, data irregularities, chance, or reporting or publication bias, which require further investigation to differentiate.

No. Controlled research testing how well researchers could judge asymmetry by eye found poor performance, especially with studies of similar precision, which is why major methods groups now recommend accompanying the plot with a formal statistical test such as Egger’s or Begg’s.

Cochrane Handbook guidance recommends against relying on funnel-plot asymmetry tests with fewer than ten included studies, because the tests have low statistical power at that sample size and conclusions drawn from them are correspondingly unreliable.

Standard error is generally recommended as the most useful default, though sample size and inverse variance are also used; the choice affects the plot's shape and should be reported explicitly in the methods.

It provides an estimated, adjusted pooled effect under the assumption that missing studies follow a specific pattern consistent with small-study effects, but it does not confirm that publication bias specifically caused any asymmetry, and it should be reported as a sensitivity analysis rather than a correction of known fact.

References

  1. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-634. read 10 October 2026. View on BMJ
  2. Sterne JAC, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. Journal of Clinical Epidemiology. 2001;54(10):1046-1055. read 10 October 2026. View on Journal of Clinical Epidemiology
  3. Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. read 10 October 2026. View on BMJ
  4. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101. read 10 October 2026. View on Biometrics
  5. Page MJ, Higgins JPT, Sterne JAC (eds). Chapter 13: Assessing risk of bias due to missing results. Cochrane Handbook for Systematic Reviews of Interventions, version 6.4. read 10 October 2026. View the Cochrane Handbook
  6. Duval S, Tweedie R. Trim and Fill: A Simple Funnel-Plot-Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis. Biometrics. 2000;56(2):455-463. read 10 October 2026. View on Biometrics
  7. Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology. 2005;58(9):894-901. read 10 October 2026. View on Journal of Clinical Epidemiology
  8. Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597-600. read 10 October 2026. View on BMJ
  9. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-680. read 10 October 2026. View on JAMA
  10. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology. 2008;61(10):991-996. read 10 October 2026. View on Journal of Clinical Epidemiology

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Understanding Funnel Plots: A Key Tool in Meta-Analysis and the Limits of Visual Interpretation. Pubrica. https://pubrica.com/academy/study-guide/research-writing/funnel-plots-meta-analysis/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Funnel Plot Interpretation Checklist

This checklist should be completed before conclusions are drawn from a funnel plot.

[ ] At least 10 studies included before relying on asymmetry tests or visual interpretation
[ ] Precision measure (standard error recommended) on the vertical axis, stated explicitly
[ ] Formal asymmetry test (Egger's and/or Begg's) run and reported alongside the plot
[ ] Alternative explanations for asymmetry considered: heterogeneity, data irregularities, chance, reporting bias
[ ] Avoided stating asymmetry as direct proof of publication bias without further investigation
[ ] Trim-and-fill or other sensitivity analyses reported as supplementary, not as a confirmed correction

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s statistical support team produces funnel plots alongside formal asymmetry tests and assists authors in writing discussion sections that interpret detected asymmetry with appropriate caution.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Funnel plot interpretation supported by formal statistical testing

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.