Contents
Academy · Research Data, Analytics & AI · Detection Limits of Funnel Plots
Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations
Publication bias cannot always be detected using a funnel plot. Researchers asked to identify publication bias visually from simulated funnel plots achieved an accuracy of about 53%, statistically indistinguishable from chance, and formal statistical tests perform little better when a meta-analysis includes fewer than about ten studies.
A funnel plot is often described as a means of detecting publication bias in a meta-analysis, but ‘detection’ overstates what it offers. The plot shows whether the pattern of results across studies of different sizes is consistent with a body of evidence from which nothing is missing; an asymmetric pattern is consistent with, but does not prove, missing studies. Several studies have directly tested how reliably such detection works in practice, both for human readers interpreting a chart visually and for the formal statistical tests developed to replace visual judgement, and both lines of evidence support the same conclusion: publication bias cannot always, or even usually, be detected in this way.
Learning objectives
- The exact accuracy rate found when researchers were tested on identifying publication bias from funnel plots
- Why formal statistical tests for funnel-plot asymmetry also have a documented ceiling on their reliability
- What a funnel plot can responsibly be used to say, instead of a yes-or-no detection claim
Common misconceptions
- “A funnel plot is a reliable detection tool for publication bias.” Terrin, Schmid, and Lau’s 2005 empirical study in the Journal of Clinical Epidemiology found that researchers’ accuracy in visually identifying publication bias from simulated funnel plots was 53%, which the authors describe as practically identical to chance for a binary judgement.
Findings of a direct test of funnel-plot accuracy
The most direct test of whether funnel plots allow researchers to detect publication bias was conducted by Terrin, Schmid, and Lau, who published their findings in the Journal of Clinical Epidemiology in 20051. The investigators constructed a set of simulated funnel plots, some from data into which publication bias had been deliberately introduced and some without it, and asked experienced reviewers to judge from the plot alone whether bias was present. The accuracy of these judgements was 53%, a figure the authors describe as practically identical to chance for a task with only two possible answers1. Lau and colleagues’ 2006 BMJ editorial, published the following year, cited this finding as direct evidence for a broader argument: visual inspection of the shape of a funnel plot is not a dependable method for detecting publication bias, however intuitive the inverted-funnel shape may make the underlying logic appear2.
The same 2006 editorial reported a related finding on how funnel plots were used in practice at the time: of 47 systematic reviews with quantitative synthesis published in the BMJ between July 2003 and June 2005, 20 used a funnel plot to assess publication bias, and of those 20, 14 drew a categorical conclusion from the plot without questioning whether that conclusion was warranted2. The discrepancy between a 53% accuracy rate and the confidence with which many published reviews drew conclusions from this kind of visual judgement is the central problem identified by this body of research.
Limitations of formal statistical tests
Replacing visual judgement with a formal statistical test, such as Egger’s regression test or Begg and Mazumdar’s rank correlation test, reduces the problem but does not eliminate it, because these tests have their own, separately documented limitations in power. Sterne, Gavaghan, and Egger’s 2000 study of publication and related bias found that formal tests for small-study effects routinely had low statistical power in real meta-analyses, and therefore frequently failed to detect genuine asymmetry when it was present3. Begg and Mazumdar’s own 1994 evaluation of their rank correlation test reported the same underlying issue: power depends heavily on the number of studies and the variation in their sizes, so a test applied to a small or homogeneous set of studies can return a reassuring, non-significant result that reflects the limited sensitivity of the test rather than a genuine absence of bias4.
Ioannidis and Trikalinos’s 2007 paper introduced a conceptually different test, which examines whether a body of studies contains an excess of statistically significant findings rather than asymmetry in effect size against precision, specifically because asymmetry-based tests missed certain patterns of selective reporting that this alternative approach could sometimes identify5. The need for a second, structurally different test is itself evidence that no single statistical method, let alone visual inspection of the plot, reliably detects every form that publication bias can take. Sterne and colleagues’ 2011 BMJ methods paper consolidates this evidence into a formal recommendation: asymmetry tests should not be used with fewer than about ten studies, and even above that threshold, a non-significant result should be interpreted as inconclusive rather than as confirmation that no bias exists6.
Appropriate interpretation of a funnel plot
Given these documented limitations, the appropriate use of a funnel plot is narrower than detection: it is a prompt for investigation rather than a verdict. Peters and colleagues’ 2006 JAMA comparison of detection methods found that a contour-enhanced funnel plot, which displays the statistical-significance thresholds directly on the chart, helps to distinguish a gap plausibly caused by missing non-significant studies from a gap with another cause; this narrows the plausible explanations without resolving the question definitively7. Duval and Tweedie’s trim-and-fill method extends this approach by estimating how many studies might be missing and what the pooled effect would be if they were included, but the method’s developers frame it as a sensitivity analysis, indicating how far the conclusion could plausibly shift, rather than as a corrected, definitive estimate8.
The Cochrane Handbook’s guidance on reporting biases reflects the same caution, instructing review authors to consider funnel-plot asymmetry alongside other explanations — genuine heterogeneity, methodological differences correlated with study size, and chance — rather than reporting it as a standalone finding of publication bias9. The PRISMA 2020 reporting checklist reinforces this point from the other direction, requiring authors to describe in their methods and results, separately from any figure, exactly what they did to assess the risk of bias from missing results and what that assessment did and did not establish10. Taken together, this body of work supports a single practical principle: a funnel plot or its formal test can indicate whether the evidence is consistent or inconsistent with the absence of bias, but neither detects publication bias with anything approaching certainty on its own.
The argument for prevention over detection
Because retrospective detection has these documented limitations, Lau and colleagues’ 2006 editorial argues explicitly that preventing publication bias from entering the evidence base is a more productive focus than attempting to diagnose it statistically once a body of literature already exists2. Their argument rests on an asymmetry between the two tasks: a funnel-plot test applied after publication can operate only on the fraction of the true evidence base that has been published, whereas a prospective trial registry, by recording a study’s existence and design before its results are known, removes the mechanism that permits selective non-publication. The authors further suggest that an asymmetry finding misread as confirmed bias, when it is only one of several possible explanations, can mislead a review’s conclusions more than it informs them; this is the basis for their specific warning that mechanical, uncritical use of these tests may do more harm than good2.
This prevention-focused view does not render detection tools worthless; rather, it redefines their role. A funnel plot and its associated tests remain useful for identifying a body of evidence that warrants closer scrutiny — cross-checking trial registries, searching for unpublished conference abstracts and contacting trial authors directly — even though they cannot resolve the question of bias on their own. Sterne and colleagues’ 2011 recommendations describe this kind of follow-up investigation as the appropriate next step after an asymmetry finding, rather than treating the finding itself as a conclusion to be reported without further examination6.
In practice, this implies a change in the purpose of the funnel-plot section of a review. Rather than stating whether publication bias was “detected”, a defensible methods and results section describes what was examined: the shape of the plot, whether a formal test was appropriate given the number of studies, which alternative explanations were considered, and what, if anything, that examination ruled in or out. This takes longer to write than a single confident sentence, but it is the form of the claim that the statistical evidence on detection accuracy supports.
Key points
Key points
Direct testing found that researchers reading funnel plots visually identified publication bias at approximately chance level (53% accuracy), and the formal statistical tests developed to replace visual judgement have well-documented limits of statistical power; consequently, no funnel-plot-based method detects publication bias with certainty.
Practical application
Practical application
- Any funnel-plot finding should be treated as a prompt for investigation rather than a conclusion. An asymmetric or symmetric result should be followed by examination of the specific studies involved rather than interpretation of the shape of the plot alone.
- A non-significant asymmetry test should be reported as inconclusive, not as evidence of the absence of bias. The number of included studies should be stated alongside the test result so that readers can judge whether the test had adequate power.
- A contour-enhanced plot and, where relevant, trim-and-fill should be used as sensitivity analyses. These methods narrow the plausible explanations for an observed pattern without settling the question definitively.
Frequently asked questions
Terrin, Schmid, and Lau's 2005 study found 53% accuracy, which the authors describe as practically identical to chance for a task with two possible answers (bias present or absent).
They are generally considered more objective, but Sterne, Gavaghan, and Egger’s 2000 analysis and Begg and Mazumdar’s 1994 evaluation both found that these tests have low statistical power in many real meta-analyses; a non-significant result therefore does not reliably exclude bias.
No single test has been shown to do so. Ioannidis and Trikalinos’s 2007 excess-significance test was developed specifically because asymmetry-based tests miss certain patterns of selective reporting that a different statistical approach can sometimes identify.
A larger number of studies substantially improves the reliability of the assessment, since the principal documented failure mode is low statistical power with few studies; however, a large study count does not remove the need to investigate alternative explanations for any observed asymmetry.
References
- Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology. 2005;58(9):894-901. read 10 October 2026. View on Journal of Clinical Epidemiology
- Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597-600. read 10 October 2026. View on The BMJ
- Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology. 2000;53(11):1119-1129. read 10 October 2026. View on Journal of Clinical Epidemiology
- Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101. read 10 October 2026. View on PubMed
- Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clinical Trials. 2007;4(3):245-253. read 10 October 2026. View on Clinical Trials
- Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;342:d4002. read 10 October 2026. View on The BMJ
- Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-680. read 10 October 2026. View on JAMA
- Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455-463. read 10 October 2026. View on Biometrics
- Sterne JAC, Egger M, Moher D (editors). Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration; 2011. read 10 October 2026. View on Cochrane Training
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on The BMJ
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations. Pubrica. https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/
Pubrica Academy. “Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations.” Pubrica, 10 Oct. 2026, pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/.
Pubrica Academy (2026) Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations. Pubrica. Available at: https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/ (Accessed: 15 October 2026).
Pubrica Academy. “Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations.” Pubrica. October 10, 2026. https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/
Pubrica Academy, ‘Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/> accessed 15 October 2026.
Pubrica Academy. Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/
Pubrica Academy, “Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/
@misc{/la_detection_publication_bias_funnel_plot_2026,
author = {{Pubrica Academy}},
title = {Detection of Publication Bias Using Funnel Plots: Accuracy and Limitations},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/detection-of-publication-bias-in-funnel-plot/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Publication Bias Investigation Steps
This sequence should be followed instead of treating a single funnel-plot reading as a definitive answer.
[ ] Built the funnel plot with standard error on the vertical axis [ ] Ran a formal asymmetry test only if 10+ studies are included [ ] Checked the test's reported power/confidence given the actual study count [ ] Investigated the specific smaller or more extreme studies directly, not just the plot's shape [ ] Reported the finding as 'consistent with' or 'inconsistent with' bias, not as a detection claim
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s systematic review team supplements any funnel-plot assessment with a written, study-by-study investigation of alternative explanations, so that a publication-bias judgement does not rest on the appearance of the plot alone.