Contents
Academy · Research Data, Analytics & AI · Limitations of the Method
Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity
Funnel plots have four well-documented shortcomings: human readers detect bias from them at approximately chance level, formal tests require about ten or more studies for adequate power, asymmetry has multiple causes other than bias, and no test has ever been validated against a true, known measure of publication bias.
The funnel plot is one of the most widely used tools for investigating publication bias in a meta-analysis, which makes its documented shortcomings particularly consequential. Lau and colleagues’ 2006 BMJ editorial, titled “The case of the misleading funnel plot,” sets out four distinct limitations that together explain why the method should be used with considerably more caution than its frequent appearance in published systematic reviews might suggest: unreliable visual interpretation, low statistical power in typical meta-analyses, multiple non-bias explanations for the same visual pattern, and the absence of any validated gold-standard test1.
Learning objectives
- The four distinct shortcomings of funnel-plot methodology identified in the 2006 analysis by Lau and colleagues
- Why no funnel-plot-based test has ever been validated against a true, known measure of bias
- The implications of these shortcomings for reporting a funnel-plot finding
Common misconceptions
- “A funnel plot, if read or tested carefully enough, can give a definitive answer on publication bias.” The analysis by Lau and colleagues states that no test for funnel-plot asymmetry has ever been validated against a true, independently confirmed measure of publication bias, since such validation would require prospective trial registries that did not then exist at the necessary scale. Every available test therefore remains unverified against the phenomenon it is intended to detect.
Shortcoming one: unreliable visual interpretation
The first documented shortcoming is that human readers cannot reliably identify publication bias from the shape of a funnel plot. Terrin, Schmid, and Lau’s 2005 empirical study, cited directly in Lau and colleagues’ 2006 editorial, tested this by showing researchers a set of simulated funnel plots, some generated with publication bias deliberately introduced and some without, and asking them to judge from the plot alone whether bias was present2. The accuracy of these judgements was 53%, described in the editorial as practically identical to chance for a task with only two possible answers1. This is consequential because visual inspection remains, in practice, the most common way in which funnel plots are used and reported in published reviews, so this shortcoming affects the majority of real-world applications of the method rather than a narrow edge case.
Part of the reason for this unreliability is that the appearance of the plot depends on several technical choices — outcome coding, the effect metric selected, and the precision measure used on the vertical axis — that are not always made consistently, and a reader judging a plot visually generally cannot determine from the finished chart alone whether any of these choices were handled correctly1. A plot that appears asymmetric because of an inconsistent effect metric and a plot that appears asymmetric because of genuinely missing studies are visually indistinguishable to the reader, even though they call for entirely different responses.
Shortcoming two: low statistical power in typical meta-analyses
The second shortcoming is that the formal statistical tests developed to replace visual judgement — among them Egger’s regression test and Begg and Mazumdar’s rank correlation test — generally require a substantial number of included studies to have adequate power, whereas Lau and colleagues note that the average Cochrane systematic review at the time included fewer than ten studies, well below that threshold1. Sterne, Gavaghan, and Egger’s own empirical analysis of publication and related bias had already demonstrated this power problem directly, finding that formal tests for small-study effects routinely lacked the power to detect real asymmetry in meta-analyses of the size typically encountered in practice6.
Begg and Mazumdar’s own 1994 evaluation of their rank correlation test reinforces the same conclusion from the developers’ perspective: statistical power depends heavily on both the number of studies and the variation in their sizes, so a non-significant test result from a meta-analysis with few or similarly sized studies reflects the limited sensitivity of the test as much as any genuine absence of bias7. The practical consequence is that a “negative” funnel-plot test result cannot be reported as confirmation that no bias exists without also reporting the study count and, ideally, an indication of the power of the test given that count.
Shortcoming three: multiple non-bias explanations for the same pattern
The third shortcoming is that asymmetry, the pattern that the funnel plot and its formal tests are designed to detect, is not specific to publication bias. Sterne and colleagues’ 2011 BMJ methods paper lists genuine heterogeneity in intervention effect by study size, methodological weaknesses correlated with smaller studies, and statistical artefacts associated with the chosen effect measure as separate, documented causes of the same visual and statistical pattern3. Tang and Liu’s earlier 2000 analysis had already made a related point specifically about heterogeneity, showing that the mathematical relationship between the heterogeneity of an outcome and the shape of its funnel plot was often overlooked by review authors who proceeded directly to a publication-bias interpretation8.
This shortcoming means that even a correctly executed funnel plot — with an appropriate axis, an adequate study count and a properly applied test — cannot on its own distinguish which of several possible explanations accounts for an observed asymmetry. Lau and colleagues’ audit of BMJ systematic reviews found that 14 of the 20 reviews using a funnel plot drew a categorical conclusion directly from an asymmetry finding, without the further investigation required to distinguish between these explanations, which the editorial presents as a direct, documented consequence of this shortcoming remaining unaddressed in practice1.
Shortcoming four: no validated gold-standard test exists
The fourth and most fundamental shortcoming is that no funnel-plot-based test has ever been validated against a true, independently known measure of publication bias. Lau and colleagues state this directly: establishing such a gold standard would require comparing the funnel-plot result of a meta-analysis against a complete, prospectively registered account of every study actually conducted on a topic, published or not, which was not available at the necessary scale when their analysis was written1. Without that comparison, the accuracy figures for every funnel-plot test, including the power estimates discussed above, derive from simulations built on assumptions about the characteristics of biased and unbiased evidence bases, rather than from validation against confirmed real-world cases.
This shortcoming also explains why Lau and colleagues’ editorial concludes with a broader methodological recommendation rather than a call for a better test: that preventing publication bias through mechanisms such as prospective trial registration is likely to be more beneficial than continuing to refine diagnostic tools applied after the fact, specifically because a prevention-focused approach avoids the validation problem entirely rather than attempting to solve it1. Duval and Tweedie’s trim-and-fill method, Peters and colleagues’ contour-enhanced plot, and Ioannidis and Trikalinos’s excess-significance test all represent genuine refinements of the original method, but none of them resolves this fourth, foundational shortcoming, since each remains a model-based estimate rather than a result checked against a confirmed ground truth459. The Cochrane Handbook’s guidance on reporting biases reflects the same limitation in its instructions to review authors, framing every funnel-plot-based finding as a contribution to a broader judgement rather than as a stand-alone, validated measurement10.
Accommodating these shortcomings in practice
None of these four shortcomings is an argument for abandoning the funnel plot entirely; Lau and colleagues’ editorial continues to treat it as a useful component of systematic review methodology, although not as a method capable of delivering the definitive yes-or-no answer it is often expected to provide1. Simmonds’ 2015 simulation study, which quantified the error rates involved in funnel-plot interpretation in greater detail, reached a similar practical conclusion: visual inspection is unlikely to yield a valid assessment of bias on its own, and even the proposed numerical alternatives to visual judgement have low power with ten or fewer studies, which indicates that a formal test, used and reported carefully, remains more defensible than visual inspection alone wherever sufficient studies are available11.
In practice, accommodating these shortcomings means that the methods section of a review should state plainly which of the four limitations apply to the case at hand, how many studies were available, whether a formal test was performed or only visual inspection, which non-bias explanations were examined, and, given all of this, what the authors concluded and with what degree of confidence. This is a longer and more qualified way of reporting a funnel-plot analysis than a single confident sentence about bias being “detected” or “ruled out,” but it is the version that the documented limits of the method support, and it provides readers with the information needed to weigh the finding appropriately rather than accepting a confident claim at face value.
Key points
Key points
Funnel plots have four well-documented shortcomings: unreliable visual interpretation (about 53% accuracy), low statistical power with fewer than approximately ten studies, multiple non-bias explanations for the same asymmetric pattern, and the absence of a validated gold-standard test. Any funnel-plot finding should therefore be reported as a contribution to a broader judgement about the evidence rather than as a stand-alone, validated measurement of publication bias.
Practical application
Practical application
- A funnel-plot finding, whether visual or tested, should never be reported as a stand-alone conclusion. It should be accompanied by the number of studies, the likely power of the test given that number, and an account of the alternative explanations considered.
- A non-significant asymmetry test should be treated as inconclusive rather than as evidence of no bias. This is particularly important with fewer than approximately ten included studies, where the documented power problem is most severe.
- Preventive measures, such as checking trial registries, should be considered alongside any diagnostic test. Lau and colleagues recommend this approach as more effective in addressing bias than continued refinement of detection methods applied retrospectively.
Frequently asked questions
No single method has resolved all four. Later refinements such as contour-enhanced plots, trim-and-fill and the excess-significance test address specific aspects of the original method but remain simulation-based and have not been validated against a confirmed, independent measure of true publication bias.
A larger number of studies substantially improves the statistical-power shortcoming specifically, but it does not resolve the unreliability of visual interpretation, the multiple possible causes of asymmetry, or the absence of a validated gold standard.
Lau and colleagues' editorial does not argue for abandoning the method, but for using it with explicit awareness of its limits, as one input into a broader investigation of a body of evidence rather than as a stand-alone diagnostic conclusion.
References
- Lau J, Ioannidis JPA, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597-600. read 10 October 2026. View on The BMJ
- Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. Journal of Clinical Epidemiology. 2005;58(9):894-901. read 10 October 2026. View on Journal of Clinical Epidemiology
- Sterne JAC, Sutton AJ, Ioannidis JPA, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;342:d4002. read 10 October 2026. View on The BMJ
- Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455-463. read 10 October 2026. View on Biometrics
- Ioannidis JPA, Trikalinos TA. An exploratory test for an excess of significant findings. Clinical Trials. 2007;4(3):245-253. read 10 October 2026. View on Clinical Trials
- Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology. 2000;53(11):1119-1129. read 10 October 2026. View on Journal of Clinical Epidemiology
- Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101. read 10 October 2026. View on PubMed
- Tang JL, Liu JLY. Misleading funnel plot for detection of bias in meta-analysis. Journal of Clinical Epidemiology. 2000;53(5):477-484. read 10 October 2026. View on Journal of Clinical Epidemiology
- Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676-680. read 10 October 2026. View on JAMA
- Sterne JAC, Egger M, Moher D (editors). Chapter 10: Addressing reporting biases. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration; 2011. read 10 October 2026. View on Cochrane Training
- Simmonds M. Quantifying the risk of error when interpreting funnel plots. Systematic Reviews. 2015;4:24. read 10 October 2026. View on Systematic Reviews
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity. Pubrica. https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/
Pubrica Academy. “Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity.” Pubrica, 10 Oct. 2026, pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/.
Pubrica Academy (2026) Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity. Pubrica. Available at: https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/ (Accessed: 15 October 2026).
Pubrica Academy. “Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity.” Pubrica. October 10, 2026. https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/
Pubrica Academy, ‘Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/> accessed 15 October 2026.
Pubrica Academy. Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/
Pubrica Academy, “Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/
@misc{/la_limitations_funnel_plots_meta_analysis_2026,
author = {{Pubrica Academy}},
title = {Shortcomings of Funnel Plots in Meta-Analysis: Interpretation, Power and Validity},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/limitations-of-funnel-plots-in-meta-analysis/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Funnel Plot Shortcomings Disclosure Note
The following wording may serve as a starting point for a methods or limitations section discussing a funnel-plot analysis.
[ ] Stated the number of studies included and whether this meets the ~10-study threshold for formal tests [ ] Noted that visual inspection alone has documented accuracy close to chance [ ] Listed which non-bias explanations (heterogeneity, methodological quality, artefacts) were considered [ ] Stated that no funnel-plot test has been validated against a true, known measure of bias [ ] Framed any finding as contributing to, not concluding, the overall risk-of-bias judgement
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s systematic review team reports funnel-plot findings with their documented shortcomings fully disclosed, so that a review’s bias assessment reads as a careful judgement rather than an overstated test result.