Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data Analytics & AI · Statistical tests in meta-analysis

Statistical Tests in Meta-Analysis: The Question Addressed by Each Test

A meta-analysis performs several tests that address different questions, and most reporting problems arise from using one test to answer the question of another. The test of the summary effect, the test of homogeneity, the tests of funnel plot asymmetry and the tests within a meta-regression have different null hypotheses, different assumptions and different limitations, and several are uninformative at the number of studies that most reviews include.

Pubrica AcademyWritten by the medical writing and biostatistics teams Reviewed byPubrica biostatistics team Published9 October 2026 Last checked9 October 2026 Reading time13 minutes Sources43 peer-reviewed

The output of a meta-analysis package typically contains a pooled estimate, a p-value, a Q statistic, an I-squared value and often an Egger test. These results are not a coherent set of conclusions: they test different null hypotheses under different assumptions, and at least two of them are usually unreliable in a review of typical size. Understanding which question each test addresses, and the number of studies below which each ceases to be informative, distinguishes a defensible synthesis from unexamined software output. This article examines each test in turn.

Learning objectives

  • The four distinct questions tested in a meta-analysis, and the null hypothesis associated with each
  • Why the test of the summary effect depends on the confidence interval method — of which Veroniki and colleagues identified 15 in 7 categories
  • What Q, H, I-squared and R each do, and why a homogeneity test is not a measure of heterogeneity
  • When funnel plot asymmetry tests are appropriate, and the 10-study and tau-squared thresholds that govern the choice
  • How meta-regression increases the rate of false-positive findings, and the recommendations of Higgins and Thompson
  • Which influence and outlier diagnostics should be performed, and what a Baujat plot shows that a forest plot does not

Common misconceptions

  • “A non-significant Q means the studies are homogeneous.” Q has low statistical power when few studies are included. A non-significant result is compatible with substantial heterogeneity, which is why Higgins and Thompson proposed descriptive statistics to accompany it.
  • “I-squared measures how much the studies differ.” It is the proportion of total variability attributable to between-study variation, not an absolute amount, and it does not indicate whether the differences are clinically important.
  • “A significant Egger test means publication bias.” It indicates asymmetry, which has several possible causes, including genuine heterogeneity and the relationship between effect size and sample size in the condition studied.
  • “Random effects handles heterogeneity.” The random-effects model accommodates heterogeneity in the interval; it does not explain it, and it changes the interpretation of the summary estimate. Riley, Higgins and Deeks, in the BMJ, describe what the random-effects summary does and does not represent.
  • “Meta-regression can test a subgroup hypothesis.” With few studies and several covariates, it is likely to produce spurious findings. Higgins and Thompson published a paper in Statistics in Medicine specifically about controlling that risk.

Four questions and their null hypotheses

Before any test, the question being asked should be clear. A meta-analysis typically asks four, and conflating them is the source of most misreporting.

Is there an average effect? Null: the summary effect is zero. This is the test to which most readers attend, and it is the one most sensitive to the choices described in the next section.

Do the studies agree? Null: all studies estimate the same underlying effect. This is Cochran’s homogeneity test, reported as Q on degrees of freedom 41.

Is the set of studies systematically incomplete? Null: no relationship between estimated effect and precision. This is what funnel plot asymmetry tests address, imperfectly 7.

Does the effect depend on a study characteristic? Null: the regression coefficient on that covariate is zero. This is meta-regression, the most susceptible of the four to misuse.

Each is a separate hypothesis with its own assumptions, and none of them answers another. A review that reports a significant pooled effect and a non-significant Q has not shown that the studies agree; it has shown that the test of agreement did not reject, which with few studies is close to uninformative. What a hypothesis test can and cannot deliver in general is covered on the statistical hypothesis page.

Which test answers which question

QuestionNull hypothesisTest or statisticPrincipal sourceMain limitation
Is there an average effect?The summary effect equals the null valueWald or t-type test from the pooled estimate and its standard errorDerSimonian and Laird, Controlled Clinical Trials; Hartung and Knapp, Statistics in MedicineDepends on the between-study variance estimator and the interval method — 15 methods in 7 categories exist
Do the studies agree?All studies estimate the same underlying effectCochran’s Q on k−1 degrees of freedomCochran, BiometricsLow power with few studies; over-sensitive with many or very precise ones
How much do they differ?Not a test — a descriptionH, I-squared, R; tau-squared with a confidence intervalHiggins and Thompson, Statistics in Medicine; Viechtbauer, Statistics in MedicineI-squared is a proportion, not an amount, and says nothing about clinical importance
Is the evidence base incomplete?No relationship between estimated effect and precisionEgger; Begg and Mazumdar; Harbord; Peters; Rücker arcsineEgger et al, BMJ; Sterne et al, BMJGenerally not advisable below 10 studies; test choice depends on tau-squared, with 0.1 as the stated threshold
Does a study characteristic explain the variation?The regression coefficient on the covariate is zeroMeta-regression, with the Knapp-Hartung adjustmentThompson and Higgins, Statistics in Medicine; Knapp and Hartung, Statistics in MedicineStudies are the unit of analysis, so false positives arise readily; aggregation across study means does not license individual-level claims
Does one study drive the result?Not a test — a diagnosticLeave-one-out, Cook’s distance, standardised residuals, Baujat plotViechtbauer and Cheung, Research Synthesis Methods; Baujat et al, Statistics in MedicineDiagnostics flag influence; they do not indicate whether any study should be excluded

Numerical thresholds as reported in Sterne et al (BMJ, 2011) and Veroniki et al (Research Synthesis Methods). Limitations are those stated by the cited authors, not added here.

Testing the summary effect and the importance of the interval method

The summary effect is a weighted average, and the test of whether it differs from the null value depends on how its standard error was computed. Under a fixed-effect model that is straightforward. Under a random-effects model it is not, because the between-study variance has to be estimated and then propagated, and there is no single accepted way to do either.

Veroniki and colleagues, in Research Synthesis Methods, identified 15 confidence interval methods for the random-effects summary effect and grouped them into 7 categories 42. This is not a theoretical curiosity. Different methods give different intervals on the same data, and the difference is largest precisely where reviews are weakest: few studies and substantial heterogeneity. The DerSimonian and Laird approach, introduced in Controlled Clinical Trials, remains the most widely implemented default and is known to produce intervals that are too narrow when the number of studies is small 37,42. The Hartung-Knapp-Sidik-Jonkman approach, developed by Hartung and Knapp in Statistics in Medicine and by Sidik and Jonkman in Applied Statistics, and examined by Röver, Knapp and Friede in BMC Medical Research Methodology for the few-studies case, is the usual alternative 5,15,26.

The practical consequence is that “the pooled effect was significant” is an incomplete statement. The model, the between-study variance estimator and the interval method should be named, and where the review is small, a sensitivity analysis across methods should be reported. The fixed versus random effects page covers the model choice itself.

Q, H, I-squared and R: one test and three descriptive statistics

Cochran, in Biometrics, gave the statistic that tests homogeneity. DerSimonian and Laird, in Controlled Clinical Trials, report it for Baum’s review of antibiotic prophylaxis in colon surgery, where the within-study homogeneity statistic Q was 40.4 on 25 degrees of freedom 37. Under the null, Q is expected to be close to its degrees of freedom, and the test is interpreted entirely through that comparison.

Q has two well-known problems. With few studies it has low power, so failure to reject carries little information. With many studies or very precise ones, it will reject on differences too small to be important. In either case it answers a dichotomous question when a quantity is required.

Higgins and Thompson, in Statistics in Medicine, addressed this directly by proposing three statistics — H, I-squared and R — to describe heterogeneity independently of the effect metric and the number of studies, illustrated on five datasets 25. Higgins, Thompson, Deeks and Altman then set out the practical use of I-squared in the BMJ 19. The important distinction is that I-squared is a proportion, not an amount: it describes how much of the observed variability is between-study rather than within-study, which means a review of very precise studies can show a high I-squared on clinically trivial differences.

For the amount itself, the between-study variance and an interval around it are required. Viechtbauer, in Statistics in Medicine, developed a confidence interval method for the amount of heterogeneity that maintains nominal coverage probabilities even in small samples when model assumptions hold, and had the most accurate coverage in simulations where normality and known sampling variances hold only asymptotically 12. Reporting an interval around tau-squared remains uncommon and is recommended. Estimator choice also matters: Viechtbauer compared the bias and efficiency of meta-analytic variance estimators in the Journal of Educational and Behavioral Statistics, and Sidik and Jonkman, Biggerstaff and Tweedie, and Higgins, Thompson and Spiegelhalter all offer alternatives to the moment estimator to which most packages default 9,14,34.

Tests of funnel plot asymmetry: selection and conditions for use

Egger and colleagues, in the BMJ, introduced the regression test that most reviews still report 33. Begg and Mazumdar, in Biometrics, had earlier set out a rank correlation test and examined its operating characteristics 36. Harbord, Egger and Sterne, and Rücker, Schwarzer and Carpenter, both in Statistics in Medicine, developed tests intended to behave better for binary outcomes, where the original Egger test can reject because of a mathematical association between the log odds ratio and its standard error rather than because of bias 10,13.

Two numbers from Sterne and colleagues’ BMJ recommendations should govern the decision before any test is run. First, tests for funnel plot asymmetry should generally not be used when a meta-analysis contains fewer than 10 studies, and the minimum may be substantially higher when heterogeneity is substantial 7. Second, the Harbord, Peters and Rücker tests are advised when the estimated between-study variance of log odds ratios is below 0.1; above that only the random-effects arcsine test performs reasonably well 7. These two conditions exclude a large proportion of published meta-analyses from testing altogether, and the correct response in that case is to state that the number of studies did not permit a meaningful test, not to run one and report it as reassurance.

Where tests are available, it should be noted that they disagree. Sterne and colleagues report that for the intravenous magnesium meta-analysis the Harbord test gave P less than 0.001 and the Peters test P equal to 0.002 for the same null on the same trials. Both are correct; they model the relationship between effect and precision differently.

The adjustment methods deserve more scepticism than they usually receive. Duval and Tweedie’s trim and fill, in Biometrics, imputes studies and should be treated as a sensitivity analysis rather than a correction 7,31. The fail-safe N family — Rosenthal in Psychological Bulletin, Orwin in the Journal of Educational Statistics, Rosenberg in Evolution — answers a question that is rarely of practical relevance, namely how many null studies would be needed to overturn the result, and the number is easily misread as a measure of robustness 16,38,39. Sterne, Gavaghan and Egger, in the Journal of Clinical Epidemiology, examined the power of these tests and the prevalence of bias in the literature; low power is the recurring finding 30.

10minimum studies before an asymmetry test is generally advisable
0.1tau-squared above which only the arcsine test performs reasonably
15confidence interval methods for the random-effects summary effect
7categories those methods fall into

Sources: Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002 (first two cells). Veroniki AA, Jackson D, Bender R, Kuss O, Langan D, Higgins JPT, Knapp G, Salanti G. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. Research Synthesis Methods (last two cells).

Meta-regression and the risk of false-positive findings

Meta-regression tests whether a study-level characteristic explains variation in effect, and it is the most commonly over-interpreted analysis in a review. Thompson and Higgins, in Statistics in Medicine, set out how it should be undertaken and interpreted; Higgins and Thompson wrote a separate paper in the same journal on controlling the risk of spurious findings from it 17,24. The problem is structural: studies, not patients, are the units, so the effective sample size is the number of studies, and testing several covariates across a dozen studies will produce significant coefficients by chance.

Three safeguards are recommended. First, the covariates should be pre-specified, with a stated rationale and a limit on their number; an analysis that tests five covariates on twelve studies is a data-dredging exercise whatever its p-values indicate. Second, a test that accounts for the uncertainty in the between-study variance should be used: Knapp and Hartung, in Statistics in Medicine, developed improved tests for a random-effects meta-regression with a single covariate, and these are implemented in the main packages 18. Third, the aggregation problem should be stated explicitly: a study-level association between mean age and effect size does not establish that older individuals respond differently, because the relationship across study means need not hold within studies.

Where subgroup questions are important enough to require a reliable answer, the appropriate conclusion is often that study-level data cannot provide one. Pubrica’s biostatistics page covers the alternatives.

Influence diagnostics, outliers and recommended plots

A pooled estimate can rest on one study, and a forest plot will not always make this apparent. Viechtbauer and Cheung, in Research Synthesis Methods, set out outlier and influence diagnostics for meta-analysis (leave-one-out estimates, standardised residuals, Cook’s distance and hat values), and these are the quickest check on whether the conclusion is a property of the evidence or of one trial.

Baujat and colleagues, in Statistics in Medicine, developed a graphical method for exploring heterogeneity, illustrated on a meta-analysis of 65 trials, which plots each study’s contribution to the overall heterogeneity statistic against its influence on the pooled estimate 23. Studies in the upper right of that plot are driving both, which indicates where further scrutiny is needed. Kossmeier, Tran and Voracek, in the Zeitschrift für Psychologie, extended the funnel plot with power contours, which helps to separate “no asymmetry” from “no ability to detect asymmetry”, a distinction that the test alone cannot provide 1.

None of these is a test, and that is their purpose. They are diagnostics that indicate whether the tests performed were answering the intended question. They should be performed before the conclusion is written, not after a reviewer requests them.

Reporting statistical tests to permit verification

Because every test here depends on choices — estimator, interval method, which asymmetry test — a statistics paragraph that names none of them is not reproducible. The weak version below would be consistent with dozens of different analyses giving different answers.

The same analysis, reported two ways

Not reproducible

Data were pooled using a random-effects model. Heterogeneity was assessed using the I-squared statistic and was found to be acceptable. Publication bias was assessed using Egger’s test and was not significant. Subgroup analyses were performed to explore sources of heterogeneity.

Which estimator, which interval? “Random-effects model” leaves 15 interval methods in 7 categories open, and they disagree most where reviews are smallest.

“Acceptable” heterogeneity against what? I-squared is a proportion; no threshold on it establishes clinical comparability.

Egger’s test on how many studies? Below 10 it should generally not have been run, so a non-significant result is not reassurance.

How many subgroup analyses, decided when? “Explore” with no number and no pre-specification is a source of false positives.

Was the result driven by one trial? No diagnostic is reported, so this cannot be determined.

Reproducible, and bounded

Effects were pooled on the log odds ratio scale using a random-effects model with the restricted maximum likelihood estimator of tau-squared and Hartung-Knapp-Sidik-Jonkman confidence intervals, chosen in advance because fewer than ten studies were anticipated; results using the DerSimonian and Laird estimator with Wald intervals are given in the appendix as a sensitivity analysis. Heterogeneity is reported as tau-squared with a confidence interval alongside I-squared, and the pooled result is accompanied by a prediction interval. Because the review includes fewer than 10 studies, no test for funnel plot asymmetry was conducted and none is reported; the funnel plot is shown with power contours for inspection only. One pre-specified covariate was examined by meta-regression using the Knapp-Hartung adjustment; no further covariates were tested. Influence was assessed by leave-one-out estimates and a Baujat plot, both reported in full.

Scale, model, variance estimator and interval method are all named, and the choice is justified by the expected number of studies.

A sensitivity analysis across methods is reported rather than asserted, which is what the few-studies case requires.

Heterogeneity is reported as an amount with an interval, not only as a proportion, and a prediction interval is given.

The asymmetry test is explicitly not run, and the reason is given; this is a stronger statement than a non-significant test.

Meta-regression is limited to one pre-specified covariate with an adjustment that accounts for variance uncertainty.

Influence diagnostics are reported, so a reader can see whether one study carries the conclusion.

The second version is longer and reports fewer tests. This is the trade-off: it declines the asymmetry test that it was not entitled to run and limits meta-regression to one covariate, while adding the estimator, the interval method, a sensitivity analysis and a prediction interval. Nothing in it is a new statistic; it is the same analysis with the choices made visible, which is the only form in which these tests can be verified.

Limitations that statistical testing cannot address

Three limitations should be stated in the review’s own limitations section rather than left for a reviewer to identify.

None of these tests addresses whether the studies should have been combined. That is a judgement about whether the populations, interventions and outcomes are sufficiently similar for a common question to exist. A low I-squared across clinically incomparable trials is a statistical fact about an answer of no clinical relevance.

Asymmetry tests do not identify publication bias. They detect a relationship between effect and precision, which can arise from reporting bias, from genuine heterogeneity, from differences in trial conduct between small and large studies, or from the mathematical properties of the effect measure. Sterne and colleagues are explicit that asymmetry has multiple possible explanations, and the appropriate conclusion names them 7.

Interpretation of the random-effects summary is not settled by any test. Riley, Higgins and Deeks, in the BMJ, argue that a random-effects meta-analysis should be reported with a prediction interval as well as a confidence interval, because the two answer different questions — where the average effect lies, and what effect a new setting might see 6,9. Reviews that report only the confidence interval are reporting the less useful of the two for most clinical readers, and the prediction interval remains more often recommended than reported.

Key points

Key points

A meta-analysis tests four different null hypotheses, and these tests are not interchangeable. The test of the summary effect depends on choices of variance estimator and interval method, of which Veroniki and colleagues identified 15 versions across 7 categories, so naming the model alone is insufficient. Cochran’s Q tests homogeneity but does not measure heterogeneity; H, I-squared, R and an interval around tau-squared serve that purpose. Funnel plot asymmetry tests should generally not be performed with fewer than 10 studies, nor should the standard tests be used when tau-squared exceeds 0.1, and reporting that a test was not performed is preferable to reporting a non-significant result. Meta-regression on study-level data is likely to produce spurious findings unless the covariates were pre-specified and few in number.

Application to your study

Application to your study

  1. The question answered by each test in the output should be documented. Any remaining statistic is a value reported without a corresponding question.
  2. The between-study variance estimator and the interval method should be named. The term “random-effects model” alone does not identify an analysis.
  3. With fewer than approximately ten studies, a sensitivity analysis across interval methods should be conducted. This is the situation in which the 15 methods diverge most.
  4. Tau-squared should be reported with a confidence interval, in addition to I-squared. I-squared is a proportion and can be high when differences are trivial.
  5. The number of studies should be established before an asymmetry test is selected. Below 10 studies, it should be reported that a test was not appropriate; above this number, the test should be selected according to tau-squared, with 0.1 as the stated threshold.
  6. Trim and fill and fail-safe N should be treated, at most, as sensitivity analyses. Neither method corrects a biased evidence base, and fail-safe N addresses a question that is rarely relevant to the review.
  7. Meta-regression should be limited to pre-specified covariates and should use the Knapp-Hartung adjustment. The text should state that study-level associations do not support individual-level inferences.
  8. Leave-one-out analysis and a Baujat plot should be performed before the conclusion is written. If one study drives both the heterogeneity and the estimate, the conclusion should state this.
  9. A prediction interval should accompany any random-effects result. It addresses the question of greatest relevance to most clinical readers.

Frequently asked questions

At minimum: the summary effect with a confidence interval and the test of whether it differs from the null value; a measure of heterogeneity, which should include tau-squared with an interval as well as I-squared; and, where the number of studies permits, an examination of funnel plot asymmetry. Influence diagnostics, such as leave-one-out estimates, should be added. Each of these answers a different question, and the first does not depend on the second being non-significant. More important than the list is that the choices are named: the between-study variance estimator, the confidence interval method, and the asymmetry test, if any, since Veroniki and colleagues identified 15 interval methods in 7 categories and they do not all agree.

Q is a test; I-squared is a description. Q tests the null that all studies estimate the same underlying effect, and is read against its degrees of freedom: DerSimonian and Laird report Q of 40.4 on 25 degrees of freedom in Baum’s colon surgery review, which is more dispersion than the null predicts. However, Q has low power with few studies and over-rejects with many precise ones. Higgins and Thompson, in Statistics in Medicine, proposed H, I-squared and R to describe heterogeneity independently of the effect metric and the number of studies. I-squared expresses the proportion of total variability attributable to between-study variation, which is not the same as the amount of heterogeneity.

Sterne and colleagues, in the BMJ, advise that tests for funnel plot asymmetry should generally not be used when a meta-analysis contains fewer than 10 studies, and that the minimum may be substantially higher when heterogeneity is substantial. Below that number, the appropriate report is that the number of studies did not permit a meaningful test, which is a stronger and more transparent statement than a non-significant test offered as reassurance. Where a test is performed, it should be selected according to heterogeneity: the Harbord, Peters and Rücker tests are advised when the estimated between-study variance of log odds ratios is below 0.1, and above that only the random-effects arcsine test performs reasonably well.

Because they model the relationship between effect size and precision differently, and a p-value is computed within the model. Sterne and colleagues report that for the intravenous magnesium meta-analysis the Harbord test gave P below 0.001 and the Peters test P equal to 0.002 for the same null on the same trials. Both results are correct. For this reason, naming the test is not optional, and selecting a test after observing which one gives the preferred result turns a legitimate modelling decision into a biased one. It is also why asymmetry, if detected, should be reported as asymmetry with its possible explanations rather than as publication bias.

It is valid but easily misused. The unit of analysis is the study, so the effective sample size is the number of studies, and testing several covariates across a dozen studies will generate significant coefficients by chance; Higgins and Thompson devoted a paper in Statistics in Medicine to controlling precisely that risk. It should be used with few, pre-specified covariates, with the Knapp-Hartung adjustment applied so that uncertainty in the between-study variance is accounted for, and with an explicit statement that an association across study means does not establish the same relationship within studies. Where the subgroup question is genuinely important, study-level data often cannot answer it.

It separates two things a forest plot mixes together: how much each study contributes to the overall heterogeneity statistic, and how much it influences the pooled estimate. Baujat and colleagues, in Statistics in Medicine, developed it and illustrated it on a meta-analysis of 65 trials. Studies in the upper right of the plot are driving both the inconsistency and the result, which is precisely the situation in which a conclusion is a property of one trial rather than of the evidence. Used alongside the leave-one-out estimates and other diagnostics described by Viechtbauer and Cheung, it addresses the question of robustness before a reviewer raises it.

Riley, Higgins and Deeks argue in the BMJ that a prediction interval should be reported, because the two answer different questions. The confidence interval describes uncertainty about the average effect across studies; the prediction interval describes the range of effects a new setting might plausibly see, which is usually closer to what a clinical reader wishes to know. In a review with substantial heterogeneity the prediction interval can cross the null value while the confidence interval does not, and that discrepancy is informative rather than problematic. It remains more often recommended than reported, so its inclusion still distinguishes a carefully conducted synthesis.

No, and this is the most consequential confusion in the list. I-squared is computed from the observed effect estimates and their precisions; it contains no information about whether the populations, interventions, comparators and outcomes were similar enough for a common question to exist. A set of clinically incomparable trials can produce a low I-squared, and the resulting summary is a precise answer to a question of no clinical relevance. The decision to pool is a judgement about clinical and methodological similarity made before the statistics are computed, and it should be argued in the text rather than inferred from a heterogeneity statistic afterwards.

References

  1. Kossmeier M, Tran US, Voracek M. Power-enhanced funnel plots for meta-analysis. Zeitschrift fur Psychologie. 2020;228:43-49. read 9 October 2026. arxiv.org
  2. Efthimiou O, Rucker G, Schwarzer G, Higgins J, Egger M, Salanti G. Network meta-analysis of rare events using the Mantel-Haenszel method. Statistics in Medicine. 2019;38(16):2992-3012. doi:10.1002/sim.8158 read 9 October 2026. doi.org/10.1002/sim.8158
  3. Amrhein V, Greenland S, McShane B. Retire statistical significance. Nature. 2019;567(7748):305-307. doi:10.1038/d41586-019-00857-9 read 9 October 2026. doi.org/10.1038/d41586-019-00857-9
  4. Ahlbom A. Significance testing: why does it prevail? European Journal of Epidemiology. 2017;32(1):1-2. doi:10.1007/s10654-016-0220-0 read 9 October 2026. doi.org/10.1007/s10654-016-0220-0
  5. Rover C, Knapp G, Friede T. Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies. BMC Medical Research Methodology. 2015;15:99. doi:10.1186/s12874-015-0091-1 read 9 October 2026. doi.org/10.1186/s12874-015-0091-1
  6. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549. doi:10.1136/bmj.d549 read 9 October 2026. doi.org/10.1136/bmj.d549
  7. Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JPT. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. doi:10.1136/bmj.d4002 read 9 October 2026. doi.org/10.1136/bmj.d4002
  8. Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software. 2010;36(3):1-48. doi:10.18637/jss.v036.i03 read 9 October 2026. doi.org/10.18637/jss.v036.i03
  9. Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society: Series A. 2009;172(1):137-159. read 9 October 2026. discovery-pp.ucl.ac.uk
  10. Rucker G, Schwarzer G, Carpenter J. Statistics in Medicine. 2008;27:746-763. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
  11. Bax L, Yu LM, Ikeda N, Tsuruta N, Moons KGM. A systematic comparison of software dedicated to meta-analysis of causal studies. BMC Medical Research Methodology. 2007;7:40. doi:10.1186/1471-2288-7-40 read 9 October 2026. doi.org/10.1186/1471-2288-7-40
  12. Viechtbauer W. Confidence intervals for the amount of heterogeneity in meta-analysis. Statistics in Medicine. 2007;26(1):37-52. doi:10.1002/sim.2514 read 9 October 2026. doi.org/10.1002/sim.2514
  13. Harbord RM, Egger M, Sterne JA. Statistics in Medicine. 2006;25:3443-3457. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
  14. Viechtbauer W. Bias and efficiency of meta-analytic variance estimators in the random-effects model. Journal of Educational and Behavioral Statistics. 2005;30(3):261-293. doi:10.3102/10769986030003261 read 9 October 2026. doi.org/10.3102/10769986030003261
  15. Sidik K, Jonkman JN. Simple heterogeneity variance estimation for meta-analysis. Applied Statistics. 2005;54(2):367-384. read 9 October 2026. metafor-project.org
  16. Rosenberg MS. The file-drawer problem revisited: a general weighted method for calculating fail-safe numbers in meta-analysis. Evolution. 2005;59:464-468. read 9 October 2026. arxiv.org
  17. Higgins JPT, Thompson SG. Controlling the risk of spurious findings from meta-regression. Statistics in Medicine. 2004;23:1663-1682. read 9 October 2026. ageconsearch.umn.edu
  18. Knapp G, Hartung J. Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine. 2003;22:2693-2710. read 9 October 2026. arxiv.org
  19. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560. doi:10.1136/bmj.327.7414.557 read 9 October 2026. doi.org/10.1136/bmj.327.7414.557
  20. Bradburn MJ, Deeks JJ, Altman DG. Metan: an alternative meta-analysis command. Stata Technical Bulletin. 2003. read 9 October 2026. link.springer.com
  21. Steichen TJ. Nonparametric trim and fill analysis of publication bias in meta-analysis (metatrim). Stata Technical Bulletin. 2003. read 9 October 2026. link.springer.com
  22. Steichen TJ. Tests for publication bias in meta-analysis (metabias). Stata Journal. 2003. read 9 October 2026. link.springer.com
  23. Baujat B, Mahe C, Pignon JP, Hill C. A graphical method for exploring heterogeneity in meta-analyses: application to a meta-analysis of 65 trials. Statistics in Medicine. 2002;21:2641-2652. read 9 October 2026. arxiv.org
  24. Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine. 2002;21(11):1559-1573. doi:10.1002/sim.1187 read 9 October 2026. doi.org/10.1002/sim.1187
  25. Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine. 2002;21(11):1539-1558. doi:10.1002/sim.1186 read 9 October 2026. doi.org/10.1002/sim.1186
  26. Hartung J, Knapp G. On tests of the overall treatment effect in meta-analysis with normally distributed responses. Statistics in Medicine. 2001;20:1771-1782. read 9 October 2026. arxiv.org
  27. Deeks JJ, Altman DG, Bradburn MJ. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Health Care: Meta-analysis in Context. 2001:285-321. read 9 October 2026. ageconsearch.umn.edu
  28. Sterne JAC, Egger M, Davey Smith G. Systematic reviews in health care: investigating and dealing with publication and other biases in meta-analysis. BMJ. 2001;323:101-105. read 9 October 2026. www-users.york.ac.uk
  29. Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F. Methods for meta-analysis in medical research. Wiley (book). 2000. read 9 October 2026. link.springer.com
  30. Sterne JAC, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology. 2000;53:1119-1129. read 9 October 2026. ageconsearch.umn.edu
  31. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56:455-463. read 9 October 2026. arxiv.org
  32. Follmann DA, Proschan MA. Valid inference in random effects meta-analysis. Biometrics. 1999;55:732-737. read 9 October 2026. arxiv.org
  33. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629-634. read 9 October 2026. www-users.york.ac.uk
  34. Biggerstaff BJ, Tweedie RL. Incorporating variability in estimates of heterogeneity in the random effects model in meta-analysis. Statistics in Medicine. 1997;16(7):753-768. read 9 October 2026. metafor-project.org
  35. Eysenck HJ. Meta-analysis and its problems. BMJ. 1994;309:789-792. read 9 October 2026. link.springer.com
  36. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101. doi:10.2307/2533446 read 9 October 2026. doi.org/10.2307/2533446
  37. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7(3):177-188. doi:10.1016/0197-2456(86)90046-2 read 9 October 2026. doi.org/10.1016/0197-2456(86)90046-2
  38. Orwin RG. A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics. 1983;8:157-159. read 9 October 2026. arxiv.org
  39. Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin. 1979;86:638-641. read 9 October 2026. arxiv.org
  40. Fleiss JL. Measuring nominal scale agreement among many raters. Psychological Bulletin. 1971;76:378-382. read 9 October 2026. trialsjournal.biomedcentral.com
  41. Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10(1):101-129. doi:10.2307/3001666 read 9 October 2026. doi.org/10.2307/3001666
  42. Veroniki AA, Jackson D, Bender R, Kuss O, Langan D, Higgins JPT, Knapp G, Salanti G. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. doi:10.1002/jrsm.1319 read 9 October 2026. doi.org/10.1002/jrsm.1319
  43. Stang A, Deckert M, Poole C, Rothman KJ. Statistical inference in abstracts of major medical and epidemiology journals 1975-2014: a systematic review. European Journal of Epidemiology. doi:10.1007/s10654-016-0211-1 read 9 October 2026. doi.org/10.1007/s10654-016-0211-1

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 9). Statistical Tests in Meta-Analysis: The Question Addressed by Each Test. Pubrica. https://pubrica.com/academy/concepts-definitions/statistical-tests-in-meta-analysis/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Statistical methods paragraph skeleton for a meta-analysis

This skeleton should be completed before the analysis is run, and the completed version transferred to the methods section. Every blank corresponds to a choice that changes the result, which is why leaving one blank renders the analysis unreproducible rather than merely under-described.

META-ANALYSIS STATISTICAL METHODS - SPECIFICATION
Review: ______________________    Version / date: ______________
Anticipated number of studies: ____

1. EFFECT MEASURE AND SCALE
   Effect measure: ______________________________________
   Analysis scale (e.g. log odds ratio): ________________
   Why this measure for this outcome: ___________________

2. MODEL
   [ ] Fixed effect   [ ] Random effects
   Justification in terms of the clinical question, not the
   heterogeneity statistic: _____________________________

3. BETWEEN-STUDY VARIANCE AND INTERVALS  (random effects only)
   tau-squared estimator: _______________________________
   Confidence interval method for the summary effect: ___
   Why this method given the expected number of studies: _
     ____________________________________________________
   Sensitivity analysis across methods: [ ] yes  [ ] no
     if yes, which: _____________________________________

4. HETEROGENEITY REPORTING
   [ ] Q with degrees of freedom
   [ ] I-squared
   [ ] tau-squared WITH a confidence interval
   [ ] prediction interval for the summary effect

5. FUNNEL PLOT ASYMMETRY
   Number of studies: ____
   [ ] 10 or more  -> test planned: ______________________
        test selected on the basis of estimated tau-squared
        (threshold 0.1 for log odds ratios): ____________
   [ ] fewer than 10 -> NO test will be run. The methods
        section will state this and the reason.
   Funnel plot shown for inspection only: [ ] yes  [ ] no
   Adjustment methods, if any, labelled as sensitivity
   analyses: ____________________________________________

6. META-REGRESSION / SUBGROUPS
   Pre-specified covariates (list; keep the number small):
     1. _________________________________________________
     2. _________________________________________________
   Total number of covariates to be tested: ____
   Adjustment used (e.g. Knapp-Hartung): _______________
   Sentence to appear in the discussion about aggregation:
     ____________________________________________________

7. INFLUENCE AND OUTLIER DIAGNOSTICS
   [ ] leave-one-out summary estimates
   [ ] standardised residuals / Cook's distance
   [ ] Baujat plot
   Rule for what happens if one study dominates: ________
     (report it - exclusion requires a stated prior rule)

8. SOFTWARE
   Package and version: _________________________________
   Function and non-default arguments: __________________
   Analysis script archived at: _________________________

9. WHAT WE WILL NOT CLAIM
   [ ] asymmetry will not be reported as publication bias
   [ ] a non-significant Q will not be reported as homogeneity
   [ ] a low I-squared will not be offered as justification
        for having pooled
   [ ] study-level associations will not be stated as
        individual-level effects

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s biostatisticians specify the estimator, the interval method and the diagnostics before a review is conducted, when these choices can still be made freely, and indicate clearly when a review is too small to support an asymmetry test or a meta-regression. For completed analyses, the team re-runs the pooling across interval methods, adds the prediction interval and influence diagnostics that are frequently absent from drafts, and revises the statistical methods paragraph so that another analyst could reproduce it.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Pubrica can assess which of these tests a review’s extracted data can support.

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.