Why Meta-Analyses Get Desk-Rejected or Torn Apart in Peer Review: 12 Reviewer Criticisms and How to Fix Them

Editors screen for novelty, journal fit, relevance and contribution, often before any external review; our guide to avoiding desk rejection in Q1 journals covers that stage generally. Methodological reviewers then probe study selection, statistical analysis, risk of bias and interpretation, and reporting gaps can sink a manuscript at either stage. The 12 criticisms below recur often enough to plan for; they are not a ranking, and their weight varies by journal and specialty.

Editors reject reviews that duplicate existing syntheses without showing why another is needed. Map prior reviews by population, intervention, comparator, outcomes, date and analytical approach, then state the evidence gap in the introduction: new trials, an overlooked population or evidence that could change the effect estimate or its certainty. Two extra studies in a six-study review are not enough unless they move the estimate or narrow its uncertainty.

One database, unexplained date limits or missing syntax undermine confidence in the evidence base. Choose sources for the question (MEDLINE, Embase and CENTRAL suit many clinical intervention reviews) and add trial registries, reference lists and citation tracking where relevant. Search strategy reproducibility means reporting every database and platform, full search strings, dates and limits, as the Cochrane Handbook sets out [5]. Justify your sources rather than assuming more databases mean a better search.

Use independent duplicate screening and extraction where feasible, with a defined route for resolving disagreements, and keep full-text exclusion reasons. Data extraction accuracy needs checking against source tables, figures and supplements: entering a standard error as a standard deviation shrinks a study’s variance and inflates its weight. A high kappa shows reviewers agreed, not that they were right. If resources forced a lighter workflow, describe the safeguards honestly.

Generic quality scores hide distinct weaknesses. Match the tool to the design: RoB 2 for randomized trials [2], ROBINS-I for non-randomized intervention studies [3], and QUADAS-3 (or QUADAS-2 where a protocol already specifies it [4]) for diagnostic accuracy. Our risk-of-bias decision tree covers the choice in detail. Then show how the judgments shape interpretation: an observational treatment comparison stays vulnerable to confounding by indication however sophisticated the adjustment.

Pooling clinically incompatible populations, interventions or outcomes produces a number nobody can use. Define the estimand, eligibility criteria, effect measures and outcome time points before analysis, and assess clinical heterogeneity before statistical pooling decisions. Choose a fixed-effect or random-effects model from the research question and its assumptions, not from I² alone, and report the variance estimator and confidence-interval method, which matter most when studies are few.

Statistical heterogeneity is described by I², the share of variability not due to chance, and τ², the between-study variance; neither explains its cause or clinical importance. Read both alongside effect directions, confidence intervals and study characteristics, prespecify subgroup analyses or meta-regression, and avoid causal claims from study-level associations. Prediction intervals show the range of effects a new setting might see [6], though they are unstable with few studies. If heterogeneity stays unexplained, qualify the conclusion.

A statistically significant pooled estimate can still be biased, indirect, imprecise or clinically trivial. For health intervention reviews, rate the certainty of evidence with GRADE for each critical outcome across risk of bias, inconsistency, indirectness, imprecision and publication bias [7], and give absolute effects. A risk ratio of 0.90 at a 2% baseline risk is 2 fewer events per 1,000 people (2.0% to 1.8%), which readers can weigh; the relative figure alone hides that.

Use PRISMA 2020 and supply its checklist and flow diagram [1]. Distinguish records, reports and studies, explain full-text exclusions, state which studies feed each synthesis, and report funding, conflicts, software and data availability. Reconcile totals across abstract, methods, tables and figures. A missing flow diagram is correctable if the screening records exist, but PRISMA compliance does not prove the methods are valid.

Changing eligibility, outcomes or subgroups after results are visible looks like selective reporting. Write a dated protocol and complete protocol registration prospectively, in PROSPERO where the review is eligible. Compare the manuscript with the protocol, label post hoc analyses and record what changed, when and why. Registration supports transparency; it does not guarantee quality. See our guide to preventing undocumented PROSPERO deviations.

Small-study effects and funnel-plot asymmetry can arise from heterogeneity, chance, methodological differences or genuine links between effect size and precision, not only missing studies. Asymmetry tests are generally recommended only with at least 10 studies [8], a minimum rather than a guarantee of power, and a non-significant test does not rule out publication bias. Search registries and grey literature, compare registered with reported outcomes, and present trim-and-fill as an assumption-laden sensitivity analysis, not a correction.

Label effect measures, units, reference groups and the direction of benefit, and keep forest plots, tables and text consistent. Treating dependent effect estimates from the same study as independent understates uncertainty; use prespecified selection, multivariate models or robust variance estimation. Justify how zero-event studies are handled and test that choice in a sensitivity analysis.

A sound review can be desk-rejected for falling outside a journal’s scope. Check journal fit against recent publications and author guidelines, and align conclusions with risk of bias, precision and certainty: low-certainty or imprecise pooled results do not establish superiority, and pooled observational associations do not establish causation. Our meta-analysis submission rules guide compares what major journals ask for.

Fix validity problems before presentation. Work down the table, then re-run any analysis a fix touches.