Answering a statistical objection

What a statistical objection usually means

Reviewers rarely object to statistics for statistical reasons. They object because a number in your results does not behave the way they expect given your design, and the analysis is the first place to look. That distinction matters, because it tells you what a convincing answer has to do.

Three objections account for most of what we see. The analysis ignores structure in the data — clustering, repeated measures, nesting. The analysis answers a slightly different question from the one the paper claims to answer. Or the analysis is fine and the discussion claims more than it supports, and the reviewer has attached the complaint to the nearest technical target.

The third is the commonest and the most often mishandled, because authors defend an analysis nobody was really attacking.

The answer is usually to run it

If a reviewer names an analysis, run it. Not because they are always right, but because the cost of running it is almost always lower than the cost of another round arguing about whether it was necessary.

Then report three things, in this order: what the alternative analysis gives, whether the conclusion survives, and where in the manuscript the change now sits. Reporting only the first two leaves the editor unable to verify; reporting only the third leaves the reviewer unsatisfied.

When the conclusion does not survive

Sometimes the requested analysis changes the answer. This is not a disaster and it is not the moment to look for a third analysis that restores the original result — that is the behaviour the reviewer is watching for.

What works is to report both, say which is primary and why the decision was made on methodological grounds rather than on which result you preferred, and narrow the claim to what the primary analysis supports. A paper that says “the effect is smaller than we first estimated and remains present in the primary analysis” is publishable. A paper that quietly presents the third analysis is a retraction risk.

When you cannot run it

Some requests are not feasible: the data will not support the model, the sample is too small for the method, the variable was never collected. Say so in the first sentence, give the real constraint, and offer what you did instead.

• Too few events per variable for the model they suggested — state the events-per-variable ratio and cite the rule you are applying.
• The variable was not collected — say so plainly and add it to the limitations rather than substituting a proxy without flagging it.
• The design does not permit it — a cross-sectional dataset cannot answer a question about change, and saying so is a better answer than a model that appears to.

Say which question you answered

A surprising share of statistical objections dissolve once the paper states its estimand explicitly: the population, the treatment condition, the outcome, how intercurrent events are handled, and the summary measure. The ICH E9(R1) addendum formalised this for trials, and the discipline helps outside trials too 1.

When a reviewer says “this should have been intention-to-treat” and you ran a per-protocol analysis on purpose, the disagreement is about the estimand, not the arithmetic. Naming it converts an argument into a decision the editor can adjudicate.

Wording that works, and wording that does not

Common questions

@misc{response_to_reviewers_2026,
  author       = {{Pubrica Academy}},
  title        = {Answering a statistical objection},
  howpublished = {Pubrica},
  year         = {2026},
  month        = {feb},
  url          = {https://pubrica.com/insights/response-to-reviewers/},
  note         = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}