Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Pre-Specified Analysis Plans

Pre-Specification of Statistical Analysis Plans in Clinical Trials: The Current State of the Evidence

Recent reviews of published trials indicate that statistical analysis plans are frequently changed, undocumented or unavailable, and that this constitutes a measurable and common threat to the validity of research conclusions.

Pubrica AcademyWritten by a clinical research methodologist Reviewed bya senior biostatistician Published10 October 2026 Last checked10 October 2026 Reading time7 minutes Sources10 peer-reviewed

A statistical analysis plan (SAP) written before any data are seen is one of the most direct safeguards against bias available in clinical research. When the analysis model, the population and the handling of missing data are chosen after the results have been examined, the distinction between genuine analysis and results-driven interpretation rapidly becomes unclear.

This article reviews the current evidence on how often analysis plans are pre-specified and how often trials deviate from them without explanation, and considers why this matters for the credibility of research design and analysis.

Learning objectives

  • Why statistical analysis plans should be written before data collection, not only before data are unblinded
  • What recent reviews found about the frequency with which published trials deviate from their pre-specified analysis
  • Which analysis decisions are most frequently changed retrospectively, and why these specific choices are most consequential
  • The characteristics of a well-governed SAP amendment process, in contrast with undocumented deviation

Common misconceptions

  • “Most published trials adhere to the analysis originally planned.” A review of 101 randomised trials published in six high-impact general medical journals found that, of the 89 trials with a publicly available pre-specified approach, only 25% had no unexplained discrepancy from it, while 61% had one or more.
  • “Changing the statistical model after seeing the data is a minor technical adjustment.” The same review found that the analysis model and the analysis population were the two most frequently changed elements (35% and 31% of discrepancies, respectively), and these are the choices most capable of directly altering an estimated treatment effect.
  • “A statistical analysis plan is a formality required only for regulatory submissions.” Methodology reviews describe pre-specification as a direct safeguard against selective analysis and selective reporting bias, and only 11% of reviewed trials reported that the responsible statistician remained blinded until the SAP was finalized or the database was locked.

Pre-specification as a methodological safeguard

Choosing or adjusting the statistical analysis after trial data have been seen, a practice sometimes termed p-hacking, can bias results, because analysis choices such as the population analysed, the statistical model used, the covariates adjusted for and the handling of missing data can each appreciably shift the estimated treatment effect1. Pre-specifying these decisions before the analysis is conducted, ideally before data collection, is recognised as important precisely because it both reduces the likelihood of such bias and facilitates its detection when it occurs2. Pre-specification is required by established frameworks including ICH-E9, SPIRIT, and CONSORT1.

Findings of a recent discrepancy review

A 2020 review examined 101 randomised trials published between January and April 2018 across six high-impact general medical journals1. Of these, 88% (89 of 101) had a publicly available pre-specified analysis approach for their primary outcome, usually documented in a protocol or SAP1. Among those 89 trials, only 25% showed no unexplained discrepancy between the planned and the conducted analysis; 61% had one or more unexplained discrepancies, and the remaining 15% could not be assessed because of incomplete reporting1.

Analysis elements most frequently changed and their significance

The same review found that discrepancies were most common in the choice of analysis model (35% of discrepancies) and the analysis population (31%), followed by covariate adjustment (26%) and the method for handling missing data (18%)1. These are not minor details; they are the factors that most directly determine the magnitude and even the direction of a reported treatment effect, which is why unexplained changes to them constitute the most concerning type of deviation.

The review also found that many pre-specified documents were dated after participant recruitment had begun, or were undated, so that earlier undocumented changes could have gone entirely undetected1. Only 11% of trials reported that the responsible statistician was kept blinded to outcome data until the SAP was finalized or the database locked1, a governance step intended specifically to prevent results from covertly shaping the “pre-specified” plan.

Governed amendments and undocumented deviations

Statistical analysis plans are legitimately living documents and may be amended as a trial progresses; however, methodology guidance recommends that such amendments be recorded on the trial registry, dated, and clearly marked to indicate whether a published SAP is preliminary or final2. Detailed, standardised SAP content also supports an important secondary aim: enabling an independent statistician to replicate the analysis from the documented plan alone2. The distinction between a defensible amendment and a credibility problem depends almost entirely on whether the change was dated, explained, and made before or after the data were seen; the review found that only two of the trials with discrepancies explained them directly in the trial publication itself, although more explained them in later protocol or SAP versions1.

Requirements of good practice in study design

Taken together, the current evidence supports treating the statistical analysis plan as part of the study design itself, finalised and dated before the data are analysed rather than written retrospectively to match the observed result. A trial whose model, population and missing-data method were fixed in advance, with a blinded statistician and a dated public record of any amendment, provides readers with a substantially stronger basis for trusting its conclusions than one in which those choices became visible only after publication.

Foundational evidence from comparisons of protocols and publications

The 2020 discrepancy review described above builds on an earlier, foundational study that first demonstrated the scale of the problem directly. A 2004 study in JAMA by Chan and colleagues compared the protocols of 102 randomized trials, obtained from research ethics committees, with their subsequent published reports, and found that approximately half of efficacy outcomes and about 65% of harms outcomes per trial were incompletely reported3. Sixty-two percent of the trials had at least one primary outcome added, changed or removed between the protocol and the publication, and the changes were not random: statistically significant outcomes were more likely than non-significant ones to be fully reported3. Notably, when the study authors wrote to the trial investigators to enquire about unreported outcomes, 86% of respondents denied that any such outcomes existed, even where the protocol clearly specified them3, a finding that supports independent protocol comparison, rather than self-report, as the only reliable means of detecting such deviation.

SPIRIT and the standardization of protocol content

Partly in response to evidence such as Chan's 2004 findings, an international group of methodologists and journal editors developed the SPIRIT 2013 Statement, published in the Annals of Internal Medicine, which defines a standard 33-item checklist of information every clinical trial protocol should contain, including the primary and secondary outcomes, the statistical methods for the primary outcome, and the planned handling of missing data4. A companion explanation-and-elaboration paper published the same year in the BMJ provides a detailed rationale for each checklist item, specifically so that protocols can be compared against a common, enforceable standard rather than each sponsor determining its own level of detail4. SPIRIT was designed specifically to create the dated, public baseline on which later discrepancy reviews, including Cro and colleagues' 2020 analysis, depend for comparison.

Trial registration and gaps in compliance

A pre-specified SAP is a safeguard against bias only if it is registered publicly, before the trial starts, where it can subsequently be checked. The International Committee of Medical Journal Editors formalized this expectation in a 2004 statement requiring prospective trial registration as a condition of publication in member journals, explicitly citing selective outcome reporting as the problem the policy was designed to prevent5. In the United States, the FDA Amendments Act of 2007 (FDAAA 801) subsequently made results reporting on ClinicalTrials.gov a legal requirement for many trials, not merely a journal-publication condition6. Nevertheless, compliance with that legal requirement has lagged considerably: a 2015 analysis in the New England Journal of Medicine found that only 13.4% of 13,327 applicable trials completed between 2008 and 2012 reported summary results within the legally required 12 months, a proportion that had risen to only 38.3% by late 20137. A 2016 follow-up article by some of the same authors, describing a strengthened “final rule” intended to resolve reporting ambiguities, acknowledged that compliance had remained low across much of the clinical research enterprise even after the legal requirement had been in force for several years8.

Prospective monitoring of outcome reporting: the COMPare project

Rather than reviewing trials retrospectively, the COMPare project, a 2015-2016 initiative based at the University of Oxford, checked every trial published in five leading medical journals over several months in real time against its registered or published protocol, and submitted correction letters wherever the reported primary or secondary outcomes of a trial diverged from those pre-specified9. The project's published analysis of the correction letters it submitted found that most journals responded to a documented discrepancy reluctantly or dismissively rather than correctively; the project's authors argue that this indicates that outcome switching persists less because it is undetectable than because journals and editors do not currently treat its correction as a priority once a paper has been published9. An earlier systematic review had already established that the problem is not confined to any single journal or period: pooling evidence across multiple cohorts of trials, it found consistent evidence that statistically significant outcomes were more likely than non-significant ones to be fully and promptly reported, across outcome reporting generally and not only in the specific cohorts examined by Chan's group10.

Key points

Key points

Most trials in a large sample of recent high-impact trials had at least one unexplained discrepancy between the planned and the conducted statistical analysis, and the elements most frequently changed, the analysis model and population, are those most capable of altering a trial’s conclusion.

Practical application

Practical application

  1. The statistical analysis plan (SAP) should be finalized and dated before any data are analyzed. The analysis model, population definition, covariate adjustment plan and missing-data method should be fixed in a dated document before outcome data are unblinded.
  2. The responsible statistician should remain blinded until the SAP is finalized. This governance step, reported in only 11% of the reviewed trials, protects against the plan being influenced by knowledge of the results.
  3. Amendments should be registered publicly, with dates and reasons. If the SAP legitimately requires change during the trial, the amendment should be recorded on the trial registry, with an explicit statement of whether a published version is preliminary or final.
  4. The analysis model and population should be designated as items requiring the greatest scrutiny. Because these two elements accounted for the majority of discrepancies in the review, they should receive the most rigorous approval before data collection begins.

Frequently asked questions

In a 2020 review of 101 trials in six high-impact journals, 61% of those with a publicly available pre-specified plan had one or more unexplained discrepancies between the planned and conducted analysis.

The analysis model (35% of discrepancies) and the analysis population (31%) were the most commonly altered elements, followed by covariate handling and missing-data methods.

Yes; however, methodology guidance recommends that any amendment be dated, recorded on the trial registry and clearly explained. The concern is undocumented deviation, not amendment itself.

References

  1. Cro S, Forbes G, Johnson NA, Kahan BC. Evidence of unexplained discrepancies between planned and conducted statistical analyses: a review of randomised trials. BMC Medicine. 2020. read 10 October 2026. View on PubMed Central
  2. Hemming K, Kearney A, Gamble C, Li T, Juni P, Chan AW, Sydes MR. Prospective reporting of statistical analysis plans for randomised controlled trials. Trials. 2020;21:898. read 10 October 2026. View on PubMed Central
  3. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to publications. JAMA. 2004;291(20):2457-2465. read 10 October 2026. View source
  4. Chan AW, Tetzlaff JM, Altman DG, et al. SPIRIT 2013 Statement: defining standard protocol items for clinical trials. Annals of Internal Medicine. 2013;158(3):200-207. read 10 October 2026. View source
  5. International Committee of Medical Journal Editors. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. CMAJ. 2004;171(6):606-607. read 10 October 2026. View source
  6. U.S. Food and Drug Administration / ClinicalTrials.gov. History, Policies, and Laws (FDAAA 801). read 10 October 2026. View source
  7. Anderson ML, Chiswell K, Peterson ED, Tasneem A, Topping J, Califf RM. Compliance with results reporting at ClinicalTrials.gov. New England Journal of Medicine. 2015;372(11):1031-1039. read 10 October 2026. View source
  8. Zarin DA, Tse T, Williams RJ, Carr S. Trial reporting in ClinicalTrials.gov — the final rule. New England Journal of Medicine. 2016;375(20):1998-2004. read 10 October 2026. View source
  9. COMPare Project. Bennett Institute for Applied Data Science, University of Oxford. read 10 October 2026. View source
  10. Dwan K, Altman DG, Arnaiz JA, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE. 2008;3(8):e3081. read 10 October 2026. View source

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Pre-Specification of Statistical Analysis Plans in Clinical Trials: The Current State of the Evidence. Pubrica. https://pubrica.com/academy/concepts-definitions/the-current-state-of-data-and-evidence-biostatistics-clinical-research-design-and-analysis/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

SAP pre-specification checklist

This checklist should be used before a trial's database lock to confirm the governance of the analysis plan.

[ ] Analysis model, population, covariates, and missing-data method are documented and dated
[ ] SAP was finalized before outcome data were unblinded
[ ] Responsible statistician remained blinded until SAP finalization or database lock
[ ] Any mid-trial amendments are dated and recorded on the trial registry
[ ] Published SAP version is clearly marked preliminary or final
[ ] Any deviation from the pre-specified plan is explained in the publication

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s biostatistics team drafts and finalizes statistical analysis plans before database lock and documents any amendment transparently, so that a study’s conclusions can withstand this form of scrutiny.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Pubrica can review whether a statistical analysis plan is finalized and dated before data are analyzed.

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.