How to Prevent Undocumented Protocol Deviations in PROSPERO-Registered Systematic Reviews: A Concise Guide to HTA Assessment Risk

High-Impact Journals

Interesting topics

Systematic reviews registered in PROSPERO represent the evidence backbone for health technology assessment (HTA) bodies making multibillion-dollar reimbursement decisions [1,2].

When government agencies like CADTH (now Canada’s Drug Agency), NICE, or regional HTA councils decide whether to fund new medications or diagnostics, they depend on high-quality evidence synthesis conducted according to predetermined protocols [2]. However, registration alone does not guarantee protocol adherence [3,4].

QUICK ANSWER

To prevent undocumented protocol deviations, compare the final systematic-review manuscript against its PROSPERO registration before submission; document every substantive methodological change, explain its rationale, distinguish prespecified from exploratory analyses, and update the registered record where appropriate [5,6]. This creates a clear audit trail for journal and HTA review.

Protocol transparency concerns can increase scrutiny during HTA assessment and may require additional clarification, analysis, or evidence review. The practical impact depends on the HTA agency, submission pathway, therapeutic area, and nature of the methodological concern. Avoiding undocumented protocol deviations therefore helps reduce avoidable uncertainty during evidence assessment.

Why PROSPERO Registration Does Not Guarantee Transparency

PROSPERO registration provides a public record of the review protocol, but registration alone does not demonstrate that the final review followed the registered methods [1]. When substantive methodological changes occur, reviewers should document what changed, why the change was made, and how the change affects interpretation. The final manuscript should clearly distinguish prespecified methods and analyses from subsequent or exploratory decisions [6].

PROSPERO provides a public record of registered systematic-review protocols, but registration does not by itself guarantee adherence to the registered methods. Researchers remain responsible for documenting substantive methodological changes and explaining deviations in the final review [7].

Why Reclassification Is Not Just a Registration Update

A higher class can change the conformity-assessment route and the evidence expected by the notified body. MDR Article 52 establishes different conformity pathways by class, while Article 54 creates a Clinical Evaluation Consultation Procedure for certain Class III implantable and Class IIb active devices. [3]

The distinction is critical: an EMDN update alone is a registration consistency check; a changed classification is a regulatory impact assessment; and a higher classification is a CER, PMCF, risk-management and notified-body readiness review.

THE CRITICAL PROBLEM

PROSPERO registration provides a publicly accessible record of the planned review methods, but registration alone does not ensure that the published review follows the registered protocol. When methods change during the review, authors should document the change, explain why it was necessary, and clearly distinguish protocol-specified analyses from post hoc or exploratory analyses [6].

The published review should allow readers to understand what was planned initially and what changed during execution. Instead, deviations often appear silently in published papers, discoverable only through manual comparison by HTA assessors [3,8].

How Often Do Registered Reviews Deviate from Their Protocol?

Current systematic-review reporting practice emphasizes transparent reporting of protocol registration, amendments, eligibility criteria, outcomes, searches, analyses, and deviations [5]. A current protocol-fidelity review should therefore compare the registered record with the final manuscript across each of these methodological domains rather than treating registration itself as evidence of adherence. However, deviation estimates vary according to the review sample, definition of a deviation, registration source, and assessment method [4].

Figure 2. Frequency of protocol-to-publication changes and the share left unreported [4].

Protocol-Fidelity Pre-Submission Checklist [5], [9]
Sno	Check item	Status
1	Population and eligibility criteria match the registered protocol.	☐ Yes ☐ No
2	Intervention and comparator definitions are consistent.	☐ Yes ☐ No
3	Primary and secondary outcomes are consistent.	☐ Yes ☐ No
4	Search databases, dates, restrictions, and strategies are accurately reported.	☐ Yes ☐ No
5	Data-extraction methods have not materially changed without documentation.	☐ Yes ☐ No
6	Risk-of-bias methods are consistent with the planned approach.	☐ Yes ☐ No
7	Statistical methods and effect measures are consistent.	☐ Yes ☐ No
8	Subgroup and sensitivity analyses are identified as prespecified or exploratory.	☐ Yes ☐ No
9	All substantive amendments are documented.	☐ Yes ☐ No
10	The final manuscript clearly explains important deviations.	☐ Yes ☐ No

Published research has identified discrepancies between registered protocols and subsequently published systematic reviews [3,8]. However, reported deviation rates vary according to the review population, definition of a protocol deviation, registration source, and assessment method. These findings support the need for systematic protocol-to-publication comparison rather than a single universal deviation benchmark [4].

Illustrative Hypothetical Example: Sensitivity-Analysis Transparency

Consider a hypothetical systematic review evaluating a diabetes medication. The registered protocol specifies HbA1c reduction at 12 weeks as the primary outcome. The initial meta-analysis produces a non-significant result.

Sensitivity analyses are not inherently problematic when introduced after registration [10]. The methodological concern arises when post hoc analyses are presented without disclosure, selected selectively from multiple possible analyses, or used to replace the prespecified primary analysis without explanation [8,11].

A defensible approach is to retain the prespecified primary analysis, report the additional analysis separately, explain why it was conducted, and assess whether the interpretation changes across analytical specifications [6,10].

SCENARIO: When preliminary meta-analysis shows 0.3% HbA1c reduction (p = 0.08, not significant), the team conducts post hoc sensitivity analyses excluding studies with >20% dropout rates, non-English publications, those published before 2015, and studies with unclear bias risk. After removing 8 of 20 studies, the effect becomes 0.5% reduction (p = 0.03, significant).

Figure 3. Hypothetical illustration of how undisclosed post hoc exclusions can shift a result.

HTA Assessment: The Assessor Compares the Paper’s Methods to PROSPERO and Finds…

Finding on comparison	Why it raises concern
No mention of sensitivity analyses in registration	Analyses cannot be shown to be prespecified
Exclusion criteria appear data-driven (chosen after seeing results)	Suggests selective analysis [8]
No pre-specified cutoffs for dropout rates or publication year	Thresholds could have been tuned to the result
Pattern: criteria systematically strengthened weak primary results	Signals possible selective reporting [12]

Potential HTA consequence

If an assessor considers the additional analyses to have introduced a meaningful risk of selective reporting, confidence in the evidence may be reduced [9,12]. The exact consequence will depend on the HTA methodology, the strength of the remaining evidence, and whether the deviation materially affects the decision-relevant results [2].

Better approach

Prespecify sensitivity analyses where possible, document amendments transparently, report both prespecified and exploratory analyses, and explain how each analysis affects interpretation [6,10].

What HTA Assessors Actually Check

HTA agencies apply their own evidence-assessment frameworks, and the specific review process varies by jurisdiction and submission type [2]. For protocol-fidelity assessment, an evidence-synthesis team should be prepared to demonstrate consistency between the registered protocol and published methods, including:

Domain	What should be consistent with the protocol
Search strategy	Databases, search dates, language restrictions, and date limits
Eligibility	Population, intervention, comparator, outcomes, and study-design criteria
Outcomes	Primary and secondary outcomes and their definitions
Analysis	Statistical methods, subgroup analyses, and sensitivity analyses
Amendments	What changed, why it changed, and when the change occurred
Reporting	Whether deviations and exploratory analyses are clearly identified

A NOTE ON ROBIS

ROBIS is a tool for assessing risk of bias in systematic reviews [12]. It can help reviewers evaluate methodological concerns across the review process. However, ROBIS should not be presented as a universal HTA requirement. HTA organizations may apply their own evidence-assessment frameworks, and the role of ROBIS should therefore be described according to the specific assessment context [2].

Important or inadequately explained protocol deviations may reduce confidence in the review findings and can become relevant to evidence assessment [9]. Their effect on an HTA recommendation or funding decision depends on the specific evidence base and assessment framework.

How to Document Legitimate Deviations Transparently

When deviations become necessary, transparent documentation requires four steps [6,10].

Figure 4. Four steps for documenting a legitimate protocol deviation.

Step 1 Record the amendment decision	Document what changed, when the change was identified, why it was necessary, and whether the decision occurred before or after relevant review results were available.
Step 2 Update the registered record where appropriate	Follow the current requirements and available mechanisms of the relevant registry [7].
Step 3 Report the change in the manuscript	Describe the original method, revised method, rationale, timing, and potential effect on interpretation [6].
Step 4 Separate prespecified and exploratory analyses	Clearly identify analyses that were planned versus analyses introduced after results became available [10].

Why Industry-Funded Reviews Face Higher Scrutiny

Industry-sponsored systematic reviews should provide particularly clear disclosure of funding, author roles, conflicts of interest, protocol registration, amendments, and analytical decisions [5]. Sponsorship alone does not establish methodological bias. The key issue is whether the review methods are transparent, reproducible, appropriately prespecified, and consistently reported.

For industry teams, a defensible evidence-synthesis process should therefore make it easy for an external reviewer to distinguish prespecified decisions from decisions made after results became available. Industry-sponsored research has been shown to report favourable results and conclusions more often than non-industry research, which is why assessors look closely at sponsored reviews [13].

HTA assessors ask

Did the change favour the sponsor’s product? Were analyses selectively reported? Industry sponsorship doesn’t establish methodological bias by itself, but funding sources must be transparently reported as they affect HTA assessor interpretation of review credibility

[5,13].

Critical principle

Industry teams seeking credibility must exceed documentation standards, not merely meet them. Prospective registration amendments, pre-specified sensitivity analyses, and published deviation justifications are minimal requirements.

Pubrica Protocol Integrity Review Service

Pubrica can support systematic-review teams with a pre-submission protocol-fidelity review designed to identify inconsistencies between the registered protocol and final manuscript.

The review can assess:

Protocol-to-manuscript consistency
Eligibility and outcome changes
Search-method changes
Statistical and sensitivity-analysis changes
Documented versus undocumented deviations
Classification of potentially consequential deviations
Transparency of amendment reporting

The objective is to identify methodological and reporting issues before journal or HTA submission. The review does not guarantee journal acceptance, HTA approval, or a particular reimbursement outcome.

Key Takeaways

DO	DON’T
Register in PROSPERO before study selection	Modify search strategies without documenting changes and rationale
Conduct pre-registration feasibility checks	Tighten eligibility criteria after seeing preliminary results
Document every deviation in PROSPERO amendments before publication	Extract “surprise” outcomes absent from PROSPERO
Pre-specify all sensitivity analyses	Conduct selective sensitivity analyses
Report all analyses (favourable and unfavourable) transparently	Hide unfavourable analyses in supplementary materials
Conduct independent protocol integrity audit before submission	Wait until after publication to acknowledge deviations

Get expert support from Pubrica for addressing reviewer comments, preparing point-by-point rebuttals, and strengthening manuscripts for journal resubmission. Prepare your revised manuscript with confidence.

References

Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews. Systematic Reviews, 1, 2. https://doi.org/10.1186/2046-4053-1-2
National Institute for Health and Care Excellence. (2022). NICE health technology evaluations: The manual (PMG36). NICE. https://www.nice.org.uk/process/pmg36
Tricco, A. C., Cogo, E., Page, M. J., Polisena, J., Booth, A., Dwan, K., MacDonald, H., Clifford, T. J., Straus, S. E., & Moher, D. (2016). A third of systematic reviews changed or did not specify the primary outcome: A PROSPERO register study. Journal of Clinical Epidemiology, 79, 46–54. https://doi.org/10.1016/j.jclinepi.2016.03.025
Siebert, M., et al. (2023). Assessing the magnitude of changes from protocol to publication—A survey on Cochrane and non-Cochrane systematic reviews. PeerJ, 11, e16016. https://doi.org/10.7717/peerj.16016
Page, M. J., McKenzie, J. E., Bossuyt, P. M., et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Page, M. J., Moher, D., Bossuyt, P. M., et al. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160. https://doi.org/10.1136/bmj.n160
Page, M. J., Shamseer, L., & Tricco, A. C. (2018). Registration of systematic reviews in PROSPERO: 30,000 records and counting. Systematic Reviews, 7, 32. https://doi.org/10.1186/s13643-018-0699-4
Kirkham, J. J., Altman, D. G., & Williamson, P. R. (2010). Bias due to changes in specified outcomes during the systematic review process. PLoS ONE, 5(3), e9810. https://doi.org/10.1371/journal.pone.0009810
Shea, B. J., Reeves, B. C., Wells, G., et al. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008. https://doi.org/10.1136/bmj.j4008
Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds.). (2024). Cochrane Handbook for Systematic Reviews of Interventions (Version 6.5). Cochrane. https://www.training.cochrane.org/handbook
Kirkham, J. J., Dwan, K. M., Altman, D. G., et al. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, c365. https://doi.org/10.1136/bmj.c365
Whiting, P., Savović, J., Higgins, J. P. T., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005
Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017). Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews, 2(2), MR000033. https://doi.org/10.1002/14651858.MR000033.pub3

AI and ML Services

Our Quality