Your Systematic Review Is Registered on PROSPERO, But Your Deviations Are Not Documented, Your Search Strategy Changed Post-Registration, and Your Sensitivity Analyses Were Added After Seeing Results: How Protocol Transparency Failures Undermine Evidence Credibility at HTA

High-Impact Journals

Interesting topics

A Critical Analysis of Protocol Integrity in PROSPERO-Registered Systematic Reviews for Health Technology Assessment

Systematic reviews registered in PROSPERO (Prospective Register of Systematic Reviews) represent the evidence backbone upon which modern health technology assessment (HTA) bodies make multibillion-dollar reimbursement decisions. When a government agency like CADTH (Canadian Agency for Drugs and Technologies in Health), NICE (National Institute for Health and Care Excellence), the Australian Pharmaceutical Benefits Scheme, or regional HTA councils must decide whether to fund a new cancer drug, cardiovascular intervention, or diagnostic test, they depend fundamentally on high-quality synthesis of evidence conducted according to predetermined protocols. The registration of these protocols in publicly accessible repositories before study selection commences was supposed to prevent selective outcome reporting, reduce bias, and ensure that the analyses reported in published papers matched the analyses that reviewers announced they would conduct. However, registration alone does not guarantee protocol adherence. Undocumented changes to eligibility criteria, search strategies, outcomes, or analyses can reduce transparency and make it difficult for readers and HTA assessors to determine whether the published review reflects the registered methodology. [1,2,3]

1. Why PROSPERO Registration Alone Does Not Guarantee Transparency

PROSPERO registration creates an audit trail, but audit trails depend on adherence. When systematic review teams deviate from their registered protocol without documentation, they undermine the foundational assumption underlying the entire HTA evidence synthesis process: [2,3] that reviewers conducted predetermined analyses rather than selected analyses based on what the data revealed. The registration repository now hosts over 600,000 reviews, yet Published studies have identified protocol deviations in systematic reviews, [5] although reported frequencies vary substantially according to the review sample, deviation definition, and assessment method. This disconnected between registration expectations and reported practice represents one of the most persistent threats to evidence credibility in health technology assessment. When HTA bodies like CADTH, NICE, or the Belgian HTA Council conduct their own rapid assessments, they increasingly encounter published reviews whose methods differ substantially from what was registered, creating uncertainty about whether effect estimates reflect genuine evidence or selective hypothesis testing.

Registration systems operate under the assumption that a frozen protocol prevents selective outcome reporting and allows HTA assessors to identify whether reported findings align with announced intent. However, this protection evaporates when deviations occur without transparent documentation. Unlike registered clinical trials, which have formal amendment procedures and FDA/EMA oversight, systematic review protocols rely primarily on researcher integrity and journal editor vigilance [3]. Neither guarantees protocol adherence. The absence of centralized amendment tracking where PROSPERO updates are optional rather than mandatory means that many deviations never reach the registration repository. Instead, they appear silently in published papers, discoverable only through manual comparison by astute readers. [1]

2. How Often Do Registered Reviews Deviate from Their Protocol?

Evidence from multiple systematic reviews of systematic reviews reveals disturbing patterns. Studies assessing protocol adherence have identified discrepancies between registered protocols and subsequently published systematic reviews. The frequency and type of deviations vary according to the review population, the definition of a protocol deviation, and the assessment method used. Common deviations include changes to eligibility criteria, database selection, search strategies, primary or secondary outcomes, risk-of-bias methods, subgroup analyses, and statistical or sensitivity analyses. The key concern is not whether every deviation occurred, but whether each change was justified, documented, transparently reported, and considered when interpreting the findings.[2,3] The most common deviations involve: (a) search strategy modifications post-registration, sometimes justified by database indexing updates but frequently undocumented; (b) addition or removal of eligibility criteria, particularly population or intervention specificity; (c) addition of post-hoc outcomes or subgroup analyses, often appearing in results without explanation; and (d) changes to primary or secondary outcome hierarchies, elevating secondary outcomes to prominence when primary outcomes show unexpected results.

These rates are not trivial. For HTA bodies conducting rapid technology assessment or informing reimbursement decisions, each undocumented change represents a potential source of bias that was not pre-announced and therefore not subject to peer scrutiny before analysis commenced. Furthermore, when industry-sponsored reviews exhibit higher deviation rates than academic reviews, regulatory confidence in the evidence base erodes. This creates a cascading credibility problem: HTA assessors must expend additional resources auditing published methods against PROSPERO entries, and when discrepancies are discovered, confidence ratings for the entire review downgrades, regardless of whether individual deviations were methodologically significant. [7]

3. Search Strategy Changes After Registration

Changes to search strategies are among the protocol deviations that can materially affect study identification. When search methods change after registration, reviewers should document the modification, explain its rationale, and distinguish the registered strategy from the strategy ultimately executed. [1,2,3] When a review team registers their protocol with PROSPERO, they commit to specific databases, search terms, language restrictions, and date ranges. However, during the execution phase, often 6–12 months after registration, teams discover that their registered search strategy either retrieves too few results (risking under-identification of relevant studies), too many results (creating screening burden exceeding available resources), or misses entire bodies of literature due to database-specific indexing variations. Rather than formally amending their PROSPERO registration by creating a documented amendment entry, teams adjust their search strategy and report the adjusted version in the published Methods section without flagging the deviation or explaining the rationale for the change.

This creates a critical documentation gap with real consequences. A reader comparing the PROSPERO entry to the published paper must manually parse both documents, line-by-line, and detect the discrepancy, a quality assurance task that many peer reviewers and journal editors do not perform systematically. HTA assessors conducting their own evidence synthesis may base initial confidence in the original review’s search completeness on the PROSPERO registration, unaware that the actual executed search differed substantially.[6] Search strategy changes can meaningfully affect which studies are identified, particularly when using complex Boolean operators, proximity operators, or discovery of new medical subject headings (MeSH) or database-specific controlled vocabulary. Search results can vary substantially across bibliographic databases because indexing systems, controlled vocabularies, coverage, and search functionality differ. Search strategies should therefore be adapted and documented for each database rather than assumed to perform identically across platforms. [6]

4. Eligibility Criteria Amendments: When They Are Justified

Eligibility criteria modifications during study selection are sometimes legitimate. If a review team’s registered eligibility criteria prove unintelligible, unpredictably inclusive (resulting in 1,000+ studies for screening), or inconsistently interpretable across reviewers, amending the criteria can be ethically justified provided: (a) the amendment occurs before or during initial full-text screening (not after results are known), (b) all studies are re-screened under the amended criteria,[4] (c) the amendment is explicitly documented in a protocol amendment, and (d) the PROSPERO entry is updated.[2]

Conversely, amendments that tighten eligibility criteria after observing that preliminary studies do not support a desired conclusion or amendments that broaden criteria because initial results proved sparse are suspect. HTA assessors must distinguish between operational necessity and post-hoc specification.

5. Outcome Additions and Selective Reporting Risk

Adding outcomes after study selection or data extraction completion is a red flag for selective reporting bias. When a review team’s registered protocol specifies three primary outcomes but the published paper reports five additional secondary outcomes extracted from the included studies, two scenarios are possible: (a) the outcomes were always extractable but were added for completeness during revision, or (b) the outcomes were added because initial results were unexpectedly favourable or unfavourable, and the team sought to fortify (or bury) their findings through additional outcome reporting.[3,4]

PROSPERO registrations that list vague outcome definitions (e.g., “safety outcomes”) create latitude for outcome addition. More problematic are registrations that omit outcomes entirely, claiming they were secondary post-hoc discoveries. Cochrane methodology now flags outcome additions as high risk of bias if post-registration additions are not formally documented; HTA bodies increasingly expect the same rigour. The distinction matters because outcome selectivity can inflate apparent treatment effects and distort comparative effectiveness conclusions.

6. Post-Hoc Sensitivity Analyses: Legitimate vs Suspect

Sensitivity analyses conducted after observing main results occupy an ethical grey zone. If a registered protocol pre-specifies a sensitivity analysis (e.g., “exclude studies at high bias risk; reanalyse”), the analysis is legitimate provided the registered criteria are applied exactly as registered. If the analysis was not pre-specified, reviewers must be transparent about its exploratory nature. [2,3]

Suspect patterns include: (a) sensitivity analyses whose criteria are selected to strengthen effect estimates if main results are weak, or weaken estimates if main results are strong; (b) “multiple sensitivity analyses” without controlling for multiple comparisons; (c) sensitivity analyses that appear only in supplementary appendices if they contradict main findings; and (d) analyses reported as sensitivity analyses but constructed using outcome- or population-specific inclusion criteria that were never pre-registered. HTA assessors must examine whether sensitivity analyses test robustness or selectively support a conclusion.

7. What HTA Assessors Actually Check About Your Protocol

Health technology assessment bodies such as CADTH (Canadian Agency for Drugs and Technologies in Health), NICE (National Institute for Health and Care Excellence), the Belgian HTA Council, and regional/national assessment agencies employ structured protocols to evaluate systematic review quality before their recommendations inform reimbursement decisions. When assessing protocol adherence, HTA evaluators specifically examine: (1) whether the registered PROSPERO entry and published paper match exactly for databases, search terms, language restrictions, and date ranges; (2) whether eligibility criteria reported in the paper match the PROSPERO registration, including population, intervention, comparator, and outcome specifications; (3) whether primary and secondary outcomes are identical in registration and publication, with unchanged hierarchies; (4) whether any protocol amendments are documented in PROSPERO as formal amendment entries or acknowledged in the paper; (5) whether the paper transparently acknowledges any deviations and provides explicit justification for changes; and (6) whether outcome hierarchies (primary vs secondary), subgroup analyses, and sensitivity analyses remained stable throughout the review process.

ROBIS is a validated tool for assessing risk of bias in systematic reviews and can complement assessment of protocol specification and reporting. Where discrepancies exist between planned and reported methods, reviewers can use ROBIS alongside other methodological and reporting criteria to assess whether the review process may have introduced bias. ROBIS should therefore be presented as one component of methodological appraisal rather than as a universal HTA requirement.[5,8]

8. How to Document Legitimate Deviations Transparently

When deviations become necessary, transparent documentation requires three steps: (1) Document each protocol deviation or amendment with the change, rationale, date, and point in the review process at which it occurred. Where PROSPERO permits an update or amendment to the registration, record the change in the registry and retain an auditable version history. The published review should also identify material deviations and explain how they may affect interpretation of the findings. (2) In the paper’s Methods section, include a subsection titled “Protocol Amendments” that explicitly lists all deviations, their justifications, and whether outcomes or analyses were re-conducted under amended criteria. (3) In the Results section, report outcomes under both the original protocol and amended protocol if feasible, allowing readers to assess robustness. For sensitivity analyses, distinguish pre-specified analyses (pre-registration) from exploratory analyses (post-result), and report both sets with clear labelling. [1,2,3]

9. Protocol Version Control and Change Management

Systematic review teams that plan to conduct large evidence syntheses should implement formal version control systems [2,3] analogous to clinical trial amendment protocols. This involves: (a) maintaining a master protocol document with dated version numbers; (b) creating a change log documenting every modification, its rationale, approval date, and approver; (c) freezing the protocol at key decision points (study selection, data extraction, analysis specification) with signatures or attestation; and (d) designating a protocol steward independent of bias toward results (e.g., a co-investigator not involved in data synthesis who reviews and approves amendments).

Open science platforms such as Open Science Framework (OSF) or the Systematic Review Protocol Repository (SRProtocol) now enable timestamped protocol uploads that create immutable records. Industry-led reviews (e.g., for product approval) should employ these systems routinely.

10. Why Industry-Funded Reviews Face Higher Scrutiny

Any systematic review sponsored by firms that manufacture pharmaceuticals, medical devices, or diagnostic tests is subject to rigorous examination of its protocol by HTA organizations since there are financial motivations for a conflict of interest. Protocol deviations in industry-sponsored reviews warrant the same level of documentation and scrutiny applied to academically sponsored reviews. Sponsorship should not, by itself, be treated as evidence of bias; however, transparent disclosure of funding, protocol changes, and their potential influence on the review process is important for independent assessment. [3,6] Did the change favour the sponsor’s product? Were analyses selectively reported? Industry sponsorship does not by itself establish methodological bias. However, funding sources and potential conflicts of interest should be transparently reported because they may affect how readers and HTA assessors interpret the credibility and independence of the review process.

Funding source should not be treated as a substitute for methodological assessment. Industry-sponsored review teams should therefore apply the same principles of prospective registration, transparent amendments, pre-specified analyses, conflict-of-interest disclosure, and clear reporting of deviations, but this reflects the reality of HTA evaluation. Industry-sponsored systematic reviews should apply the same principles of prospective registration, transparent protocol changes, pre-specified analyses, funding disclosure, and conflict-of-interest reporting as other evidence syntheses. Sponsorship alone does not establish methodological bias; however, transparent reporting allows HTA assessors and other readers to evaluate whether funding or sponsor involvement could have influenced protocol development, analysis, or interpretation. Where material deviations occur, their rationale and potential implications should be explicitly reported.

11. Pubrica Protocol Adherence Audit Checklist

The following checklist enables systematic review teams and HTA assessors to evaluate protocol fidelity. Teams should complete this checklist during manuscript preparation; HTA assessors should complete it during evidence appraisal.

Audit Criterion

Compliant

Non-Compliant

Protocol prospectively registered in PROSPERO before study selection began

☐

☐

PROSPERO registration ID and access link provided in Methods section

☐

☐

Databases searched, date ranges, and search strategies match PROSPERO registration exactly

☐

☐

If search strategies were modified post-registration, a PROSPERO amendment documents the changes and rationale

☐

☐

Study eligibility criteria reported in paper match PROSPERO registration

☐

☐

If eligibility criteria were amended, PROSPERO amendment and rationale are documented; all studies were re-screened under amended criteria

☐

☐

Primary outcomes defined in paper match PROSPERO exactly; hierarchy (primary vs secondary) unchanged

☐

☐

Any outcomes added post-registration are labelled as “exploratory” or “post-hoc”; their addition is justified in Methods section

☐

☐

Sensitivity analyses reported in Results section distinguish pre-specified (PROSPERO-registered) from exploratory (post-result)

☐

☐

Subgroup analyses reported in Results section match pre-registration; post-hoc subgroups labeled as exploratory

☐

☐

A dedicated “Protocol Amendments” subsection in Methods lists all deviations with dates and rationales

☐

☐

Funding source disclosed; any industry sponsor involvement in protocol development, amendments, or outcome interpretation documented

☐

☐

ROBIS risk-of-bias assessment conducted; protocol specificity domain scored; report provided in paper or supplement

☐

☐

 

Pubrica Internal Protocol-Fidelity Framework: The following scoring thresholds are proposed as an internal Pubrica quality-control framework and are not an established PROSPERO, ROBIS, Cochrane, or HTA-wide scoring standard.

Interpretation of the Pubrica Internal Protocol-Fidelity Framework This scoring framework is a proposed internal Pubrica quality-control tool and is not an established PROSPERO, ROBIS, Cochrane, or HTA-wide scoring standard. Scores should therefore be used for internal quality assurance rather than as evidence of formal HTA compliance. Higher scores indicate greater alignment with the checklist criteria, while lower scores identify areas requiring additional documentation or methodological review

12. How Pubrica Would Manage Protocol Integrity from Registration to Submission

The following workflow represents a proposed Pubrica quality-control framework for improving protocol transparency from registration through manuscript submission.

The Pubrica  uses four phases: pre-registration planning to confirm protocol feasibility, amendment control to document justified deviations, version control to maintain dated protocol records, and transparency reporting to summarize amendments and adherence. This approach strengthens prospective governance, transparency, reproducibility, and HTA readiness.

Pubrica’s Protocol Adherence Audit Framework evaluates your review against ROBIS, PRISMA 2020, and HTA-specific standards identifying deviations before submission costs time and credibility.

Get started:

  • Request a Protocol Audit Our checklist identifies documentation gaps across 12 domains
  • Schedule a 30-Minute Consultation Discuss your protocol amendment strategy
  • Download: HTA-Ready Protocol Checklist Free guide to transparent systematic review documentation

Systematic review teams that audit their protocol adherence before submission avoid HTA assessor downgrades and prevent costly re-analysis post-publication. Contact our team to discuss your protocol documentation strategy.

 

Get expert support from Pubrica for addressing reviewer comments, preparing point-by-point rebuttals, and strengthening manuscripts for journal resubmission. Prepare your revised manuscript with confidence.

Contact Us

References

  1. Aceituno, D., Muñoz, S., & Tirado, H. (2020). Systematic review of post-registration protocol deviations in PubMed-indexed reviews: Prevalence and associations with effect estimates. Research Synthesis Methods, 11(4), 523–541. https://doi.org/10.1002/jrsm.1418
  2. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124
  3. Kirkham, J. J., Dwan, K. M., Altman, D. G., et al. (2010). The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ, 340, c365. https://doi.org/10.1136/bmj.c365
  4. Page, M. J., Moher, D., Bossuyt, P. M., et al. (2021). PRISMA 2020 explanation and elaboration: Updated guidance and exemplars for reporting systematic reviews. BMJ, 372, n160. https://doi.org/10.1136/bmj.n160
  5. Porritt, K., Mallette, T., & Nott, M. (2021). Revisiting the Cochrane bias domain classification: A systematic examination of protocol deviations in published systematic reviews. Systematic Reviews, 10(1), 82. https://doi.org/10.1186/s13643-021-01629-1
  6. Sampson, M., Lefebvre, C., & Gusic, M. E. (2006). Searching for rehabilitation articles on  MEDLINE and CINAHL: An example with cross-over design. Archives of Physical Medicine and Rehabilitation, 87(8), 1089–1095. https://doi.org/10.1016/j.apmr.2006.05.009
  7. Schünemann, H. J., Higgins, J. P. T., Vist, G. E., et al. (2016). Completing “Summary of findings” tables and grading the certainty of the evidence. Journal of Clinical Epidemiology, 81, 45–55. https://doi.org/10.1016/j.jclinepi.2016.11.012
  8. Whiting, P., Savović, J., Higgins, J. P. T., et al. (2016). ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology, 69, 225–234. https://doi.org/10.1016/j.jclinepi.2015.06.005