NICE Technology Appraisal: What the Health Technology Evaluations Manual Requires from the Systematic Review Behind a Submission

Osipenko and colleagues scored the evidence in 409 NICE technology appraisals published from 2000 to 2019. Overall data quality was rated poor in 55%, unacceptable in 10%, acceptable in 34% and good in 1%. They also flagged unclear methods for systematic reviews and indirect comparisons, which 68% of single technology appraisals used [1].

In a company submission, a review that cannot be traced to a protocol may be discounted, and closing the evidence gaps later can mean extra analysis or a resubmission. An early internal audit is a cost-saving step, not an academic exercise.

The manual sets out how NICE judges clinical effectiveness and value when it recommends technologies for NHS use, the gateway to NHS reimbursement. The NICE health technology evaluations manual was first published on 31 January 2022 and has been updated since [2]. Angelis and colleagues traced the 2022 revision through consultation: it reinforced a comprehensive evidence base, clarified when non-randomised studies can be used, and accepted more uncertainty where evidence is hard to generate, such as in children, rare diseases and innovative technologies [3].

Section numbers can change between versions. Open the manual’s update information and confirm the numbers cited here before drafting.

Section 3.3.4 requires a systematic review using a pre-defined protocol, with explicit, valid and replicable methods; section 3.4.4 says to follow a previously prepared protocol [2]. Fix the question, population, comparators, outcomes, databases, search dates, eligibility criteria and appraisal methods before screening. A protocol written after the results are known cannot meet that standard, and its dates usually show it.

The protocol also protects the company: an unfavourable result cannot be blamed on shifting eligibility rules. Our note on PRISMA reporting shows how to document it.

The statement that accompanies the review should answer four questions before the evaluator asks: which sources were searched, when, by whom, and how disagreements were resolved. The manual expects companies to identify all relevant evidence, including studies known to them, and section 3.4.5 requires at least two reviewers to assess retrieved records [2]. Record these details at the time of the search, not afterwards.

PRISMA 2020, a 27-item checklist covering identification, screening, appraisal and synthesis, gives the reporting structure [4].

Section 3.4.6 requires every included study to be critically appraised; section 3.4.7 requires effect modifiers such as patient characteristics, setting and study year to be identified before analysis, which limits selective subgroup reporting. Meta-analysis is appropriate only with enough relevant, valid data (3.4.8), and pooled results need a heterogeneity assessment and sensitivity analyses (3.4.10) [2]. The Cochrane Handbook sets out established risk of bias methods [5].

The route decides who checks the review [2].

In a multiple technology appraisal the EAG synthesises published evidence itself, so the company review should be detailed enough for an independent reviewer to follow its selection. Confirm the route early, then scope the search to the evidence that route will use.

Treat the evidence review as a permanent part of the HTA submission. Keep the protocol version, dated search logs, screening decisions, extraction forms and final report in one archive, indexed against the PRISMA 2020 checklist [4]. When the EAG challenges the search or appraisal, point to the protocol, search string and screening log rather than a new argument. Osipenko and colleagues acknowledged their own quality scoring was subjective [1], a reminder that many criticisms are judgements the record can answer. Where it cannot, say so and add a sensitivity analysis.