Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data Analytics & AI · AI imaging reviews

Systematic Reviews of Artificial Intelligence in Medical Imaging: Design, Appraisal and Synthesis

Reviews of artificial intelligence in imaging are usually attempted as diagnostic accuracy reviews but usually cannot be conducted as such, because the included studies are model development papers rather than test accuracy studies. Determining which of the two is being reviewed, before the search rather than after it, governs the eligibility criteria, the appraisal tool, whether pooling is defensible and what the review can legitimately conclude.

Pubrica AcademyWritten by the medical writing and biostatistics teams Reviewed byPubrica biostatistics team Published9 October 2026 Last checked9 October 2026 Reading time13 minutes Sources33 peer-reviewed

The volume of published work on artificial intelligence in medical imaging makes a review appear easy to justify, yet such a review is difficult to conduct well. The difficulty lies not in the search but in the fact that the same search terms retrieve two very different kinds of study. One develops a model and reports its performance on the data from which it was built; the other evaluates a fixed model against a reference standard in a defined clinical population. Treating these as equivalent evidence, pooling their accuracy figures and reporting a summary sensitivity is the central error in this literature, and it is committed by reviews as often as by primary studies.

Learning objectives

  • Distinguish a model development review from a diagnostic test accuracy review and select one deliberately
  • Formulate eligibility criteria that keep the included set interpretable: task, reference standard, validation type and operating point
  • Recognise when overlapping public datasets make included studies non-independent
  • Extract the information a model paper should report, rather than the fields a trial extraction form anticipates
  • Appraise risk of bias with the appropriate instrument and report the version used
  • Determine on principled grounds whether to pool, and what to present when pooling is inappropriate

Common misconceptions

  • “Accuracy figures from model papers can be pooled into a summary sensitivity and specificity.” Internal validation performance is not test accuracy in a clinical population, and averaging it produces a figure with no clinical referent.
  • “Each included study is an independent piece of evidence.” Where many studies train and test on the same public datasets, their results are correlated by construction, and treating them as independent overstates both the precision and the consistency of the evidence.
  • “An area under the curve summarises performance adequately.” It conceals the operating point, which determines how the model would perform in clinical use. A review that extracts only the curve statistic cannot address clinical consequences.
  • “The reference standard is a technicality.” When the ground truth in one study is a radiologist’s read, in another histopathology and in a third a database label, the three studies are not measuring the same quantity against the same standard.
  • “Reporting to PRISMA is sufficient for an AI imaging review.” PRISMA governs the reporting of the review itself. Items specific to the included studies, such as data provenance, validation design and model availability, derive from AI-specific reporting guidance, and the version applied must be stated.

Two distinct review types under a single title

The first decision, and the one on which the validity of the review depends, concerns the kind of study being synthesised.

A model development review asks what has been built: which tasks have been attempted, on what data, with what architectures, and with what reported performance under what validation design. It is a mapping exercise. Its outputs are a structured description of a field, an assessment of methodological quality, and an account of what has and has not been tested. It should not report a pooled accuracy estimate, because the included studies did not measure accuracy in a population; they measured fit on held-out data from the same source.

A diagnostic test accuracy review asks how well a defined model, used as a test, classifies patients in a defined population against a reference standard. It requires studies that applied a fixed model prospectively or to an independent consecutive series, with a reference standard applied to all participants, and it supports pooled estimates of paired sensitivity and specificity. The literature contains far fewer studies of this kind than of the first, which is the principal reason that reviews drift from the second question to the first set of studies.

A third option is transparent and often appropriate: a scoping review, which maps the literature without claiming to estimate any quantity 17,18,30. If the question is “what has been done and how well was it reported”, the appropriate design is a scoping review, and reporting it as such is more credible than a systematic review with a meta-analysis that should not have been conducted. Our pages on meta-analysis of convolutional neural networks for radiological images and meta-analysis in cardiovascular imaging AI cover the analytic side of the second option.

Three reviews of the same literature, and what each can conclude

Review typeQuestionStudies it admitsAppraisal baseCan it pool?What it may conclude
Model development reviewWhat has been built, on what data, reported how well?Development and internal validation studiesPrediction-model risk-of-bias instrumentNo — the studies did not estimate accuracy in a populationA structured account of the field, its methods and its reporting quality
Diagnostic test accuracy reviewHow well does a defined model classify patients against a reference standard?Studies applying a fixed model to a defined clinical population with a reference standard in all participantsDiagnostic accuracy risk-of-bias instrumentPotentially — if thresholds, populations and reference standards are comparable and datasets do not overlapSummary paired sensitivity and specificity, with its population and threshold stated
Scoping reviewWhat exists, and what has not been examined?Any study type, mapped rather than appraised for effectOptional, and stated as optionalNo, and does not claim toA map of the literature and an explicit account of the gaps

Compiled from the design logic of each review type; the appraisal bases are the standard instrument families for diagnostic accuracy and prediction-model studies.

Framing the review question when the intervention is a model

PICO does not fit a model evaluation question, and imposing it produces the familiar error of recording the reference standard as a comparator 33. The element set appropriate to a test accuracy question comprises the population, the index test, the reference standard and the target condition, and for a model the index test requires more specification than a laboratory assay would.

The index test should be specified as follows: the task, stated as an input and an output; the model and, where relevant, its version; the input modality, acquisition parameters and preprocessing; whether the model operates alone or as an aid to a reader; and the operating threshold at which it was evaluated. The last item is the one most often omitted and the one that most alters the result. A model reported at a threshold chosen to maximise a summary statistic in the development data is not the same test as the same model at a fixed clinical threshold.

The comparator should be specified separately and deliberately. “Compared with radiologists” comprises three different questions: the model alone against readers alone, the model as a second reader, and readers assisted by the model. These have different designs, different outcomes and different clinical meanings, and a review that combines them has no interpretable result. See framing the review question for the general discipline, and the challenges of delivering clinical impact for why the reader-assistance question is the one that clinicians most often ask.

Eligibility criteria that preserve the interpretability of the included set

Five criteria are central, and each should be stated as an operational rule rather than a preference 3,6.

Task. Detection, classification, segmentation, triage, prognosis and image reconstruction are different tasks with different performance metrics. A review admitting several of them cannot summarise performance across them, so the review should either restrict the task or plan to report by task and state this explicitly.

Validation design. This criterion distinguishes the two review types. The protocol should specify whether external validation on data from a different source, institution or time period is required; whether temporally split internal validation qualifies; and whether cross-validation on a single dataset is eligible at all. Whatever is decided, the decision defines what the review can conclude.

Reference standard. The protocol should state which reference standards are acceptable and whether studies using a weaker standard are eligible, excluded, or included and analysed separately. Database labels, single-reader consensus, multi-reader consensus and histopathology are not interchangeable.

Reporting of an operating point. Reviewers should consider requiring that a study report sensitivity and specificity at a defined threshold, not only a summary curve statistic. This criterion deliberately excludes papers: a paper that reports only a curve cannot contribute to a statement about clinical behaviour.

Population and setting. Consecutive or random sampling from a defined clinical population, versus a curated case-enriched dataset, is the difference between an accuracy estimate and an upper bound on one.

An eligibility criterion for an AI imaging review

Admits a set that cannot be synthesised

We included studies evaluating deep learning algorithms for the diagnosis of disease on medical imaging, reporting diagnostic performance metrics such as accuracy, sensitivity, specificity or area under the curve.

“Deep learning algorithms” is not a test specification. It admits every architecture, every task and every output definition into one analysis.

No validation design requirement. Cross-validated performance on development data and external validation on an independent population will appear in the same forest plot, although they measure different quantities.

No reference standard requirement. Database labels, single-reader reads and histopathology all qualify, so the studies are not assessed against the same reference standard.

Metrics are listed as alternatives. Admitting a study that reports only a curve statistic means that the review cannot draw conclusions about behaviour at a threshold.

Standalone and reader-aid designs are not distinguished. These are different clinical questions, and mixing them yields an uninterpretable summary.

Nothing about dataset overlap. Ten studies on one public dataset will be counted as ten independent results, narrowing the interval around an estimate that contains no additional information.

Admits a set with one interpretation

We included studies evaluating a fully specified, fixed convolutional or transformer-based model for the detection of a single target condition on non-contrast computed tomography, in which the model was applied without further training to an evaluation set drawn from a different institution, scanner population or time period from the development data. The reference standard had to be histopathology or a documented multi-reader consensus applied to all participants in the evaluation set, and the study had to report sensitivity and specificity at a stated operating threshold, with the method of choosing that threshold described; studies reporting only a summary curve statistic were excluded, as were studies reporting only cross-validated performance on the development data. Where the model was used as a reader aid rather than standalone, the study was eligible only if the comparator arm was the same readers unaided, and these studies are reported separately from standalone evaluations. For every eligible study we recorded the identity and provenance of each dataset used; where two or more studies evaluated on the same public dataset, they were analysed as one dataset rather than as independent studies, and the overlap map is reported in full.

The index test is fully specified (task, modality, output, model class), so every included study estimated the same quantity.

External validation is a stated requirement, which converts the review from a map of development work into an accuracy review.

The acceptable reference standards are named and must be applied to all participants, so the comparison has a consistent definition of truth.

Requiring a stated operating point and its selection method excludes precisely those papers whose performance cannot be interpreted clinically.

Standalone and reader-aid evaluations are separated by design rather than reconciled afterwards in the discussion.

Dataset overlap is handled in advance as a unit-of-analysis rule, and the map is published; this map is the single most informative output that a review of this literature can produce.

The first criterion will retrieve many more studies and support no conclusion; the second will retrieve far fewer and support a specific one. A reviewer who encounters a small included set with a stated external-validation requirement interprets it as methodological rigour rather than as a weak search, and a transparent statement that few studies met the criteria is itself a finding.

Non-independence of studies that share datasets

A small number of public imaging datasets underlie a large share of published models in several imaging domains. When ten included studies train and evaluate on the same publicly available collection, they are not ten independent tests of a hypothesis; they are ten analyses of one sample, sharing its case mix, its label noise, its acquisition characteristics and its particular idiosyncrasies.

Two consequences follow. Pooling across such studies narrows the confidence interval around the summary estimate without adding information, so the review reports a precision that is not justified. Moreover, consistency across them, which resembles replication, is partly an artefact of the shared data, so apparent agreement between studies is weak evidence that the finding generalises.

In practice, the dataset identity should be extracted for every included study as a primary variable, the overlap mapped explicitly, and the map reported. Where overlap is substantial, the defensible options are to analyse by dataset rather than across studies, to select one study per dataset by a pre-specified rule, or to decline to pool and report the overlap as a specific rather than generic limitation. Whichever option is chosen, readers require the dataset map, because it is the most informative output a review of this literature can supply, yet very few reviews supply it.

The same logic applies to label provenance. Where several studies use labels derived from the same automated pipeline or the same institutional reporting system, the errors in those labels are shared, and the models’ apparent agreement partly reflects agreement with a common error.

Data extraction from model development studies

A trial extraction form does not fit a model development paper, and improvising fields during extraction leads to inconsistency 2,31. The extraction form should be organised into five blocks.

Data. Source and provenance of every dataset used, whether public or institutional, number of images and number of patients separately, class distribution, inclusion and exclusion of images, the method of splitting, and whether any patient appears in both training and evaluation data. The distinction between images and patients is important: thousands of images from a few hundred patients constitute a smaller sample than the image count suggests.

Reference standard. The definition of ground truth, who applied it, how disagreements between labellers were resolved, and whether the labellers had access to the information the model was given.

Model and training. Architecture family, whether pretrained weights were used and on what data, augmentation, the method of hyperparameter selection, and whether the evaluation data influenced any selection decision, which is the most common and least visible form of leakage.

Evaluation. Validation design, the metrics reported, the operating point and the method of choosing it, uncertainty estimates, and whether performance was reported by subgroup or by site.

Availability and reproducibility. Whether code, weights and data are available, and under what terms. This information is extractable, rarely extracted, and directly relevant to whether any finding in the review could be verified. Our page on what data to extract covers the general principles this form instantiates.

Extraction fields a model paper requires that a trial form does not have

BlockFieldsWhy it changes the interpretation
DataDataset identity and provenance; public or institutional; number of images AND number of patients; class distribution; split method; any patient appearing in both training and evaluation dataShared public datasets make studies non-independent; thousands of images from a few hundred patients constitute a small sample
Reference standardWhat defined ground truth; who applied it; how labeller disagreements were resolved; what information labellers hadDatabase labels, single-reader reads and histopathology are not equivalent reference standards, so the estimates are not comparable
Model and trainingArchitecture family; pretraining source; augmentation; how hyperparameters and thresholds were selected; whether evaluation data influenced any selectionThreshold or hyperparameter selection on evaluation data is leakage, and it inflates every reported metric
EvaluationValidation design (internal, temporal, external); metrics reported; operating point and how chosen; uncertainty; results by subgroup or siteWithout an operating point, no conclusion can be drawn about clinical behaviour; without subgroup results, none can be drawn about equity of performance
AvailabilityCode, trained weights and data availability, and the termsDetermines whether any claim in the paper is independently checkable

Compiled as a working extraction structure for model development and validation studies in imaging.

Selecting a risk-of-bias instrument appropriate to the study type

The instrument follows from the study type, and use of the wrong instrument produces an appraisal that appears rigorous but assesses nothing.

For studies evaluating a model as a diagnostic test, a structured diagnostic accuracy risk-of-bias instrument such as QUADAS-2 is the appropriate base, with its domains covering patient selection, the index test, the reference standard, and flow and timing. For studies developing or validating a prediction model, a prediction-model risk-of-bias instrument such as PROBAST is the appropriate base, with domains covering participants, predictors, outcome and analysis. AI-specific extensions and adaptations of both have been developed, and the field is evolving; reviewers are therefore obliged to name the instrument and the version used, with its date, and to report any domain that was modified and the reason.

Three failure modes are specific to this literature and merit explicit assessment whichever instrument is used 14,16. The first is leakage, in any of its forms: patient overlap between training and test data, preprocessing or feature selection performed before the split, or test data used to choose a threshold or a stopping point. The second is case-enriched evaluation, in which a model is tested on a dataset with a prevalence unlike that of any clinical population. The third is selective reporting of metrics or thresholds, which in this literature is easy and rarely detected.

Where a study does not report enough information to appraise a domain, the domain should be recorded as unclear, and reviewers should not infer the answer from the authors’ reputation or the journal. Our page on the challenges of deep learning in medical imaging covers the methodological failure modes in more detail.

The decision to pool and alternatives to pooling

Pooling requires that the included studies estimated the same quantity in comparable populations with comparable reference standards 3. In this literature that condition is frequently unmet, and the appropriate response is to state this rather than to pool with a heterogeneity caveat.

Four conditions should be checked before any meta-analysis. The studies evaluated the same task with the same output definition. They reported paired sensitivity and specificity at a defined threshold, not only a curve statistic. Their populations were sampled in comparable ways, with prevalences that are at least interpretable. Their evaluation datasets do not substantially overlap. If any of these conditions is not met, pooling produces a figure that is precise but meaningless.

Where pooling is inappropriate, the alternatives are not weak. A structured narrative organised by task and validation design, with a table of per-study estimates and their operating points, is more informative than a summary figure. A dataset-and-overlap map shows the data on which the field has actually been tested. A reporting-quality assessment, presented as a matrix of studies against items, shows what the literature does and does not permit anyone to conclude. An explicit account of what would need to exist for pooling to become possible is the most useful contribution a review can offer the field at this stage. The analytic methods themselves, where they apply, are covered on our meta-analysis service page and in meta-analysis methodology.

Reporting requirements beyond PRISMA

The review itself should be reported in accordance with the PRISMA 2020 statement, and the diagnostic accuracy extension should be used if the review is of that type 12. This covers how the review was conducted and reported, but it does not cover the characteristics of the included studies, which is where the informative content of an AI imaging review resides.

The following should be added as review-level reporting: a table of datasets with overlap mapped; the validation design of every included study, categorised consistently; the reference standard of every included study; the operating point for every reported estimate, or a statement that none was reported; code, weights and data availability for every included study; and an explicit statement of which of these items the included studies failed to report. The last item is the review’s most transferable finding: a field-level account of what is not being reported is useful to every subsequent reader, and only a reviewer is positioned to produce it.

The AI-specific reporting guidance against which included studies were assessed should be named, with its version and date, rather than referred to generically. The field’s reporting guidance is developing actively, and readers need to know which instrument and which version the assessment used. Reporting checklist review identifies gaps before submission, and living review formats merit consideration for a literature that evolves this rapidly.

Limits of these reviews for clinical decision-making

These limits should be stated explicitly, because the discussion section of an AI imaging review is where overclaiming typically occurs.

A review of model development studies cannot indicate to a clinician how a model would perform in that clinician’s department. The gap between held-out performance and local performance depends on scanner, protocol, population and prevalence, none of which the development study controlled for 14.

A review of accuracy studies, even a well-conducted one, cannot indicate to a clinician whether using the model improves outcomes. Accuracy is a property of a classification; benefit depends on what is done with the classification, on the alternative pathway, and on the harms of false positives and false negatives in that pathway. Evidence on benefit comes from studies of the model in use, of which this literature contains comparatively few 16.

Furthermore, no review can resolve what the included studies did not measure. Performance in subgroups that were not sampled, behaviour under distribution shift, degradation as the model ages against changing practice, and the effect of the model on the reader rather than on the classification are gaps in the evidence base rather than in the review, and they should be identified specifically. A review that concludes with a specific list of what should be studied next is more valuable than one that concludes with a pooled estimate that should not have been computed.

Key points

Key points

The first decision is whether the review addresses model development or diagnostic test accuracy, because almost every subsequent decision follows from it and the literature contains far more of the former than of the latter. The index test should be specified as a task, a model, an input, an output and an operating threshold, and it should be stated separately whether the question concerns the model alone, the model as a second reader, or readers assisted by the model. A validation design and a reference standard should be explicit eligibility requirements, and a stated operating point may also be required; this excludes papers reporting only a curve statistic, which is the intended effect. Dataset identity should be extracted for every study and the overlap map published, because studies sharing a public dataset are not independent evidence and pooling them increases apparent precision without adding information. Appraisal should use a diagnostic accuracy or prediction-model instrument as the study type requires, the version should be named, and leakage, case enrichment and selective metric reporting should be assessed explicitly. Where the conditions for pooling are not met, the review should state this and present a structured alternative rather than a summary estimate with a heterogeneity caveat.

Application to your study

Application to your study

  1. The review type should be recorded explicitly. A model development map, a diagnostic accuracy review or a scoping review should be specified in one sentence in the protocol, before the search.
  2. The index test should be specified in five parts. Task, model and version, input modality and preprocessing, output definition, operating threshold.
  3. The comparator question should be divided. The model alone, the model as a second reader and readers assisted by the model require separate analyses, stated in advance.
  4. Validation design should be an eligibility criterion. The protocol should state whether external validation is required and what qualifies as external.
  5. Dataset identity should be extracted first for every study. The overlap map should then be constructed and the unit of analysis decided before any pooling.
  6. An operating point should be required, or its absence recorded as a finding. A curve statistic alone cannot support a clinical statement.
  7. The appraisal instrument and its version should be named. Leakage, case enrichment and selective metric reporting should be assessed as explicit items regardless of the instrument used.
  8. The review should conclude with a specific research agenda. The agenda should state what would need to be studied, in which population and against which reference standard, for the question to become answerable.

Frequently asked questions

Occasionally, and far less often than it is done. Four conditions must hold. The studies must have evaluated the same task with the same output definition. They must report paired sensitivity and specificity at a defined operating threshold, not only a summary curve statistic. Their populations must have been sampled in comparable ways, with prevalences that are at least interpretable. Their evaluation datasets must not substantially overlap, because studies sharing a public dataset are analyses of one sample rather than independent tests. Where these conditions hold, standard diagnostic accuracy meta-analysis applies. Where they do not, pooling produces a precise figure with no clinical referent, and the preferable output is a structured narrative with per-study estimates, operating points and a dataset overlap map.

Internal validation evaluates a model on data held out from the same source as the data on which it was developed, usually by a random or cross-validated split. External validation applies a fixed model, without further training, to data from a different institution, scanner population or time period. The distinction is important because internal validation performance is a property of the model-and-dataset pair, not an estimate of how the model would classify patients elsewhere, and the gap between the two is often large and unpredictable. For a review, this is an eligibility decision rather than a detail: requiring external validation converts a map of development activity into evidence that can support an accuracy claim, and admitting both without distinction produces an analysis that combines two different quantities.

Dataset identity should be treated as a primary extracted variable, the overlap mapped across all included studies, and the map published. Studies evaluating on a shared dataset are correlated by construction: they inherit its case mix, its label noise and its acquisition characteristics, so their apparent agreement is partly an artefact, and pooling them narrows the confidence interval without adding information. The defensible options are to analyse by dataset rather than by study, to select one study per dataset using a rule specified in advance, or to decline to pool and report the overlap as a specific limitation. All three are preferable to treating ten analyses of one dataset as ten independent results, and the map itself is often the most useful output of the review.

The instrument should be matched to the study type. For studies evaluating a model as a diagnostic test in a clinical population, a diagnostic accuracy instrument such as QUADAS-2 is the appropriate base, with domains for patient selection, the index test, the reference standard, and flow and timing. For model development and validation studies, a prediction-model instrument such as PROBAST is the appropriate base, covering participants, predictors, outcome and analysis. AI-specific extensions and adaptations of both exist and the field is developing actively, so the instrument, the version and the date of use should be named, and any modified domain should be reported with the reason. Whichever instrument is chosen, leakage, case-enriched evaluation and selective reporting of metrics or thresholds should be assessed explicitly, because these are the dominant failure modes in this literature.

Because the operating point determines how the model behaves for an individual patient. A summary curve statistic describes performance across all thresholds simultaneously and corresponds to no deployable configuration; sensitivity and specificity at a stated threshold describe what the model would actually do. Two further problems follow from omitting it. A threshold chosen to maximise performance on the evaluation data is a form of leakage, and a paper that does not state how the threshold was chosen provides no means of detecting it. Without a threshold, moreover, false positive and false negative counts cannot be estimated for any realistic prevalence, so no conclusion can be drawn about the clinical consequences of using the model. A review that extracts only curve statistics has collected a field’s worth of figures that cannot be converted into a clinical statement.

PRISMA governs the reporting of the review; it does not require the study-level characteristics that make this literature interpretable. The following should be added at review level: a dataset table with overlap mapped; the validation design of every included study, categorised consistently; the reference standard of every included study; the operating point for every reported estimate or an explicit note that none was given; code, weights and data availability for each study; and a summary of which of these items the included studies failed to report. The final item is the most transferable finding a review of this field can produce, because a systematic account of what is not being reported is useful to every subsequent reader and only a reviewer is positioned to compile it.

Yes, and it is often the more useful result. Methods reviewers interpret a small included set under a stated external-validation and reference-standard requirement as methodological rigour rather than as a weak search, provided the search itself is fully reported. The finding that a large literature exists but only a small number of studies evaluated a fixed model against an adequate reference standard in an independent population is a substantive statement about the maturity of the field. Such a review is undermined not by the small number but by a criterion loosened after the yield was seen; the criteria should therefore be declared in a registered protocol, and any amendment reported with its date and with a statement of whether results were visible at the time.

Only within narrow limits, and those limits should be stated clearly. Accuracy is a property of a classification; benefit depends on what is done with the classification, what the alternative pathway was, and the relative harms of false positives and false negatives in that pathway. Evidence about benefit comes from studies of the model in use (comparative studies with patient-relevant outcomes or, at minimum, studies of reader behaviour with and without the model), and this literature contains comparatively few of them. A review of development or accuracy studies can legitimately report what the models classify and how well, can report the absence of in-use evidence as a finding, and should not translate a summary sensitivity into a claim about patient outcomes.

References

  1. Chapter 4: Searching for and selecting studies. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
  2. Chapter 5: Collecting data. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
  3. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
  4. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews. 2021;10(1):1-19. read 9 October 2026. doaj.org
  5. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews. 2015;4:78. read 9 October 2026. cochrane.org
  6. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1):1. doi:10.1186/2046-4053-4-1 read 9 October 2026. doi.org/10.1186/2046-4053-4-1
  7. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews. 2014;3(1):74. doi:10.1186/2046-4053-3-74 read 9 October 2026. doi.org/10.1186/2046-4053-3-74
  8. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews. 2011;2011(8):MR000026. doi:10.1002/14651858.MR000026.pub2 read 9 October 2026. doi.org/10.1002/14651858.MR000026.pub2
  9. Bolanos F, Salatino A, Osborne F, Motta E. Artificial intelligence for literature reviews: opportunities and challenges. Artificial Intelligence Review. 2024;57(10):259. doi:10.1007/s10462-024-10902-3 read 9 October 2026. doi.org/10.1007/s10462-024-10902-3
  10. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, et al. Large language models encode clinical knowledge. Nature. 2023;620(7972):172-180. doi:10.1038/s41586-023-06291-2 read 9 October 2026. doi.org/10.1038/s41586-023-06291-2
  11. Karim MR, Beyan O, Zappa A, Costa IG, Rebholz-Schuhmann D, Cochez M, Decker S. Deep learning-based clustering approaches for bioinformatics. Briefings in Bioinformatics. 2021;22(1):393-415. doi:10.1093/bib/bbz170 read 9 October 2026. doi.org/10.1093/bib/bbz170
  12. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71 read 9 October 2026. doi.org/10.1136/bmj.n71
  13. Murphy A, Ekpo E, Steffens T, Neep MJ. Radiographic image interpretation by Australian radiographers: a systematic review. Journal of Medical Radiation Sciences. 2019;66(4):269-283. doi:10.1002/jmrs.356 read 9 October 2026. doi.org/10.1002/jmrs.356
  14. Zhang Z, Sejdic E. Radiological images and machine learning: Trends, perspectives, and prospects. Computers in Biology and Medicine. 2019;108:354-370. doi:10.1016/j.compbiomed.2019.02.017 read 9 October 2026. doi.org/10.1016/j.compbiomed.2019.02.017
  15. Bramer WM, de Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association. 2018;106(4):531-541. doi:10.5195/jmla.2018.283 read 9 October 2026. doi.org/10.5195/jmla.2018.283
  16. Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, Brink J. Artificial Intelligence and Machine Learning in Radiology: Opportunities, Challenges, Pitfalls, and Criteria for Success. Journal of the American College of Radiology. 2018;15(3 Pt B):504-508. doi:10.1016/j.jacr.2017.12.026 read 9 October 2026. doi.org/10.1016/j.jacr.2017.12.026
  17. Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine. 2018;169(7):467-473. doi:10.7326/M18-0850 read 9 October 2026. doi.org/10.7326/M18-0850
  18. Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology. 2018;18(1):143. doi:10.1186/s12874-018-0611-x read 9 October 2026. doi.org/10.1186/s12874-018-0611-x
  19. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association. 2016;104(3):240-243. doi:10.3163/1536-5050.104.3.014 read 9 October 2026. doi.org/10.3163/1536-5050.104.3.014
  20. Angermueller C, Parnamaa T, Parts L, Stegle O. Deep learning for computational biology. Molecular Systems Biology. 2016;12(7):878. doi:10.15252/msb.20156651 read 9 October 2026. doi.org/10.15252/msb.20156651
  21. Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. Journal of Clinical Epidemiology. 2016;74:119-123. read 9 October 2026. cochrane.org
  22. Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. International Journal of Epidemiology. 2016;45:266-277. read 9 October 2026. cochrane.org
  23. Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association. 2016;23:193-201. read 9 October 2026. cochrane.org
  24. Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, Gruen RL. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLOS Medicine. 2014;11(2):e1001603. doi:10.1371/journal.pmed.1001603 read 9 October 2026. doi.org/10.1371/journal.pmed.1001603
  25. James AP, Dasarathy BV. Medical Image Fusion: A survey of state of the art. Information Fusion. 2014;19:4-19. doi:10.1016/j.inffus.2013.12.002 read 9 October 2026. doi.org/10.1016/j.inffus.2013.12.002
  26. Gross A, Schirm S, Scholz M. Ycasd - a tool for capturing and scaling data from graphical representations. BMC Bioinformatics. 2014;15(1):219. doi:10.1186/1471-2105-15-219 read 9 October 2026. doi.org/10.1186/1471-2105-15-219
  27. Wang S, Summers RM. Machine learning and radiology. Medical Image Analysis. 2012;16(5):933-951. doi:10.1016/j.media.2012.02.005 read 9 October 2026. doi.org/10.1016/j.media.2012.02.005
  28. Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information and Libraries Journal. 2010;27(2):114-122. doi:10.1111/j.1471-1842.2009.00863.x read 9 October 2026. doi.org/10.1111/j.1471-1842.2009.00863.x
  29. Roobottom CA, Mitchell G, Morgan-Hughes G. Radiation-reduction strategies in cardiac computed tomographic angiography. Clinical Radiology. 2010;65(11):859-867. doi:10.1016/j.crad.2010.04.021 read 9 October 2026. doi.org/10.1016/j.crad.2010.04.021
  30. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal. 2009;26(2):91-108. doi:10.1111/j.1471-1842.2009.00848.x read 9 October 2026. doi.org/10.1111/j.1471-1842.2009.00848.x
  31. Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology. 2006;59:697-703. read 9 October 2026. cochrane.org
  32. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology. 2005;58:741-742. read 9 October 2026. cochrane.org
  33. Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club. 1995;123(3):A12-A13. doi:10.7326/ACPJC-1995-123-3-A12 read 9 October 2026. doi.org/10.7326/ACPJC-1995-123-3-A12

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 9). Systematic Reviews of Artificial Intelligence in Medical Imaging: Design, Appraisal and Synthesis. Pubrica. https://pubrica.com/academy/latest-topics/a-systematic-review-of-artificial-intelligence-in-imaging/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Protocol and extraction stub for an AI imaging review

The top block should be completed before the search, because it determines everything below it. The dataset overlap map is the component most reviews omit and the one most readers would use.

AI IMAGING REVIEW — PROTOCOL AND EXTRACTION STUB

=== DECIDE FIRST (one box only) ===
  [ ] MODEL DEVELOPMENT MAP  — no pooled accuracy will be reported
  [ ] DIAGNOSTIC TEST ACCURACY REVIEW — external validation required
  [ ] SCOPING REVIEW — mapping only, no effect claim
  Reason for this choice: _______________________________

=== INDEX TEST SPECIFICATION ===
  Task (input -> output): ________________________________
  Model class / named model and version: ________________
  Input modality and acquisition constraints: ___________
  Preprocessing required or permitted: __________________
  Output definition (binary / multiclass / mask / score): ______
  Operating threshold: [ ] must be stated by the study
                       [ ] not required — justify: __________

=== COMPARATOR QUESTION (choose; do not mix in one analysis) ===
  [ ] model standalone vs reference standard
  [ ] model standalone vs readers unaided
  [ ] readers aided by model vs the same readers unaided
  [ ] other: _____________________________________________

=== ELIGIBILITY RULES ===
  Validation design required:
    [ ] external (different institution / scanner population / time period)
    [ ] temporal split accepted  [ ] internal cross-validation accepted
  Acceptable reference standards:
    [ ] histopathology  [ ] multi-reader consensus, documented
    [ ] single reader   [ ] database / registry label
    Applied to ALL participants?  [ ] required
  Sampling: [ ] consecutive or random from a defined population required
            [ ] case-enriched datasets eligible, analysed separately
  Minimum reporting: [ ] sensitivity and specificity at a stated threshold
  Excluded: ______________________________________________

=== EXTRACTION, PER STUDY ===
DATA
  Dataset name(s) and provenance: _____________________
  Public or institutional: ______  Access terms: ______
  Images: ______   PATIENTS: ______   (record both)
  Class distribution / prevalence in evaluation set: ______
  Split method: ________________________________________
  Any patient in both training and evaluation data?  [ ] no [ ] yes [ ] cannot tell
REFERENCE STANDARD
  What defined ground truth: ___________________________
  Applied by whom: _____________  Disagreements resolved how: ______
  Information available to labellers: __________________
MODEL AND TRAINING
  Architecture family: _________________________________
  Pretrained on: _______________________________________
  Hyperparameter selection method: _____________________
  Threshold selection method: __________________________
  Did evaluation data influence ANY selection decision?  [ ] no [ ] yes [ ] cannot tell
EVALUATION
  Validation design: [ ] internal [ ] temporal [ ] external
  Metrics reported: ____________________________________
  Operating point: __________  Chosen how: ______________
  Uncertainty reported?  [ ] yes, how: ______  [ ] no
  Results by subgroup or site?  [ ] yes  [ ] no
AVAILABILITY
  Code: [ ] yes [ ] no    Weights: [ ] yes [ ] no
  Data: [ ] yes [ ] no    Terms: ____________________
APPRAISAL
  Instrument and VERSION and DATE: _____________________
  Domains modified and why: ____________________________
  Leakage assessed?  [ ] yes  Finding: ______________
  Case enrichment assessed?  [ ] yes  Finding: ______________
  Selective metric/threshold reporting?  [ ] yes  Finding: ______

=== DATASET OVERLAP MAP (publish this) ===
  Dataset | Studies evaluating on it | Unit of analysis decision
  ________|_________________________|__________________________
  ________|_________________________|__________________________
  Pre-specified rule where studies share a dataset: ____________

=== POOLING DECISION (answer all four before any meta-analysis) ===
  Same task and output definition across studies?        [ ] yes [ ] no
  Paired sensitivity/specificity at stated thresholds?   [ ] yes [ ] no
  Comparable sampling and interpretable prevalence?      [ ] yes [ ] no
  Evaluation datasets non-overlapping?                   [ ] yes [ ] no
  Any "no" -> DO NOT POOL. Produce instead:
    [ ] per-study table with operating points
    [ ] dataset and overlap map
    [ ] reporting-quality matrix (studies x items)
    [ ] explicit statement of what would make pooling possible

=== WHAT THIS REVIEW MAY NOT CONCLUDE ===
  [ ] No claim about local performance from internally validated studies
  [ ] No claim about patient outcomes from accuracy estimates
  [ ] No claim about subgroups the studies did not sample
  [ ] Gaps named specifically, as a research agenda

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica works with authors to establish which of the three review types the question represents, formulates eligibility criteria that make the included set amenable to synthesis, and constructs the dataset overlap map that determines whether pooling is defensible. Where it is not, Pubrica produces the structured alternative, comprising per-study estimates with operating points, a reporting-quality matrix and a specific research agenda, rather than a summary estimate accompanied by a heterogeneity caveat.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Pubrica assesses an AI imaging review question to determine which type of review it can support and what its eligibility criteria should specify.

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.