Contents
On this page
- Five domains without a shared evidence standard
- Framing a review question when the object of study is a method
- Baselines and time horizons
- Searching a literature that extends beyond journals
- Data extraction from machine learning studies
- Appraisal: data leakage, baselines and optimism
- Pooling and preferable alternatives
- Reporting the review and the evidence gaps identified
- Limits of what this literature can currently support
- Key points
- Application to your study
- Frequently asked questions
Academy · Research Data Analytics & AI · ML in pharma reviews
Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis
“Machine learning in the pharmaceutical industry” is not a single subject and cannot be addressed by a single review. It spans target identification, molecular property prediction, trial operations, manufacturing control and pharmacovigilance, domains with different data, different success criteria and different evidence standards. A review that does not select one domain, and does not state the comparator against which the models are evaluated, produces a catalogue rather than a synthesis.
The literature on machine learning in drug research grows faster than any review can cover, and reports its results in a form that makes synthesis unusually difficult: retrospective performance on curated datasets, with success criteria chosen by the authors. The useful question for a reviewer is rarely “how well do the models perform”, because the reported figures are not commensurable. It is rather what has been attempted, against which baseline, with what validation and with what prospective confirmation; the last of these is where the literature is weakest.
Learning objectives
- Scope the review to one domain of pharmaceutical machine learning rather than the whole field
- Frame a question where the object under study is a method rather than a treatment
- Require a baseline: the comparator against which the model was evaluated, and over what time horizon
- Search a literature that appears partly in preprints, patents and conference proceedings
- Extract the fields that distinguish a validated model from a well-fitted one
- Determine whether any part of the included set can be pooled, and what to present when it cannot
Common misconceptions
- “Reported performance metrics across studies can be compared.” Different datasets, splits, endpoints and author-selected metrics render the numbers incommensurable. A table of such figures has the appearance of evidence without its substance.
- “A model that outperforms a baseline in a retrospective benchmark is validated.” Retrospective benchmark performance is a hypothesis about prospective performance. The review should report which studies tested that hypothesis; most have not.
- “Drug discovery success can be attributed to a method.” Attribution over a development timeline this long is extremely difficult. Analyses of where new drugs come from span decades for a reason — one well-known classification of natural products as sources of new drugs covers approvals from 1981 to 2014, a 34-year window.
- “Published code and data make a study reproducible.” They make it potentially reproducible. Whether the training data are available, whether the split can be regenerated and whether the evaluation set was ever used for selection are separate questions, and the extraction form should address all three.
- “Searching MEDLINE and Embase covers this field.” Substantial parts of the field are reported first in preprints and conference proceedings, and commercially significant work appears in patents or not at all. A search restricted to bibliographic databases has a structural gap that should be acknowledged.
Five domains without a shared evidence standard
The phrase covers at least five distinct activities, and the first task of the protocol is to select one.
Discovery and target work: target identification and prioritisation, virtual screening, molecular property and activity prediction, generative design 7,8. Evaluated against retrospective benchmarks, occasionally against prospective synthesis and assay.
Preclinical and translational: toxicity prediction, pharmacokinetic modelling, biomarker discovery, image-based phenotyping 7,29. Evaluated against experimental ground truth that is itself variable.
Clinical development: site and patient selection, eligibility screening, dropout prediction, endpoint adjudication, synthetic control arms. Evaluated against trial operational outcomes, and occasionally regulated.
Manufacturing and quality: process control, deviation prediction, batch release support. Evaluated against process outcomes in a regulated environment with its own validation expectations.
Post-market: signal detection in pharmacovigilance, literature surveillance, real-world evidence generation. Evaluated against reference signal sets, with a well-known difficulty in defining truth.
These domains cannot share one eligibility criterion, one appraisal instrument or one outcome definition. A review spanning them is a map, and should be labelled as a scoping review 13,15. A review intended to synthesise evidence must select one domain and one task within it. The pages on AI use cases in clinical development, machine learning in computational pathology and literature surveillance in pharmacovigilance each cover one of these domains in its own terms.
Five domains, and why they cannot share one review
| Domain | Typical task | What counts as ground truth | What the strongest available evidence looks like |
|---|---|---|---|
| Discovery and target work | Virtual screening, property and activity prediction, generative design, target prioritisation | Assay results, where they exist; otherwise a curated benchmark label | Prospective synthesis and assay of predicted compounds, reported with the chronology |
| Preclinical and translational | Toxicity and pharmacokinetic prediction, biomarker discovery, image-based phenotyping | Experimental measurement, itself variable between labs and protocols | External validation on data from a different laboratory or platform |
| Clinical development | Site and patient selection, dropout prediction, eligibility screening, endpoint adjudication | Trial operational records and clinical outcomes | Prospective use in a trial with the operational outcome reported |
| Manufacturing and quality | Process control, deviation prediction, batch release support | Process and quality outcomes in a regulated environment | Validated deployment with documented performance over time |
| Post-market | Pharmacovigilance signal detection, literature surveillance, real-world evidence | Reference signal sets, with a known difficulty in defining truth | Evaluation against an independently constructed reference set |
Compiled as a scoping structure; the domains differ in data, ground truth and evidence standard, which is why one eligibility criterion cannot serve all five.
Framing a review question when the object of study is a method
When the object under study is a computational method rather than a treatment, the intervention-review frame does not apply, and the elements that do are the task, the input data, the baseline, the validation design and the criterion of success 32.
The task must be stated as an input and an output: predict a measured property for a compound given a structure, rank candidates for a target given assay data, classify a trial site by enrolment risk given historical records. “Applying machine learning to drug discovery” is not a task.
The baseline is the element most reviews omit and the one that determines whether any included study is interpretable. A model with no stated comparator makes no claim. Legitimate baselines include an established physics-based or statistical method, a simpler model of the same type, a human expert workflow, or a documented historical rate. If a study has none, this should be recorded — the proportion of studies without a baseline is a review-level finding.
The validation design ranges from a random split of a single dataset, through scaffold or temporal splits, to evaluation on data generated after the model was built, to prospective experimental confirmation. These are not degrees of the same property: a random split and a prospective assay answer different questions, and a review that combines them yields no interpretable result.
The criterion of success must be declared because authors select it. A model that improves a ranking metric and a model that yields a confirmed active compound are both described as “successful”. The review should specify which of these it addresses. See framing review questions for the reasons why specifying a different element set is preferable to forcing the PICO (population, intervention, comparator, outcome) framework.
Baselines and time horizons
Claims that machine learning is changing pharmaceutical research are claims about a counterfactual over a long horizon, and the horizon is the central difficulty. A compound entering discovery today reaches approval, if at all, after a timeline measured in years, which means that attributing an approval to a method used at the outset is a difficult inferential problem rather than a reporting one.
The scale of that difficulty is visible in how the field studies drug origins at all: a well-known classification of natural products as sources of new drugs covers approvals from 1981 to 2014, a 34-year window 19. Analyses of where new medicines come from are built on multi-decade windows because that is the timescale on which anything can be said about sources of new drugs 19,24. A review of machine learning in discovery is therefore examining a literature whose outcome of interest largely has not had time to be observed.
Three consequences follow for the review. The outcome under review should be stated precisely — retrospective predictive performance, prospective experimental confirmation, candidates entering development, or approvals — and the discussion should not shift between them. The strongest evidence is likely to be at the retrospective end and the claims at the approval end, and the review should state this. Where a study claims a discovery outcome, the chronology should be extracted: when the model was built, when the prediction was made, when the experimental confirmation occurred, and whether the confirmation was independent of those who made the prediction.
The time horizon on which questions about the sources of new drugs are examined, from the classification of natural products as sources of new drugs covering approvals from 1981 to 2014. It is given here to indicate the timescale over which discovery outcomes become observable, not as a performance figure for any method.
Searching a literature that extends beyond journals
Four source types, and a bibliographic-only search misses two of them 1.
Bibliographic databases cover the peer-reviewed journal literature and should be searched database-by-database in the usual way — see sources of review data. The terminology problem is acute here: the same work is described as machine learning, deep learning, artificial intelligence, data-driven modelling, cheminformatics, quantitative structure-activity modelling or simply as a named architecture, and indexing lags behind all of it 12.
Preprint servers and conference proceedings carry a substantial share of methods work, often months or years before journal publication and sometimes instead of it. The protocol should specify whether preprints are eligible and, if so, how the later journal version and any changes between the two will be handled 4. The decision should be reported in either case.
Patents and regulatory filings are where commercially significant work surfaces, if it surfaces 23. Methodological detail can rarely be extracted from them, and a review that omits them should state that commercially developed methods are structurally underrepresented in its evidence base. Such a statement is more informative than an unqualified claim about the state of the field.
Code and model repositories are primary sources about the artefacts themselves: whether code exists, whether weights are released, and what the training data were. Searching them is unusual in a review but is worthwhile when reproducibility is part of the question.
Where a field’s evidence is partly unpublished, a transparent review states the gap specifically rather than describing its search as comprehensive.
A search strategy for machine learning in pharmaceutical research
Retrieves the wrong set and misses the field
("machine learning" OR "artificial intelligence") AND ("drug discovery" OR "pharmaceutical") — searched in PubMed and Google Scholar, 2015 to present.
Two method terms only. The same work is published as deep learning, QSAR, cheminformatics or a named architecture, and all of that is outside this string.
No controlled vocabulary. Free-text-only searching in a database with mature indexing forgoes the most reliable component of retrieval.
“Drug discovery” as the task concept. It retrieves the entire pharmaceutical literature and still misses papers that describe their task specifically.
A date limit of 2015. It excludes the modelling traditions the field was built on, and it makes any claim about recent growth circular.
Google Scholar as a database. It cannot be searched with a documented, stable strategy, so it cannot be reported in the form a systematic review requires.
No preprint or proceedings coverage. A substantial share of methods work appears there first, and sometimes exclusively, so the search has a structural gap that is never declared.
No statement of what is unreachable. Commercially developed methods are invisible to this search and the review will read as a description of the whole field.
Covers the terminology and names its gaps
Searched in MEDLINE via Ovid, Embase via Ovid and Web of Science Core Collection, each on a stated date with the full line-by-line strategy in the supplement. The method concept combined controlled vocabulary terms for machine learning, artificial intelligence and neural networks with free-text terms for the architecture families and modelling traditions the same work is published under, including deep learning, neural network, random forest, gradient boosting, support vector, graph neural network, transformer, variational autoencoder, quantitative structure-activity relationship, cheminformatics and data-driven modelling, each truncated appropriately for the platform. The task concept was restricted to the single domain under review and expressed as its own set of terms — for a molecular property prediction review, terms for virtual screening, activity prediction, property prediction, hit identification and lead optimisation — rather than the phrase drug discovery, which retrieves the whole of pharmaceutical research. No date limit was applied, because restricting to recent years excludes the modelling traditions the current literature builds on and makes any temporal claim circular. Separately, arXiv, bioRxiv and ChemRxiv were searched on a stated date with a simplified strategy recorded in the supplement, and the proceedings of the named machine learning conferences were searched for the same period; the protocol states that where a preprint and a later journal version both exist the journal version is used and any change in the reported result between the two is recorded. We did not search patent databases, and we state in the limitations that methods developed commercially are therefore structurally underrepresented in this evidence base. Code repositories were searched only to establish artefact availability for already-included studies, not to identify studies, and this is reported as a supplementary method.
The method concept spans the vocabularies under which the same work is published, including the modelling traditions that predate the current terminology.
Controlled vocabulary and free text are combined per platform, which is what makes retrieval both sensitive and reportable.
The task concept is restricted to one domain and expressed in task terms, so the retrieved set is the review’s subject rather than the whole field.
No date limit is applied, and the reason is stated, which removes a circularity from any claim the review makes about change over time.
Preprints and proceedings are searched as named sources with their own recorded strategy, and the preprint-versus-journal handling rule is pre-specified.
The patent gap is declared explicitly, so the review’s claims are about the published evidence base rather than about the field.
Repository searching is labelled as a supplementary method for a specific purpose, so it is not mistaken for part of the identification strategy.
The first string will retrieve tens of thousands of records, most of which are irrelevant, while the most relevant are missing — the least favourable combination. The second is longer, reportable line by line, and states the two gaps — patents and unpublished work — that determine what the review may claim about the field as a whole.
Data extraction from machine learning studies
A model paper does not fit a clinical extraction form 2. Six blocks cover the information a reader requires.
Task and data. Input representation, output definition, data source and whether it is public, sample size in the relevant unit — compounds, assays, patients, sites — label provenance and label noise, and class balance.
Split and leakage controls. How the data were split, whether the split was random, scaffold-based, temporal or by source; whether any preprocessing, feature selection or hyperparameter choice used the evaluation data; and whether near-duplicate records appear on both sides of the split, which in chemical datasets is a standard and under-reported failure.
Model and training. Model family, feature representation, how hyperparameters were chosen, and whether the comparison baselines were tuned with the same effort as the proposed model — an untuned baseline is not a valid baseline.
Evaluation. Metrics, with the primary one identified; uncertainty; performance against the baseline rather than in isolation; and performance in the relevant subsets, such as chemical series or patient subgroups.
Prospective component. Whether any prediction was subsequently tested, how many predictions were tested, who tested them, and what the outcome was. Absence should be recorded explicitly; this field is empty for most of the literature, and the proportion is itself a finding.
Availability. Code, trained model, data, and the terms of access. The general extraction approach is described on the page on what data to extract.
Appraisal: data leakage, baselines and optimism
There is no single accepted risk-of-bias instrument for machine learning methods papers across these domains. Where the model is a clinical prediction model, a prediction-model risk-of-bias instrument applies and should be named with its version. Where it is not — for example, a virtual screening pipeline or a process control model — a transparent item list should be used and results reported item by item rather than as an invented score.
Five items merit inclusion in any such list. Leakage, in all its forms: evaluation data used in selection, duplicate or near-duplicate records across the split, temporal leakage where future information is available to a model predicting the past. Baseline adequacy: whether a comparator existed and whether it was given the same tuning effort. Metric selection: whether the primary metric was declared in advance or appears to have been chosen after the results. Dataset provenance: whether the benchmark is one on which many models have been developed and tuned, in which case reported performance on it carries community-wide optimism. And prospective confirmation: present or absent.
These should be reported as a matrix of studies against items. This matrix is usually the most informative output of a review in this area, because it indicates what the literature permits readers to conclude — which is often less than the literature claims. The page on regulatory affairs and causal inference in machine learning covers the appraisal questions that arise when a model enters a regulated process.
Appraisal items for a machine learning study, reported individually and never as a score
| Item | What to look for | Why it matters |
|---|---|---|
| Leakage | Evaluation data used in preprocessing, feature selection, hyperparameter choice or threshold setting; near-duplicate records across the split; future information available to a model predicting the past | Every form inflates reported performance, and none is visible in the headline metric |
| Baseline adequacy | Whether a comparator existed at all, what it was, and whether it received the same tuning effort as the proposed model | An untuned baseline produces an artificial improvement; the absence of a baseline means that no claim is made |
| Split strategy | Random, scaffold, temporal, by source, or evaluation on data generated after the model was built | A random split answers a different question from a temporal or source-held-out split |
| Metric selection | Whether a primary metric was declared in advance, and whether reported metrics favour the proposed model | Post hoc metric choice is the least costly and least detectable form of selective reporting |
| Benchmark provenance | Whether the dataset is one on which many models have been developed and tuned | Shared benchmarks accumulate community-wide optimism that no single study can detect |
| Prospective confirmation | Whether any prediction was tested afterwards, how many, by whom, and with what result | This is the scarcest evidence in the field and the only direct test of the retrospective hypothesis |
| Availability | Code, trained model, training data, and the terms of access | Determines whether any claim in the paper can be independently checked |
Compiled as a transparent item list. Where the model is a clinical prediction model, a published prediction-model risk-of-bias instrument applies instead and should be named with its version.
Pooling and preferable alternatives
Meta-analysis of model performance across studies is almost never defensible in this field 2. The estimates derive from different datasets of differing difficulty, different endpoints, different split strategies and different metrics; an average of them estimates no defined quantity. Where a set of studies evaluated different models on the same public benchmark with the same split, a structured comparison is possible — but the unit of evidence is then the benchmark rather than the study, and the shared-benchmark optimism applies to all of them equally.
The following alternatives are listed in descending order of usefulness. A task-by-task structured synthesis, with one table per task giving each study’s data, split, baseline, primary metric and prospective status. A baseline-relative summary: how many studies beat a stated baseline, how many had none, and how many were compared only against an untuned alternative. A prospective-evidence count: how many included studies subsequently tested a prediction, and with what result. Finally, an evidence gap map of tasks against validation levels, which shows where the field has conducted only retrospective work. See evidence gap mapping and, where pooling genuinely applies, meta-analysis methodology.
The approach to avoid is a forest plot of accuracy figures drawn from unrelated datasets, with a heterogeneity statistic reported as though it were a finding. Such a plot is interpreted as evidence but contains none.
Reporting the review and the evidence gaps identified
The review process should be reported in accordance with the PRISMA 2020 statement, and if the design is a map rather than a synthesis, it should be reported as a scoping review and labelled as such in the title 10,13. The structural limits of the search should be stated specifically: which preprint servers were and were not searched, whether patents were searched, and, where applicable, that commercially developed methods are underrepresented.
Beyond the standard items, three review-level outputs should be specified in the protocol because they make this kind of review reusable: the study-by-item appraisal matrix; the counts of studies with no baseline, with an untuned baseline, and with prospective confirmation; and a defined research agenda stating which task, evaluated on which data, against which baseline and with what prospective test, would advance the question. A review that concludes with such an agenda is useful to the field; one that concludes with “further research is needed” is not.
Where performance figures are reported at all, the dataset should remain attached to the number in the same cell. A metric without its dataset is not a result, and separating them in a table invites precisely the cross-study comparison that the review should caution against.
Limits of what this literature can currently support
Four statements should be made explicitly in a discussion section, because their absence causes reviews in this area to read as promotional.
Retrospective benchmark performance does not establish prospective usefulness. It constitutes a hypothesis about it, and that hypothesis has been tested for only a small fraction of published models.
Nothing in the published methods literature establishes an effect on development timelines or success rates. Those outcomes occur on a timescale over which attribution to a single method is extremely difficult — the multi-decade windows used in analyses of where new drugs come from, such as the 1981 to 2014 span of one widely cited natural products classification, exist because that is the timescale on which such questions can be examined at all 19.
Comparisons between published models are mostly comparisons between datasets. Where two models were never evaluated on the same data with the same split, the literature does not support a statement that one is better.
Absence of a reported failure is not absence of failure. A field in which negative results are rarely published will show a distribution of reported performance that is shifted upwards, and a review cannot correct for this — it can only state it. Specifying these four limits, with reference to what the included studies did and did not do, is the most defensible contribution a review in this field can make.
Key points
Key points
A review should address one domain and one task: discovery, preclinical, clinical development, manufacturing and post-market work do not share an evidence standard, and a review spanning them is a map rather than a synthesis. The question should be framed as task, input data, baseline, validation design and success criterion, and the baseline treated as mandatory; a model with no comparator, or with an untuned one, makes no claim, and the proportion of such studies is itself a review-level finding. The search should extend beyond bibliographic databases, because much of this literature appears first in preprints and proceedings and some of it only in patents or not at all, and this gap should be acknowledged rather than the search described as comprehensive. Split strategy, leakage controls, baseline tuning effort, primary metric and prospective confirmation should be extracted, and a study-by-item appraisal matrix reported instead of a pooled accuracy estimate, since averaging metrics measured on different datasets estimates nothing. The review should state explicitly that retrospective benchmark performance is a hypothesis about prospective usefulness, and that development-timeline and approval outcomes occur over horizons on which attribution to a single method is extremely difficult; the 1981 to 2014 window of one widely cited natural products analysis spans 34 years.
Application to your study
Application to your study
- One domain and one task should be named in the protocol. The relationship between input and output should be stated in one sentence before any search terms are written.
- The baseline should be both an extraction field and a reporting item. The record should state what the baseline was and whether it was tuned to the same extent, or that there was none.
- Every study should be classified by validation design. Random split, scaffold or temporal split, held-out source, data generated after the model, prospective experimental test.
- The search should be built from controlled vocabularies rather than general catch-all phrases. It should combine controlled terms, architecture families and older modelling traditions, with the task expressed in task-specific terms.
- The rule for preprints should be decided in advance. The protocol should state whether preprints are eligible and how a later journal version with a changed result will be handled.
- Leakage controls should be extracted explicitly. These include the split method, duplicate handling, whether evaluation data informed any selection decision, and whether the paper reports this.
- The appraisal matrix, rather than a summary score, should be published. Presenting studies against items allows readers to see which items the field omits.
- The review should conclude with a specific research agenda. The agenda should specify the task, the data, the baseline and the prospective test required.
Frequently asked questions
Almost never in this field, and the reason is not heterogeneity in the statistical sense but incommensurability. The reported estimates derive from different datasets of differing difficulty, with different endpoints, different split strategies and metrics chosen by the authors; averaging them produces a number that estimates no defined quantity. The narrow exception is a set of studies that evaluated different models on the same public benchmark with the same split, where a structured comparison is possible — but the unit of evidence is then the benchmark rather than the study, and any community-wide optimism from repeated tuning on that benchmark applies to all of them equally. More appropriate outputs are a task-by-task structured synthesis, a baseline-relative summary, a count of studies with prospective confirmation, and an evidence gap map of tasks against validation levels.
Three properties. The baseline exists: a model reported without any comparator makes no claim, since performance figures cannot be interpreted in isolation. It is the appropriate comparator: an established physics-based or statistical method, a simpler model of the same family, the human expert workflow the model would replace, or a documented historical rate — whichever the study implicitly claims to improve upon. It received comparable tuning effort, because a lightly configured baseline set against a heavily optimised proposed model produces an improvement that reflects effort rather than method. The review should extract all three as separate fields and report the counts: how many included studies had no baseline, how many had an untuned one, and how many compared against an established alternative.
Leakage is any route by which information from the evaluation data influenced the model or its configuration. The common forms are preprocessing, normalisation or feature selection performed before the split; hyperparameter or threshold selection using the evaluation set; near-duplicate records appearing on both sides of a split, which in chemical datasets is standard and under-reported; and temporal leakage, where a model predicting an earlier period had access to later information. Detecting it from a paper is partly impossible, which is why the extraction field has three states: controlled, not controlled, or cannot tell. The third state should be recorded accurately and its frequency reported — a literature in which most papers do not permit a leakage assessment constitutes a finding in itself.
There are legitimate arguments for both positions, and the essential requirement is to decide in the protocol and report the decision. Inclusion acknowledges that a substantial share of methods work appears first on preprint servers, and sometimes only there, so exclusion leaves a systematic gap in a rapidly developing field. Exclusion restricts the evidence base to peer-reviewed work, which is defensible provided the review states that recent methods work is therefore underrepresented. If preprints are included, the handling rule should be pre-specified: where a preprint and a later journal version both exist, the journal version should be used and any change in the reported result between the two recorded, since that change is itself informative about how these results are modified under peer review.
Because the outcome occurs on a timescale over which attribution to a single method is extremely difficult, and because the published methods literature does not measure that outcome at all. Analyses of where new medicines come from are built on multi-decade windows for this reason — one widely cited classification of natural products as sources of new drugs covers approvals from 1981 to 2014, a 34-year span. A compound influenced by a model at the discovery stage passes through many subsequent decisions before any approval, so even a well-documented case is a narrative rather than an estimate of effect. A review can legitimately report which studies tested predictions prospectively, how many predictions were tested, who tested them, and with what result — and that the longer-horizon outcome has largely not yet had time to be observed.
This depends on the type of model. Where the model is a clinical prediction model — predicting an outcome in patients — a published prediction-model risk-of-bias instrument applies, and it should be named with its version and date, with any modified domain reported. Where the model is not a clinical prediction model, such as a virtual screening pipeline or a manufacturing process control model, no single accepted instrument exists, and devising a scored instrument is less appropriate than using a transparent item list. Leakage, baseline adequacy, split strategy, metric selection, benchmark provenance, prospective confirmation and artefact availability should be assessed, and the results presented as a matrix of studies against items so that a reader can see which specific item failed rather than a composite number.
By stating it as a specific rather than a generic limitation. Commercially developed methods appear in patents, in regulatory filings, or not at all, and detailed methodology can rarely be extracted from any of these. A review of the published literature therefore describes the published evidence base, not the field, and should state this explicitly. Two further asymmetries should be noted: negative results are rarely published in methods work, so the distribution of reported performance is shifted upwards in a way that no review can correct for; and the published record favours approaches that are publishable, which are not necessarily approaches that work. A review that acknowledges these three gaps makes a more defensible claim than one that describes its search as comprehensive.
A transparent map of what has been validated and by what means, together with a specific research agenda. Specifically: a task-by-task table giving each study’s data, split strategy, baseline, primary metric and prospective status; a matrix of studies against appraisal items showing which methodological safeguards the literature does and does not report; counts of studies with no baseline, an untuned baseline and prospective confirmation; and a gap map of tasks against validation levels that shows where the field has conducted only retrospective work. This should be followed by a defined agenda: which task, evaluated on which data, against which baseline and with what prospective test, would advance the question. Such a list is reusable by subsequent researchers, which a pooled accuracy figure is not.
References
- Chapter 4: Searching for and selecting studies. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
- Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews. 2015;4:78. read 9 October 2026. cochrane.org
- Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1):1. doi:10.1186/2046-4053-4-1 read 9 October 2026. doi.org/10.1186/2046-4053-4-1
- Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews. 2014;3(1):74. doi:10.1186/2046-4053-3-74 read 9 October 2026. doi.org/10.1186/2046-4053-3-74
- Bolanos F, Salatino A, Osborne F, Motta E. Artificial intelligence for literature reviews: opportunities and challenges. Artificial Intelligence Review. 2024;57(10):259. doi:10.1007/s10462-024-10902-3 read 9 October 2026. doi.org/10.1007/s10462-024-10902-3
- Qureshi R, Irfan M, Gondal TM, Khan S, Wu J, Hadi MU, et al. AI in drug discovery and its clinical relevance. Heliyon. 2023;9(7):e17575. doi:10.1016/j.heliyon.2023.e17575 read 9 October 2026. doi.org/10.1016/j.heliyon.2023.e17575
- Zhang Y, Luo M, Wu P, Wu S, Lee TY, Bai C. Application of Computational Biology and Artificial Intelligence in Drug Design. International Journal of Molecular Sciences. 2022;23(21):13568. doi:10.3390/ijms232113568 read 9 October 2026. doi.org/10.3390/ijms232113568
- Karim MR, Beyan O, Zappa A, Costa IG, Rebholz-Schuhmann D, Cochez M, Decker S. Deep learning-based clustering approaches for bioinformatics. Briefings in Bioinformatics. 2021;22(1):393-415. doi:10.1093/bib/bbz170 read 9 October 2026. doi.org/10.1093/bib/bbz170
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71 read 9 October 2026. doi.org/10.1136/bmj.n71
- Gauthier J, Vincent AT, Charette SJ, Derome N. A brief history of bioinformatics. Briefings in Bioinformatics. 2019;20(6):1981-1996. doi:10.1093/bib/bby063 read 9 October 2026. doi.org/10.1093/bib/bby063
- Bramer WM, de Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association. 2018;106(4):531-541. doi:10.5195/jmla.2018.283 read 9 October 2026. doi.org/10.5195/jmla.2018.283
- Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine. 2018;169(7):467-473. doi:10.7326/M18-0850 read 9 October 2026. doi.org/10.7326/M18-0850
- Yang Y, Gao J, Wang J, Heffernan R, Hanson J, Paliwal K, Zhou Y. Sixty-five years of the long march in protein secondary structure prediction: the final stretch? Briefings in Bioinformatics. 2018;19(3):482-494. doi:10.1093/bib/bbw129 read 9 October 2026. doi.org/10.1093/bib/bbw129
- Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology. 2018;18(1):143. doi:10.1186/s12874-018-0611-x read 9 October 2026. doi.org/10.1186/s12874-018-0611-x
- Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ. 2017;356:j448. read 9 October 2026. cochrane.org
- Angermueller C, Parnamaa T, Parts L, Stegle O. Deep learning for computational biology. Molecular Systems Biology. 2016;12(7):878. doi:10.15252/msb.20156651 read 9 October 2026. doi.org/10.15252/msb.20156651
- DiMasi JA, Grabowski HG, Hansen RW. Innovation in the pharmaceutical industry: New estimates of R&D costs. Journal of Health Economics. 2016;47:20-33. doi:10.1016/j.jhealeco.2016.01.012 read 9 October 2026. doi.org/10.1016/j.jhealeco.2016.01.012
- Newman DJ, Cragg GM. Natural Products as Sources of New Drugs from 1981 to 2014. Journal of Natural Products. 2016;79(3):629-661. doi:10.1021/acs.jnatprod.5b01055 read 9 October 2026. doi.org/10.1021/acs.jnatprod.5b01055
- Libbrecht MW, Noble WS. Machine learning applications in genetics and genomics. Nature Reviews Genetics. 2015;16(6):321-332. doi:10.1038/nrg3920 read 9 October 2026. doi.org/10.1038/nrg3920
- Welch L, Lewitter F, Schwartz R, Brooksbank C, Radivojac P, Gaeta B, Schneider MV. Bioinformatics Curriculum Guidelines: Toward a Definition of Core Competencies. PLOS Computational Biology. 2014;10(3):e1003496. doi:10.1371/journal.pcbi.1003496 read 9 October 2026. doi.org/10.1371/journal.pcbi.1003496
- Gross A, Schirm S, Scholz M. Ycasd - a tool for capturing and scaling data from graphical representations. BMC Bioinformatics. 2014;15(1):219. doi:10.1186/1471-2105-15-219 read 9 October 2026. doi.org/10.1186/1471-2105-15-219
- Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347. read 9 October 2026. cochrane.org
- Swinney DC, Anthony J. How were new medicines discovered? Nature Reviews Drug Discovery. 2011;10(7):507-519. doi:10.1038/nrd3480 read 9 October 2026. doi.org/10.1038/nrd3480
- Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database: update and key issues. New England Journal of Medicine. 2011;364:852-860. read 9 October 2026. cochrane.org
- Paul SM, Mytelka DS, Dunwiddie CT, Persinger CC, Munos BH, Lindborg SR, Schacht AL. How to improve R&D productivity: the pharmaceutical industry's grand challenge. Nature Reviews Drug Discovery. 2010;9(3):203-214. doi:10.1038/nrd3078 read 9 October 2026. doi.org/10.1038/nrd3078
- Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal. 2009;26(2):91-108. doi:10.1111/j.1471-1842.2009.00848.x read 9 October 2026. doi.org/10.1111/j.1471-1842.2009.00848.x
- Benson DA, Karsch-Mizrachi I, Lipman DJ, Ostell J, Wheeler DL. GenBank. Nucleic Acids Research. 2008;36(Database issue):D25-D30. doi:10.1093/nar/gkm929 read 9 October 2026. doi.org/10.1093/nar/gkm929
- Materi W, Wishart DS. Computational systems biology in drug discovery and development: methods and applications. Drug Discovery Today. 2007;12(7-8):295-303. doi:10.1016/j.drudis.2007.02.013 read 9 October 2026. doi.org/10.1016/j.drudis.2007.02.013
- Larranaga P, Calvo B, Santana R, Bielza C, Galdiano J, Inza I, et al. Machine learning in bioinformatics. Briefings in Bioinformatics. 2006;7(1):86-112. doi:10.1093/bib/bbk007 read 9 October 2026. doi.org/10.1093/bib/bbk007
- Ouzounis CA, Valencia A. Early bioinformatics: the birth of a discipline - a personal view. Bioinformatics. 2003;19(17):2176-2190. doi:10.1093/bioinformatics/btg309 read 9 October 2026. doi.org/10.1093/bioinformatics/btg309
- Richardson WS, Wilson MC, Nishikawa J, Hayward RS. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club. 1995;123(3):A12-A13. doi:10.7326/ACPJC-1995-123-3-A12 read 9 October 2026. doi.org/10.7326/ACPJC-1995-123-3-A12
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 9). Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis. Pubrica. https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/
Pubrica Academy. “Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis.” Pubrica, 9 Oct. 2026, pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/.
Pubrica Academy (2026) Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis. Pubrica. Available at: https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/ (Accessed: 15 October 2026).
Pubrica Academy. “Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis.” Pubrica. October 9, 2026. https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/
Pubrica Academy, ‘Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis’ (Pubrica, 9 October 2026) <https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/> accessed 15 October 2026.
Pubrica Academy. Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis. [Internet]. Pubrica; 2026 Oct 9 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/
Pubrica Academy, “Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis,” Pubrica, Oct. 9, 2026. [Online]. Available: https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/
@misc{systematic_review_of_machine_learning__2026,
author = {{Pubrica Academy}},
title = {Systematic Reviews of Machine Learning in the Pharmaceutical Industry: Scope, Appraisal and Synthesis},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/latest-topics/systematic-review-of-machine-learning-in-the-pharmaceutical-industry/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Protocol and extraction stub for a machine learning review
The first block prevents the most common failure, namely a review spanning five domains. The leakage and baseline fields are those that distinguish a validated model from a well-fitted one.
MACHINE LEARNING IN PHARMA — REVIEW STUB
=== BLOCK 1: SCOPE (one domain, one task) ===
Domain (tick ONE):
[ ] discovery / target work
[ ] preclinical / translational
[ ] clinical development
[ ] manufacturing / quality
[ ] post-market / pharmacovigilance
TASK as input -> output: _____________________________
Success criterion being reviewed (tick ONE):
[ ] retrospective predictive performance
[ ] external or temporal validation
[ ] prospective experimental confirmation
[ ] candidates entering development
[ ] approvals
Design label: [ ] systematic review [ ] scoping review
[ ] evidence gap map
=== BLOCK 2: SEARCH ===
Databases, platforms, dates: _________________________
Method concept — controlled vocabulary terms: __________
Method concept — architecture and tradition terms:
deep learning / neural network / random forest / gradient
boosting / support vector / graph neural network / transformer /
autoencoder / QSAR / cheminformatics / data-driven modelling
plus: ______________________________________________
TASK concept terms (NOT "drug discovery"): ____________
Date limit: [ ] none (recommended) [ ] from ____ because ______
Preprint servers searched: ___________ date ______
Conference proceedings searched: __________ date ______
PREPRINT RULE: where preprint and journal version both exist ->
[ ] use journal version and record any change in the result
Patents searched? [ ] yes [ ] NO — declared as a limitation [ ]
Code repositories: [ ] availability checks only, not identification
=== BLOCK 3: EXTRACTION, PER STUDY ===
TASK AND DATA
Input representation: ________________________________
Output definition: __________________________________
Dataset(s) and whether public: ______________________
Sample size in the unit that matters: ____ (compounds / assays /
patients / sites — state which)
Label provenance and known label noise: _____________
Class balance: ______________________________________
SPLIT AND LEAKAGE
Split type: [ ] random [ ] scaffold [ ] temporal [ ] by source
[ ] data generated after the model was built
Near-duplicate handling across the split: [ ] controlled
[ ] not controlled [ ] cannot tell
Did evaluation data influence preprocessing, feature selection,
hyperparameters or thresholds? [ ] no [ ] yes [ ] cannot tell
MODEL AND TRAINING
Model family: _______________________________________
Hyperparameter selection method: ____________________
BASELINE: [ ] none [ ] named: ______________________
Baseline tuned with comparable effort? [ ] yes [ ] no [ ] cannot tell
EVALUATION
Metrics reported: ___________________________________
PRIMARY metric, and was it declared in advance? _______
Uncertainty reported? [ ] yes, how: ______ [ ] no
Performance relative to baseline: ___________________
Subgroup / series / site results? [ ] yes [ ] no
Benchmark widely used for model development? [ ] yes [ ] no
PROSPECTIVE COMPONENT
Any prediction tested afterwards? [ ] no [ ] yes
How many predictions tested: ____ By whom: ______
Independent of the prediction team? [ ] yes [ ] no
Outcome: __________________________________________
Chronology recorded (model built / predicted / confirmed): ______
AVAILABILITY
Code [ ] Trained model [ ] Training data [ ] Terms: ______
=== BLOCK 4: REVIEW-LEVEL OUTPUTS (commit to these) ===
Study x item appraisal MATRIX (not a score): [ ] planned
Counts: studies with no baseline ____
studies with an untuned baseline ____
studies with prospective confirmation ____
studies where leakage could not be assessed ____
Task x validation-level GAP MAP: [ ] planned
NAMED RESEARCH AGENDA: which task, on which data, against which
baseline, with what prospective test:
___________________________________________________
=== BLOCK 5: LIMITS TO STATE EXPLICITLY ===
[ ] Retrospective benchmark performance is a hypothesis about
prospective usefulness, tested for few of the included models
[ ] No claim about development timelines or success rates
[ ] Cross-study performance comparisons are dataset comparisons
[ ] Unpublished and commercially developed methods are
structurally underrepresented
[ ] Negative results are rarely published, so reported performance
is shifted upwards in a way the review cannot correct Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica scopes these reviews to one domain and one task before the search, develops a strategy covering the vocabularies under which the same work is published, and extracts the split, leakage and baseline fields that distinguish a validated model from a well-fitted one. The deliverable is a study-by-item appraisal matrix and an evidence gap map rather than a pooled accuracy figure, together with a clearly worded statement of what the published evidence base cannot support.