Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Predictive analysis · Predictive analysis in research

Predictive Analysis in Research: Scope, Limitations and Appraisal

Most published prediction models are at high risk of bias, and the step that would demonstrate benefit to patients is almost never taken.

Pubrica AcademyWritten by a research writing specialist Reviewed bya research methodology reviewer Published15 October 2026 Last checked15 October 2026 Reading time14 minutes Sources10 primary

Predictive analysis in research involves estimating an outcome for an individual from observable characteristics. It addresses a more specific and narrower set of questions than is usually attributed to it, and the published literature is in poorer condition than its volume suggests; knowing what to check is therefore the most useful content of this page.

This page covers where predictive analysis fits in a research programme — which questions it answers, which it cannot, and what a reader should check before believing a published model. The method itself, including the performance measures and the reporting checklist, is covered separately so that neither page has to summarise the other badly. See how predictive analytics works for the method, the performance measures and the specifications.

Questions addressed by prediction in research

Table 1. Research uses of predictive analysis

UseThe questionNotes
PrognosisWho is likely to experience this outcome, and when?A prognostic model. This is the most common research use, and the one most often reported without external evaluation.
DiagnosisDoes this person have the condition now?A diagnostic model. Reported under TRIPOD+AI like any other prediction model.
Risk stratification for trial designWhich participants are sufficiently likely to experience the event to justify enrolment?Prediction used as a design tool rather than as a finding.
Targeting a scarce interventionWho should be prioritised when resources are insufficient?Requires calibration, not only discrimination: the predicted values must be accurate, not merely correctly ordered.
Explaining why something happensNot a prediction question.This is where prediction is most often misused. A causal question requires a causal design, and interpreting a predictor’s coefficient as an effect is a documented error.

The causal question that prediction cannot address

The distinction with the greatest consequences

This is the most consequential distinction on the page, and the popular framing of predictive analytics misrepresents it. Business and vendor descriptions routinely state that predictive analytics answers “what will happen, and why”. It does not answer why. A prediction model selects variables according to how well they forecast, regardless of whether they are causal; a causal question requires different methods and different assumptions. Interpreting a predictor’s coefficient as the effect of intervening on it is a documented error, found in 26% of 180 observational studies across six fields and in 38% of the prediction studies specifically. A model that predicts readmission accurately does not indicate that changing a predictor would prevent a readmission.

The measurement: “Conflation of etiology and prediction is a common methodological error in observational medical research”1, because “Causal and prediction research usually require different methods, and yet their findings may get conflated”1 Of 180 observational studies across six medical fields, 46 (26%, 95% CI 19% to 33%) conflated the two1; among prediction studies specifically, “Within prediction studies 38% was conflated”1 The authors' advice: “Researchers and readers should be mindful of the differences between etiology and prediction”1

If your question is what would happen if you intervened, you need a design that can answer it. Our pages on variables and what each role implies and biostatistics in clinical research cover the alternative.

Classification of prediction model studies

Prediction research runs in “consecutive steps generally advocated for risk prediction model research”2, and the three are routinely collapsed into one word. Development builds the model; internal validation assesses it on the same data, which “will not provide the degree of heterogeneity”3 that the model will meet in practice. External evaluation means “taking the original model or simplified score, with its predictors and assigned weights”3 and measuring what happens. And “model impact studies”3 ask whether using the model changes anything, which “requires a comparative study”3 with a usual-care arm.

Distinction between accuracy and benefit

The third row is usually absent, and it addresses a different question from the first two. A model can be accurate yet change nothing, either because no one acts on it or because acting on it confers no benefit. Measuring benefit requires a comparative study with a control arm receiving usual care, which is a trial rather than a modelling exercise. Accuracy is not equivalent to benefit.

The term “validated”

This terminological point merits adoption because it alters the interpretation of every claim in the field. TRIPOD+AI deliberately replaces “validation” with “evaluation” and states unequivocally: “There is no such thing as a validated prediction model.” Performance is always performance in some population. A model described as “validated” without a statement of where and in whom conveys no information.

Characteristics of the published literature

This is the part a reader most needs and most pages omit. A systematic review of 152 studies of prediction models developed with supervised machine learning found that “Most studies on machine learning based prediction models show poor methodological quality”7:

Table 2. Risk of bias in 152 machine learning prediction model studies

FindingCountShare
Judged at high risk of bias148 of 171 analyses87% (95% CI 81% to 91%)
Risk arising in the analysis domain128 of 152 models84%
Discrimination or calibration measures missing91 of 15260%
Inadequate number of events per candidate predictor85 of 15256%
Overfitting assessed improperly59 of 15239%

Two of those are worth saying out loud. “Only half of the included studies examined potential overfitting of models”7, and “model calibration tables or plots were often not presented”7 The review's conclusion is that “these studies usually require (many) more participants and events”7 were needed, and that “Efforts to improve the design, conduct, reporting, and validation of such studies are necessary”7

The same pattern appears in a different field and a different decade. A living review of COVID-19 prediction models covering 412 studies and 731 models found 545 of 606 prognostic models (90%) at high risk of bias8, concluding that “Most published prediction model studies were poorly reported and at high risk of bias”8 “such that their reported predictive performances are probably optimistic”8, and that “unreliable predictions could cause more harm than benefit in guiding clinical decisions”8

That second review is a living review, and the figures above belong to its final update, whose searches ran to 17 February 2021 rather than to its 2020 first publication.

Checks required before a model is accepted

Table 3. Six questions for a published prediction model

Which kind of study is this?Development, external evaluation or impact. These answer different questions, and the third is almost never undertaken.
In whom was it evaluated?A model evaluated only in the data from which it was built has not been shown to generalise. “Validated” without a named population conveys no information.
Is calibration reported, or only discrimination?A c-statistic alone cannot indicate whether the predicted risks are accurate. Calibration was missing from 60% of machine learning models in one review.
How many events per predictor parameter?With too few events, the model describes its development sample. The number should be judged against the setting rather than against a rule of thumb.
Was the risk of bias assessed?Most published models are at high risk of bias. PROBAST (Prediction model Risk Of Bias ASsessment Tool) and PROBAST+AI exist for this purpose.
Is any causal claim being made?A paper that recommends intervening on a predictor has exceeded what a prediction model can support.

On calibration specifically, which is the check most often impossible to perform because the paper does not report it: it “receives little attention”9 in the assessment of risk models, and “poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making”9

On sample size, the thing to avoid is a rule of thumb in either direction. The relevant criteria target “the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP)”10, and the paper usually cited for a round number in fact says “This reinforces why rules of thumb (eg, 10 EPP) should be avoided”10

Appraisal and reporting

To appraise someone else's model, use PROBAST, which “PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis”5 and where “These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB”5 Its 2025 successor “consists of two distinctive parts: model development and model evaluation”6, and “Both parts contain four domains: participants and data sources, predictors, outcome, and analysis.”6

A substantive revision rather than a relabelling

PROBAST+AI is not simply PROBAST with machine learning added. PROBAST is a single assessment with four domains and 20 signalling questions, whereas PROBAST+AI is divided into two parts (16 signalling questions for development and 18 for evaluation) and renames the first domain from “participants” to “participants and data sources”. Applicability is rated for three domains rather than all four.

To report your own, use TRIPOD+AI, which applies “irrespective of whether regression modelling or machine learning methods have been used”4 and “supersedes the TRIPOD 2015 checklist, which should no longer be used”4 It marks items by study type: “Items in the checklist denoted D apply to studies that describe the development of a prediction model”4 and “items denoted E apply to studies that evaluate the performance of a prediction model”4

Two concurrent statements about TRIPOD

Two statements are both true and are easily conflated into a single incorrect statement. TRIPOD+AI states that the 2015 checklist “should no longer be used”. But the 2015 explanation and elaboration paper remains the one to consult, because no TRIPOD+AI equivalent has been published. The current position is therefore a new checklist accompanied by the earlier commentary.

Recommended working practices

Practical application

  1. Authors should state whether the question is predictive or causal before selecting a method; a prediction model cannot estimate the effect of intervening.
  2. The type of study reported (development, evaluation or impact) should be stated in the title or abstract.
  3. Calibration should be reported in addition to discrimination, with a plot; without it, readers cannot judge whether the predicted risks are usable.
  4. A model should never be described as “validated” without naming the population in which it was evaluated.
  5. The sample size should be derived from the setting rather than from an events-per-variable rule, and predictor parameters should be counted.
  6. Any model intended as a basis for further work should first be appraised with PROBAST or PROBAST+AI, since most published models are at high risk of bias.
  7. Studies should be reported in accordance with TRIPOD+AI rather than TRIPOD 2015, while the 2015 explanation paper, which has not yet been replaced, remains the reference commentary.
  8. Intervention on a predictor should not be recommended on the basis of a prediction model.

Key points

Predictive analysis addresses who is likely to experience an outcome (prognosis, diagnosis, risk stratification and targeting), not why it occurs. Conflation of the two has been measured in 26% of observational studies and 38% of prediction studies. Development, external evaluation and impact assessment are three different studies, and the third is almost never conducted, so accuracy is routinely mistaken for benefit. The published literature is largely at high risk of bias: 87% of analyses in one review of machine learning models, with calibration missing from 60%. Models should be appraised with PROBAST before any further work builds on them, and studies should be reported in accordance with TRIPOD+AI.

It estimates an outcome for an individual from observed characteristics: prognosis (who will experience an outcome, and when), diagnosis (whether a person has the condition now), risk stratification for trial design, and targeting of a scarce intervention. It is not designed to explain why something happens.

No, and treating it as though it could is a documented error rather than a philosophical point. A prediction model selects variables according to how well they forecast, whether or not they are causal. A scoping review of 180 observational studies found that 26% conflated prediction and causation, rising to 38% among the prediction studies.

Development (building the model, with internal validation on the same data), external evaluation (applying the original model, with its predictors and assigned weights, to new individuals), and impact assessment (whether providing clinicians with the predictions changes decisions and outcomes). The third “requires a comparative study” and is almost never undertaken.

On the available evidence, most are not. A systematic review of 152 studies of machine learning prediction models judged 148 of 171 analyses (87%) at high risk of bias. A review of COVID-19 prediction models found 545 of 606 prognostic models (90%) at high risk of bias, with performance “probably optimistic”.

Alone, it conveys very little. TRIPOD+AI states that “there is no such thing as a validated prediction model” and replaces “validation” with “evaluation”, because performance is always performance in a particular population. Readers should ask where and in whom the model was evaluated.

PROBAST, which is “organized into the following 4 domains: participants, predictors, outcome, and analysis” with 20 signalling questions — or PROBAST+AI (2025), which splits into separate development and evaluation parts with 16 and 18 questions and renames the first domain “participants and data sources”.

TRIPOD+AI (2024), which covers regression and machine learning alike and “supersedes the TRIPOD 2015 checklist, which should no longer be used”. It has 27 items containing 52 subitems, and marks each as applying to development or to evaluation studies.

References

  1. Ramspek CL, Steyerberg EW, Riley RD, Rosendaal FR, Dekkers OM, Dekker FW, van Diepen M. Prediction or causality? A scoping review of their conflation within current observational research. European Journal of Epidemiology. 2021. doi:10.1007/s10654-021-00794-w read 15 October 2026. View on a university repository copyMeasures how often prediction and causation are conflated in published research: 26% of 180 studies overall, and 38% of the prediction studies. Read from Keele's repository; volume and page numbers were not shown on that record.
  2. Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683. doi:10.1136/heartjnl-2011-301246 read 15 October 2026. View on Heart (BMJ)The first of a two-part overview, on model development and internal validation. Read on the publisher's own site.
  3. Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98(9):691. doi:10.1136/heartjnl-2011-301247 read 15 October 2026. View on a university-hosted copyThe second part, on external validation and impact assessment. The source for the point that an impact study needs a comparative design. Read from a university mirror.
  4. Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378 read 15 October 2026. View on a university repository copyThe current reporting guideline for prediction model studies: 27 items containing 52 subitems, covering regression and machine learning alike. Read from LMU Munich's repository copy because the publisher's own page was not reachable.
  5. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Annals of Internal Medicine. 2019;170(1):51–58. doi:10.7326/M18-1376 read 15 October 2026. View on University of Bristol research portalFour domains and 20 signalling questions. Verified from the institutional record; the journal's full text was not reachable, so nothing is quoted from the body.
  6. Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, van Smeden M, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505 read 15 October 2026. View on Oxford University Research ArchiveStructurally different from PROBAST rather than a relabel: two parts, separate question sets for development and evaluation, and a renamed first domain. Verified from the institutional record only.
  7. Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, Collins GS, Bajpai R, Riley RD, Moons KGM, Hooft L. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021;375:n2281. doi:10.1136/bmj.n2281 read 15 October 2026. View on a university repository copyThe strongest single citation for this page: it measures, rather than asserts, how often machine-learning prediction studies omit calibration, mishandle overfitting and run on too few events. Read from Keele's repository.
  8. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi:10.1136/bmj.m1328 (living review; final update, searches to 17 February 2021.) read 15 October 2026. View on The BMJA living systematic review, so the figures belong to the final update rather than to the 2020 first publication. Cited with the search date for that reason. Read on the publisher's own site.
  9. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230. doi:10.1186/s12916-019-1466-7 read 15 October 2026. View on DOAJ recordOn the neglect of calibration. The publisher's full text was not reachable, so only the abstract is quoted and nothing is attributed to the body of the paper.
  10. Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, Collins GS. Minimum sample size for developing a multivariable prediction model: PART II — binary and time-to-event outcomes. Statistics in Medicine. 2019;38(7):1276–1296. doi:10.1002/sim.7992 read 15 October 2026. View on Oxford University Research ArchiveCited for the sample-size criteria and, importantly, for arguing against the rule of thumb usually attributed to this literature. Published online 2018 and in issue in 2019; verified from the institutional record.

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

Related: how predictive analytics works · biostatistics in clinical research · concepts that get misstated · reporting guidelines.

How Pubrica can help

Pubrica’s biostatisticians and data scientists develop and evaluate prediction models and appraise existing models before further work builds on them, providing PROBAST assessment, sample size derived from the setting, joint reporting of discrimination and calibration, external evaluation designed as a separate study, and reporting in accordance with TRIPOD+AI.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.