Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Predictive analysis · How predictive analytics works

Predictive Analytics: Principles, Performance Measures and Limitations

A prediction model estimates what is likely, but neither why it occurs nor the effect of intervening. Conflation of these questions is a documented error rather than a minor point.

Pubrica AcademyWritten by a research writing specialist Reviewed bya research methodology reviewer Published15 October 2026 Last checked15 October 2026 Reading time15 minutes Sources10 primary

Predictive analytics estimates an outcome that has not yet occurred, or is not yet known, from observable characteristics. In health research it takes the form of a clinical prediction model, and that literature, rather than the business-analytics literature, provides the methodological authority on appropriate practice.

Definition of predictive analytics

Table 1. Three kinds of analytics, described

The questionWhat it does
DescriptiveWhat happened?Summarising existing data: counts, rates, trends and dashboards.
PredictiveWhat is likely to happen?Estimating an outcome for an individual or period from observed characteristics. Not why, and not what intervening would do.
PrescriptiveWhat should be done?Recommending an action. This requires assumptions about the effect of acting, which prediction alone does not supply.

Rationale for the absence of a citation for the table

The familiar descriptive, predictive and prescriptive taxonomy merits comment, because its provenance is more significant than it appears. A search for a methodological authority underlying it identified none: the scholarly papers using the taxonomy trace the definitions to a vendor white paper, and all other sources retrieved were marketing material. The three terms are therefore described here in plain language without citation, since attaching one would present a convention as a standard. The clinical prediction-model literature cited throughout the remainder of this page is different: it constitutes genuine methodological authority and contains the substance of the topic.

In health research the object is a clinical prediction model: diagnostic if it estimates a condition the patient has now, prognostic if it estimates a future event. TRIPOD+AI covers both, and covers them “irrespective of whether regression modelling or machine learning methods have been used”1 — which is worth knowing, because it means machine learning does not get a separate, looser standard.

Distinction between prediction and explanation

The most important distinction

This is the most consequential distinction on the page, and the popular framing of predictive analytics misrepresents it. Business and vendor descriptions routinely state that predictive analytics answers “what will happen, and why”. It does not answer why. A prediction model selects variables according to how well they forecast, regardless of whether they are causal; a causal question requires different methods and different assumptions. Interpreting a predictor’s coefficient as the effect of intervening on it is a documented error, found in 26% of 180 observational studies across six fields and in 38% of the prediction studies specifically. A model that predicts readmission accurately does not indicate that changing a predictor would prevent a readmission.

The evidence that this goes wrong in practice: “Conflation of etiology and prediction is a common methodological error in observational medical research”10, because “Causal and prediction research usually require different methods, and yet their findings may get conflated”10 In a review of 180 observational studies from leading journals across six medical fields, 46 (26%, 95% CI 19% to 33%) conflated the two10. Among the prediction studies specifically, “Within prediction studies 38% was conflated”10 The characteristic form of the error is “a causal interpretation of predictor effects”10.

So: a model that predicts readmission well tells you whom to watch. It does not tell you that changing one of its predictors would prevent a readmission. Those are different studies, with different designs. Our comparison of study designs covers which design answers which question.

Development, evaluation and impact studies

Prediction model research runs in “consecutive steps generally advocated for risk prediction model research”4, and the three are routinely collapsed into one claim about a model being “validated”.

Table 2. Three kinds of prediction model study

Study typeThe questionWhat it involves
DevelopmentDoes a model fit these data?Builds the model and assesses it on the data from which it was developed. Internal validation belongs here, and it cannot reproduce real-world heterogeneity.
External validation, or evaluationDoes it work in other people?Applies the original model, with its predictors and assigned weights, to new individuals and measures performance. No control group is needed.
ImpactDoes using it change anything?Asks whether providing clinicians with the predictions changes decisions and outcomes. Requires a comparative design with a usual-care arm, and is almost never undertaken.

Internal validation has a specific limitation worth naming: evaluating a model on the same population it was developed in “will not provide the degree of heterogeneity”5 that it will meet in practice. External validation means “taking the original model or simplified score, with its predictors and assigned weights”5 and measuring how it performs in new people. TRIPOD+AI defines internal validation as “Evaluating the performance of a prediction model on the same population on which the model was developed.”1

The step almost never taken

The third row is usually absent, and it addresses a different question from the first two. A model can be accurate yet change nothing, either because no one acts on it or because acting on it confers no benefit. Measuring benefit requires a comparative study with a control arm receiving usual care, which is a trial rather than a modelling exercise. Accuracy is not equivalent to benefit.

The term “validated”

This terminological point merits adoption because it alters the interpretation of every claim in the field. TRIPOD+AI deliberately replaces “validation” with “evaluation” and states unequivocally: “There is no such thing as a validated prediction model.” Performance is always performance in some population. A model described as “validated” without a statement of where and in whom conveys no information.

Discrimination and calibration

Discrimination and calibration: two properties, one usually unreported

Discrimination and calibration are different properties, and usually only one of them is reported. Discrimination is the ability of the model to separate those who experience the outcome from those who do not, measured by the c-statistic or area under the curve. Calibration is the agreement between predicted risks and observed frequencies: whether, among people assigned a 20% risk, approximately 20% experience the event. A model may separate cases from non-cases well and still assign inaccurate risk values, which renders it of no use for decisions based on a predicted risk. This explanation is Pubrica’s own; what the cited paper establishes is that calibration “receives little attention” and that poorly calibrated algorithms “can be misleading and potentially harmful for clinical decision-making”.

TRIPOD+AI defines them: calibration is “Agreement between observed outcomes and estimated values from the model”1 and discrimination is “How well the predictions from the model differentiate between individuals with and without the outcome”1

On the neglect, and the consequence: calibration “receives little attention”6 in the assessment of risk models, a situation the authors argue “needs to change immediately”6, because “poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making”6

A sourcing note: that paper's full text was not reachable from our environment, so only its abstract is quoted and nothing is attributed to its body. The measured version of the same point — how often calibration is simply absent — is in the review cited in the table below.

Sample size and overfitting

A model fitted to too few events describes the sample it was built on rather than the population it is meant to serve. The sample size criteria target “the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP)”7, and aim to “Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p”7 with “small optimism in predictor effect estimates as defined by a global shrinkage factor of >=0.9”7

A rule of thumb rejected by its own cited source

Two corrections apply to a figure that is often cited loosely. First, the unit in this literature is events per predictor parameter, not events per variable, since a categorical predictor or a non-linear term contributes more than one parameter. Second, and more importantly, this paper argues against the rule of thumb it is usually cited to support: “this reinforces why rules of thumb (eg, 10 EPP) should be avoided”. Its own worked examples require as few as approximately 5 and as many as approximately 23, depending on the setting. A page that cites it in support of a requirement for 10 events per variable has reversed its argument.

Quality of published prediction models

This is the part of the subject that vendor material omits. A systematic review of 152 studies of prediction models developed with supervised machine learning found that “Most studies on machine learning based prediction models show poor methodological quality”9:

Table 3. Risk of bias in 152 machine learning prediction model studies

FindingCountShare
Judged at high risk of bias148 of 171 analyses87% (95% CI 81% to 91%)
Risk arising in the analysis domain128 of 152 models84%
Discrimination or calibration measures missing91 of 15260%
Inadequate number of events per candidate predictor85 of 15256%
Overfitting assessed improperly59 of 15239%

Two of those deserve emphasis. “Only half of the included studies examined potential overfitting of models”9, and “model calibration tables or plots were often not presented”9 The review's conclusion on sample size is that “these studies usually require (many) more participants and events”9 than the studies used, and that “Efforts to improve the design, conduct, reporting, and validation of such studies are necessary”9

The pattern repeats in a different field. A living systematic review of COVID-19 prediction models covering 412 studies and 731 models found that, among the 606 prognostic models, 545 (90%) were at high risk of bias8. Its conclusion: “Most published prediction model studies were poorly reported and at high risk of bias”8 “such that their reported predictive performances are probably optimistic”8, and “unreliable predictions could cause more harm than benefit in guiding clinical decisions”8 “Models with low risk of bias should be validated before clinical implementation”8

A citation note on that second review: it is a living review, and the figures above belong to its final update, whose searches ran to 17 February 2021, not to the 2020 first publication the citation carries.

TRIPOD+AI and PROBAST+AI

Two instruments, for two purposes: one for reporting your own model, one for appraising someone else's.

Table 4. The TRIPOD family

GuidelineYearCitationScope
TRIPOD+AI2024BMJ 2024;385:e078378The current core statement, covering regression or machine learning.
TRIPOD 20152015Ann Intern Med 2015;162(1):55–63Superseded; its authors state that the checklist should no longer be used.
TRIPOD 2015 explanation and elaboration2015Ann Intern Med 2015;162(1):W1–W73Still in use; no TRIPOD+AI equivalent has yet been published.
TRIPOD-Cluster2023BMJ 2023;380:e071018An extension for clustered data, not a competing core version.
TRIPOD-SRMA2023BMJ 2023;381:e073538For systematic reviews and meta-analyses of prediction models.
TRIPOD-LLM2025Nat Med 2025For large language models.

Two true statements that are easily conflated

Two statements are both true and are easily conflated into a single incorrect statement. TRIPOD+AI states that the 2015 checklist “should no longer be used”. But the 2015 explanation and elaboration paper remains the one to consult, because no TRIPOD+AI equivalent has been published. The current position is therefore a new checklist accompanied by the earlier commentary.

TRIPOD+AI marks its items by study type: “Items in the checklist denoted D apply to studies that describe the development of a prediction model”1 and “items denoted E apply to studies that evaluate the performance of a prediction model”1 — so the checklist enforces the development-versus-evaluation distinction rather than leaving it to the author.

For appraising a model. PROBAST “PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis”2, and “These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB”2 Its 2025 successor “consists of two distinctive parts: model development and model evaluation”3, where “Both parts contain four domains: participants and data sources, predictors, outcome, and analysis.”3

A substantive revision rather than a relabelling

PROBAST+AI is not simply PROBAST with machine learning added. PROBAST is a single assessment with four domains and 20 signalling questions, whereas PROBAST+AI is divided into two parts (16 signalling questions for development and 18 for evaluation) and renames the first domain from “participants” to “participants and data sources”. Applicability is rated for three domains rather than all four.

Recommended working practices

Practical application

  1. It should first be determined whether the question is predictive or causal. If the aim is to establish the effect of intervening, a prediction model is the wrong tool and will be misleading.
  2. Both discrimination and calibration should be reported, with a calibration plot; one without the other does not establish that a model is usable.
  3. A model should never be described as “validated” without specification of the population. TRIPOD+AI holds that there is no such thing as a validated model.
  4. The sample size should be derived from the setting rather than from a rule of thumb, and the number of predictor parameters, not of variables, should be counted.
  5. TRIPOD+AI, rather than TRIPOD 2015, should be used for reporting, although the 2015 explanation paper, which has not yet been replaced, should still be consulted.
  6. An existing model should be appraised with PROBAST or PROBAST+AI before further work builds on it, since most published models are at high risk of bias.
  7. External evaluation should be planned as a separate study in a population that differs from the development population, since internal validation cannot reveal heterogeneity that it did not encounter.
  8. A claim of clinical benefit requires an impact study with a usual-care comparison, since accuracy is not equivalent to benefit.

Key points

A prediction model estimates what is likely, but neither why it occurs nor the effect of intervening, a distinction that 26% of observational studies misrepresent. Development, external evaluation and impact assessment are three different studies, and the third is almost never conducted. Calibration should be reported in addition to discrimination: calibration was missing from 60% of machine learning models in one review, which judged 87% of analyses to be at high risk of bias. TRIPOD+AI, rather than TRIPOD 2015, should be used. The rule of thumb of 10 events should be avoided, since the paper to which it is attributed argues against it. Finally, there is no such thing as a validated prediction model.

The use of observed characteristics to estimate an outcome that has not yet occurred, or is not yet known, for an individual or a period. In health research this takes the form of a clinical prediction model, which is diagnostic if it estimates a current condition and prognostic if it estimates a future event. It addresses what is likely, not why, and not what would happen following an intervention.

No, and conflating them is a documented error rather than a philosophical point. A prediction model selects variables according to how well they forecast, whether or not they are causal. A scoping review of 180 observational studies found that 26% conflated the two, rising to 38% among the prediction studies. The specific error is interpreting a predictor’s coefficient as the effect of intervening on it.

TRIPOD+AI, published in 2024, covers studies using either regression or machine learning. It has 27 items containing 52 subitems, and it states that it “supersedes the TRIPOD 2015 checklist, which should no longer be used”. The 2015 explanation and elaboration paper remains in use, because no TRIPOD+AI equivalent has yet been published.

Discrimination is the ability of the model to separate people who experience the outcome from those who do not, measured by the c-statistic. Calibration is the agreement between predicted risks and observed frequencies. A model can discriminate well yet be poorly calibrated, which renders it unsuitable for decisions based on a predicted risk. Calibration “receives little attention”, and was missing entirely from 60% of machine learning models in one systematic review.

An impact study asks whether providing clinicians with the predictions changes decisions and outcomes. This is a different question from whether the model is accurate, and it “requires a comparative study” with a usual-care arm. It is the step almost never taken, which is why an accurate model and a useful model are not equivalent claims.

There is no single number, and the paper usually cited for one argues against it. Riley and colleagues state that “this reinforces why rules of thumb (eg, 10 EPP) should be avoided”, and their own worked examples range from approximately 5 to approximately 23 events per predictor parameter, depending on the setting. The relevant unit is predictor parameters, not of variables, should be counted.

On the available evidence, most are not. A systematic review of 152 studies of machine learning prediction models judged 148 of 171 analyses (87%) at high risk of bias, with the risk concentrated in the analysis domain. A review of COVID-19 prediction models found 545 of 606 prognostic models (90%) at high risk of bias, and warned that their performance was “probably optimistic”.

Strictly, it conveys nothing on its own. TRIPOD+AI states that “there is no such thing as a validated prediction model” and deliberately replaces “validation” with “evaluation”. Performance is always performance in a particular population. A model described as validated, without a statement of where and in whom it was evaluated, provides no usable information.

References

  1. Collins GS, Moons KGM, Dhiman P, Riley RD, Beam AL, Van Calster B, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378 read 15 October 2026. View on a university repository copyThe current reporting guideline for prediction model studies: 27 items containing 52 subitems, covering regression and machine learning alike. Read from LMU Munich's repository copy because the publisher's own page was not reachable.
  2. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Annals of Internal Medicine. 2019;170(1):51–58. doi:10.7326/M18-1376 read 15 October 2026. View on University of Bristol research portalFour domains and 20 signalling questions. Verified from the institutional record; the journal's full text was not reachable, so nothing is quoted from the body.
  3. Moons KGM, Damen JAA, Kaul T, Hooft L, Andaur Navarro C, van Smeden M, et al. PROBAST+AI: an updated quality, risk of bias, and applicability assessment tool for prediction models using regression or artificial intelligence methods. BMJ. 2025;388:e082505. doi:10.1136/bmj-2024-082505 read 15 October 2026. View on Oxford University Research ArchiveStructurally different from PROBAST rather than a relabel: two parts, separate question sets for development and evaluation, and a renamed first domain. Verified from the institutional record only.
  4. Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683. doi:10.1136/heartjnl-2011-301246 read 15 October 2026. View on Heart (BMJ)The first of a two-part overview, on model development and internal validation. Read on the publisher's own site.
  5. Moons KGM, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M. Risk prediction models: II. External validation, model updating, and impact assessment. Heart. 2012;98(9):691. doi:10.1136/heartjnl-2011-301247 read 15 October 2026. View on a university-hosted copyThe second part, on external validation and impact assessment. The source for the point that an impact study needs a comparative design. Read from a university mirror.
  6. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17(1):230. doi:10.1186/s12916-019-1466-7 read 15 October 2026. View on DOAJ recordOn the neglect of calibration. The publisher's full text was not reachable, so only the abstract is quoted and nothing is attributed to the body of the paper.
  7. Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE Jr, Moons KGM, Collins GS. Minimum sample size for developing a multivariable prediction model: PART II — binary and time-to-event outcomes. Statistics in Medicine. 2019;38(7):1276–1296. doi:10.1002/sim.7992 read 15 October 2026. View on Oxford University Research ArchiveCited for the sample-size criteria and, importantly, for arguing against the rule of thumb usually attributed to this literature. Published online 2018 and in issue in 2019; verified from the institutional record.
  8. Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi:10.1136/bmj.m1328 (living review; final update, searches to 17 February 2021.) read 15 October 2026. View on The BMJA living systematic review, so the figures belong to the final update rather than to the 2020 first publication. Cited with the search date for that reason. Read on the publisher's own site.
  9. Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, Collins GS, Bajpai R, Riley RD, Moons KGM, Hooft L. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review. BMJ. 2021;375:n2281. doi:10.1136/bmj.n2281 read 15 October 2026. View on a university repository copyThe strongest single citation for this page: it measures, rather than asserts, how often machine-learning prediction studies omit calibration, mishandle overfitting and run on too few events. Read from Keele's repository.
  10. Ramspek CL, Steyerberg EW, Riley RD, Rosendaal FR, Dekkers OM, Dekker FW, van Diepen M. Prediction or causality? A scoping review of their conflation within current observational research. European Journal of Epidemiology. 2021. doi:10.1007/s10654-021-00794-w read 15 October 2026. View on a university repository copyMeasures how often prediction and causation are conflated in published research: 26% of 180 studies overall, and 38% of the prediction studies. Read from Keele's repository; volume and page numbers were not shown on that record.

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

Related: biostatistics in clinical research · concepts that get misstated · choosing a study design.

How Pubrica can help

Pubrica’s biostatisticians and data scientists develop and evaluate clinical prediction models, with sample size derived from the setting rather than a rule of thumb, joint reporting of discrimination and calibration, external evaluation designed as a separate study, and reporting in accordance with TRIPOD+AI. Appraisal of an existing model before further development can be undertaken as a discrete piece of work.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.