Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · CHARMS & PROBAST

Data Extraction and Risk-of-Bias Assessment in Prediction Model Reviews: CHARMS and PROBAST

The CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) structures what is extracted from a prediction model study, and the Prediction model Risk Of Bias ASsessment Tool (PROBAST) structures the judgement of whether model development or validation was biased. Systematic reviews of prediction models require both tools to be used together.

Pubrica AcademyWritten by a prediction model review methodologist Reviewed bya clinical epidemiology reviewer Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources10 peer-reviewed

CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) and PROBAST (Prediction model Risk Of Bias ASsessment Tool) are two companion tools used together in systematic reviews of studies that develop or validate clinical prediction models. CHARMS provides a structured checklist for what data to extract from each included study, covering source data, predictors, outcomes, and model-building methods, while PROBAST provides a structured framework for judging the risk of bias and applicability of each model, addressing concerns distinct from those in reviews of interventions or etiology.

Learning objectives

  • What CHARMS extracts that a generic data-extraction form would miss in a prediction model review
  • The four domains PROBAST uses to assess risk of bias in prediction model studies
  • How CHARMS and PROBAST relate to the separate TRIPOD reporting standard for prediction models

Common misconceptions

  • “A risk-of-bias tool built for randomised trials or etiology studies can be applied to prediction model research.” Prediction model studies face validity threats (overfitting, inappropriate predictor selection and lack of external validation) that tools such as RoB 2 or Newcastle-Ottawa do not address; PROBAST was developed specifically because reviewers needed a framework built around how prediction models are developed and validated.

Purpose of CHARMS and PROBAST in prediction model reviews

CHARMS and PROBAST exist because systematic reviews of prediction model studies ask a fundamentally different question from reviews of interventions or etiology: not whether a treatment works or an exposure causes harm, but whether a model's predicted risk or diagnosis is accurate and generalisable enough to inform clinical decisions12. That question depends on details, such as how predictors were selected, how missing data was handled during model development, and whether the model was tested on data independent from its development sample, that neither a standard data-extraction template nor a trial-oriented risk-of-bias tool is built to capture systematically1.

Prediction model studies are also of two distinct types that CHARMS and PROBAST both need to distinguish: development studies, which build a new model from a dataset, and validation studies, which test an already-developed model's performance on a different dataset. A systematic review that does not separate these two study types risks treating a model's unvalidated, in-sample performance figures as equivalent to its performance when tested on independent data, which is precisely the kind of conflation CHARMS's extraction fields and PROBAST's analysis domain are designed to prevent.

Data extraction domains structured by CHARMS

CHARMS organises data extraction into eleven domains covering the source of data, participants, outcomes to be predicted, candidate predictors, sample size, missing data handling, the model development method itself, model performance measures, and, where reported, results of model evaluation1. This level of structure matters because a prediction model's reported accuracy figures are only interpretable alongside these contextual details; a model's discrimination statistic means something different depending on whether predictors were selected using the full dataset without correction for overfitting or using a disciplined, pre-specified approach, which CHARMS extraction is designed to surface explicitly rather than leaving the reviewer to infer it from the text.

The sample size domain deserves particular attention during extraction, since a model developed on a small dataset relative to the number of candidate predictors considered is prone to overfitting, fitting patterns specific to that dataset rather than a generalisable relationship, and will typically perform worse when applied to new data than its original reported statistics suggest. CHARMS extraction captures the raw sample size and number of outcome events alongside the number of candidate predictors considered, which together allow a reviewer to judge this risk directly rather than relying on the study authors' own characterisation of their model as adequately powered.

Interpretation of the performance measures captured by CHARMS

CHARMS extraction fields distinguish between a model's discrimination, how well it separates people who will go on to have the outcome from those who will not, typically summarised through a statistic such as the area under the receiver operating characteristic curve (AUC) or C-statistic, and its calibration, how closely its predicted probabilities match observed outcome frequencies across the range of risk the model produces. A model can discriminate well while calibrating poorly, correctly ranking patients from lower to higher risk while still systematically over- or under-predicting the actual probability at any given point on that ranking, which is why extracting both measures separately, rather than a single combined performance figure, is necessary for a reviewer to judge whether a model is likely to be clinically useful as a risk-prediction tool rather than only as a relative ranking.

The four PROBAST domains for assessing risk of bias

PROBAST structures risk-of-bias assessment around four domains: participant selection, predictor definition and measurement, outcome definition and measurement, and the statistical analysis used to develop or validate the model, with each domain assessed through a set of signalling questions that lead to an overall domain judgment3. The analysis domain is often where the most serious bias is found, covering issues such as whether continuous predictors were inappropriately dichotomised, whether the model was validated on the same data used to build it rather than an independent sample, and whether performance was reported selectively3. PROBAST also includes a parallel applicability assessment, judging whether the population, predictors, and outcome in the study match the review's intended use case, a distinct question from risk of bias itself3.

A model can score well on risk of bias while still scoring poorly on applicability, and the two ratings should be reported and interpreted separately rather than combined into one overall judgment. A model developed in a narrow hospital population using rigorous, well-documented methods might be entirely low risk of bias on its own terms, yet poorly applicable to a reviewer's intended use case if that use case involves a broader primary-care population the original study never sampled from; PROBAST's structure is built specifically to let a reviewer flag this mismatch explicitly rather than letting a low-bias rating stand in for suitability.

Combined use of CHARMS and PROBAST alongside TRIPOD

In practice, CHARMS extraction is typically completed first, since the structured fields it captures, particularly around analysis methods and validation approach, supply the information PROBAST's signalling questions need to answer13. Both tools complement, rather than duplicate, TRIPOD, the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement, which sets reporting standards for authors publishing a prediction model study in the first place; CHARMS and PROBAST are applied by reviewers appraising already-published studies, some of which may predate or only partially follow TRIPOD4. A broader methodological guide to prognostic factor and prediction model reviews situates both tools within the fuller review process, from question framing through synthesis5.

A practical consequence of a published study predating or only partially following TRIPOD is that its reporting may be too incomplete for a reviewer to answer some of PROBAST's signalling questions with confidence. In such cases, the appropriate response is to record the relevant domain as having unclear, rather than low or high, risk of bias, and to state this explicitly in the review rather than inferring an answer the original publication does not support. A systematic review that reports a meaningful proportion of included studies as unclear risk of bias in one or more domains is not a weaker review for doing so; it is a more accurate one, since it reflects a genuine, documented limitation in the underlying literature rather than obscuring it with an unsupported judgment. Reporting this pattern explicitly also benefits the field: a review that identifies which specific aspects of model development and validation are consistently under-reported gives future authors of prediction model studies a concrete, evidence-based reason to follow TRIPOD more completely, closing the reporting gap the review itself had to address. Teams new to CHARMS and PROBAST often find it useful to pilot both tools on two or three included studies before extracting the full set, because PROBAST's signalling questions are easier to answer confidently once a reviewer has observed how much (or how little) detail typical studies in the field report. Independent extraction and appraisal of this pilot batch by two reviewers, followed by comparison and discussion of any disagreements before proceeding to the full set, further supports consistent judgment across the team on the inevitably subjective elements of both tools' signalling questions.

Key points

Key points

Reviewing prediction model studies requires CHARMS to structure what gets extracted and PROBAST to structure the resulting risk-of-bias judgment, because neither a generic extraction form nor a trial-oriented bias tool captures the validity threats specific to model development and validation.

Practical application

Practical application

  1. CHARMS should be used as the extraction template from the first included study onwards. All eleven CHARMS domains should be captured systematically, rather than ad hoc fields being extracted and later retrofitted to CHARMS.
  2. The four PROBAST domains should be applied individually rather than combined into one overall score. Participant selection, predictor and outcome measurement, and analysis should be judged separately, since a model may be well handled in one domain and biased in another.
  3. Applicability should be assessed separately from risk of bias. The PROBAST applicability assessment addresses fit with the review’s intended population and use case, which is a question distinct from whether the original study itself was biased.

Frequently asked questions

No. It is built specifically around how prediction models are developed and validated; reviews of interventions should use RoB 2, and reviews of etiological observational studies should use tools such as ROBINS-I or the Newcastle-Ottawa Scale instead.

It is a data-extraction and critical-appraisal checklist, not a formal risk-of-bias scoring tool; PROBAST is the companion tool specifically built to produce a structured risk-of-bias judgment.

Not automatically, but PROBAST's analysis domain does specifically probe whether a model was validated on independent data, and a model evaluated only on its own development sample is a recognised risk factor PROBAST is designed to flag.

TRIPOD is a reporting standard for authors publishing a new prediction model study; CHARMS and PROBAST are applied afterward by systematic reviewers appraising a body of such studies, some published before TRIPOD existed or without full TRIPOD compliance.

CHARMS extraction is typically completed first, since the structured details it captures about analysis methods and validation approach are needed to answer PROBAST's signalling questions accurately.

References

  1. Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist. PLOS Medicine. 2014;11(10):e1001744. read 10 October 2026. View on PLOS Medicine
  2. Wolff RF, Moons KGM, Riley RD, Whiting PF, Westwood M, Collins GS, Reitsma JB, Kleijnen J, Mallett S; PROBAST Group. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Annals of Internal Medicine. 2019;170(1):51-58. read 10 October 2026. View on Annals of Internal Medicine
  3. Moons KGM, Wolff RF, Riley RD, et al. PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration. Annals of Internal Medicine. 2019;170(1):W1-W33. read 10 October 2026. View on Annals of Internal Medicine
  4. Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). Annals of Internal Medicine. 2015;162(1):55-63. read 10 October 2026. View on Annals of Internal Medicine
  5. Riley RD, Moons KGM, Snell KIE, et al. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019;364:k4597. read 10 October 2026. View on The BMJ
  6. Debray TPA, Damen JAAG, Snell KIE, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356:i6460. read 10 October 2026. View on The BMJ
  7. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on The BMJ
  8. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. read 10 October 2026. View on The BMJ
  9. Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924-926. read 10 October 2026. View on The BMJ
  10. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews. BMJ. 2017;358:j4008. read 10 October 2026. View on The BMJ

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Data Extraction and Risk-of-Bias Assessment in Prediction Model Reviews: CHARMS and PROBAST. Pubrica. https://pubrica.com/academy/systematic-review/data-extraction-bias-assessment-charms-probast/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

CHARMS/PROBAST Review Worksheet

This checklist is intended for use when extracting data and assessing bias in a prediction model systematic review.

[ ] All eleven CHARMS domains extracted per included study
[ ] PROBAST participant selection domain assessed with signalling questions
[ ] PROBAST predictor and outcome measurement domains assessed separately
[ ] PROBAST analysis domain assessed, including validation approach and predictor handling
[ ] PROBAST applicability judged separately from risk of bias
[ ] Cross-checked extracted methods against TRIPOD items where the original study reports them

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica's systematic review team applies CHARMS extraction and PROBAST bias assessment together on prediction model reviews, so performance claims are interpreted against how each model was developed and validated.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Support for risk-of-bias assessment with tools designed for prediction models

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.