Home/Insights/Evidence synthesis/Machine learning for screening in systematic reviews
Machine learning for screening in systematic reviews
Automation can cut screening workload substantially. The question is not whether it is accurate but what recall you are willing to accept, because a missed study is invisible.
Where it is used
The highest-volume task in a systematic review is title and abstract screening, often tens of thousands of records for a few dozen includes. Machine learning is applied there, and to a lesser extent to full-text screening, data extraction and risk of bias assessment.
The common approach is active learning: a human screens a sample, the model learns from those decisions, ranks the remainder by likely relevance, and the human continues down a list ordered so that includes appear early.
Tools in this space include Abstrackr, Rayyan, ASReview and RobotReviewer, among others.
The metric that matters is recall
Precision determines how much time is wasted; recall — the proportion of truly relevant records identified — determines whether the review is complete.
A missed eligible study is undetectable from the finished review. Nobody reading it can tell. This asymmetry is why recall thresholds for review screening are set very high, commonly at or above 95%, and why a tool’s reported accuracy should always be read as a recall figure rather than an overall one.
Reported performance also varies by topic: models do better on clinically well-defined questions than on broad, conceptually fuzzy ones.
How it is used responsibly
As a second screener rather than a replacement for one, so the human decision remains and the model catches what the human missed.
As a prioritiser, ordering records so screening can stop when the yield of new includes has plateaued — with the stopping rule stated in advance.
With a validation set: screen a random sample manually and measure what the model would have missed.
Never as an unchecked filter on an unvalidated model.
Reporting it
PRISMA 2020 explicitly requires that automation tools used in screening or extraction be reported, including which tool, at which stage, and how its decisions were verified. A review that used automation and did not say so is non-compliant.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Judge tools by recall, not overall accuracy.
- Use as a second screener or prioritiser, with a stated stopping rule.
- Report: the tool, the stage and the verification — PRISMA 2020 requires it.
Sources
- PRISMA 2020, on reporting automation tools
- Cochrane guidance on technology-assisted screening
- Published evaluations of Abstrackr, Rayyan, ASReview and RobotReviewer
Sources are given so each claim can be checked at its origin. Where a tool’s current maintenance status could not be confirmed, the page says to check it rather than asserting it.
Related