Contents
Academy · Research Data, Analytics & AI · Study Selection Process
How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening
A crowd-based randomised trial on the Cochrane Crowd platform compared a single reviewer screening abstracts alone with two reviewers working independently. The comparison found different error rates between the evaluated screening approaches. The difference arises from how selection is structured rather than from the care exercised by any individual reader. This article examines that evidence and its implications for the design of the selection process.
Study selection is the stage of a systematic review at which the records retrieved by the search are screened against pre-defined eligibility criteria, first by title and abstract and then by full text, to determine which studies are included in the review. It is distinct from searching: a comprehensive search may identify every eligible study, yet a flawed selection process may still exclude a meaningful proportion of them before they reach data extraction. The strongest direct evidence on how selection should be structured comes from a randomised trial that compared single-reviewer screening with dual, independent screening on the same set of real abstracts and found a measurable, reproducible difference between the two.
Learning objectives
- The measured difference in the proportion of missed studies between single-reviewer and dual-reviewer screening
- Why independent screening, rather than joint screening by two people, is the feature responsible for the difference
- The role of machine-learning and text-mining tools in study selection without replacement of the dual-screening standard
Common misconceptions
- “A single careful and experienced reviewer can screen abstracts as reliably as two reviewers.” In a randomised trial conducted on the Cochrane Crowd platform, 280 participants made 24,942 screening decisions across 2,000 real abstracts, comparing single-reviewer screening with screening by two independent reviewers. The comparison found different error rates between the evaluated screening approaches.
Randomised evidence for dual independent screening
Gerald Gartlehner and colleagues conducted a parallel-group randomised controlled trial on the Cochrane Crowd platform specifically to measure the loss of accuracy associated with single-reviewer screening relative to the conventional two-reviewer standard1. The trial used 2,000 abstracts drawn at random from two previously published systematic reviews, in which the true inclusion status of every abstract was already known, and 280 volunteer screeners made a combined total of 24,942 screening decisions, with each abstract screened approximately 12 times by different participants1. Averaged across all screeners, the trial measured the frequency with which single-reviewer screening incorrectly excluded genuinely eligible studies1. The trial also simulated dual, independent screening by pairing the decisions of two reviewers and including any abstract that either reviewer flagged as potentially eligible; the comparison found different error rates between the evaluated screening approaches1. Because this was a randomised trial conducted against abstracts with a known, verified reference standard, the single-reviewer error rate is not an estimate of theoretical risk but a measured error rate derived from actual screening behaviour on real systematic review content.
Independence of reviewers as the determinant of screening accuracy
The mechanism underlying the difference is straightforward: screening errors made by different reviewers are not perfectly correlated, so a study incorrectly excluded by one reviewer is often correctly included by another, and requiring only one of two independent reviewers to flag a study as potentially eligible captures studies that either reviewer alone would have missed1. For this reason, the Cochrane Handbook and the Methodological Expectations of Cochrane Intervention Reviews (MECIR) standards require that at least two people independently screen titles and abstracts against the eligibility criteria of the review, with disagreements resolved by discussion or by a third reviewer, rather than regarding a single well-trained reviewer as sufficient36. A broader methodological systematic review by Waffenschmidt and colleagues, which examined the existing evidence on single versus double screening across multiple studies, likewise concluded that single screening carries a meaningfully higher risk of missing eligible studies than the conventional dual approach, indicating that the result of the Gartlehner trial is not an isolated finding4. An earlier study by Edwards and colleagues, which examined the accuracy and reliability of screening records for randomised controlled trials specifically, similarly found that screening performance varied sufficiently between individual reviewers that reliance on a single screener introduced an unpredictable, reviewer-specific error rate, which dual screening is structurally designed to average out5.
Full-text screening and resolution of reviewer disagreement
The same dual, independent principle applies at the full-text stage, at which records that passed title-and-abstract screening are checked in full against every eligibility criterion, not only those apparent from an abstract. Disagreements between reviewers at either stage should be resolved through discussion and, where discussion does not produce consensus, by a third reviewer or by an adjudication rule specified before screening began, rather than by whichever reviewer argues more persistently at the time3. The PRISMA 2020 flow diagram requires authors to record the reason for exclusion of every record screened out at the full-text stage and to report the number of records excluded for each reason, specifically so that a reader can assess whether the stated eligibility criteria were applied consistently rather than selectively2. Measuring and reporting inter-rater agreement between the two independent screeners, commonly with a kappa statistic, provides readers with a concrete, verifiable measure of how consistently the eligibility criteria of the review were applied in practice, rather than leaving the reviewers simply to assert that their screening was reliable.
The role of automation tools alongside dual screening
Text-mining and machine-learning tools that rank or pre-screen records by predicted relevance have a measured and useful role in systematic review selection, principally in reducing the volume of records that a human reviewer must read manually. However, a systematic review of these approaches by O’Mara-Eves and colleagues found that performance varied considerably across tools and review topics, and that none of the methods reviewed could be relied upon to replace human screening entirely without a documented validation step for the specific review in question7. The review of text-mining applications in systematic reviews by Thomas, McNaught and Ananiadou similarly characterises these tools as aids to workload reduction, for example by prioritising the order in which records are screened or by flagging probable duplicates, rather than as a substitute for the second independent human reviewer required by the Cochrane and MECIR standards8. A review team that uses a machine-learning classifier to assist screening should disclose precisely how it was used, trained and validated, since an unreported or unvalidated automation step is no more auditable than an undocumented single-reviewer screening process.
Piloting eligibility criteria before full-scale screening
A step that precedes full-scale screening, and that materially affects how much the dual, independent screening process described above detects, is the piloting of the eligibility criteria on a small sample of records before both reviewers commit to screening the full set. The Cochrane Handbook recommends that reviewers screen an initial pilot set of records together, compare their independent decisions and discuss any disagreements in order to clarify ambiguous or poorly worded criteria before screening proceeds at scale, because a genuinely ambiguous eligibility criterion will generate disagreements at a rate that screener diligence alone cannot correct3. This pilot step exists specifically to distinguish two sources of screening disagreement that appear identical in the raw disagreement count but require different remedies. The first is disagreement caused by one reviewer overlooking information in an otherwise clear record, which dual, independent screening is designed to detect; the second is disagreement caused by eligibility criteria that are written ambiguously enough for two careful reviewers to reach different conclusions reasonably from the same information, which additional screening effort cannot resolve unless the criteria are rewritten.
The trial by Gartlehner and colleagues measured screening performance using eligibility criteria from reviews that had already completed the piloting and refinement stage as part of their normal conduct. The single-reviewer and dual-reviewer miss rates reported in that trial therefore probably approximate a best-case scenario for the clarity of the criteria, rather than screening against an ambiguous or unpiloted protocol1. A review that omits piloting and proceeds directly to full-scale dual screening against a first draft of the eligibility criteria should anticipate a higher disagreement rate than the benchmark figures of the trial, not because the screeners perform less well, but because the ambiguities that a pilot stage is designed to identify have not yet been found and removed from the criteria. Allocating time for this pilot-and-refine step before full-scale screening begins is therefore not a procedural formality; it directly determines how closely the actual screening performance of the review can approach the dual-screening benchmark established by the evidence base. A practical rule applied by many review teams is to pilot against at least 50 to 100 records before judging the criteria sufficiently stable for full-scale screening, revising the written eligibility criteria and repeating the pilot on a new sample whenever the initial pilot yields a disagreement rate high enough to suggest that the criteria themselves, rather than the screeners, are the source of the problem. This modest initial investment is consistently less costly than discovering the same ambiguity midway through full-scale screening of several thousand records, at which point every record already screened may need to be re-examined against the revised criteria, a cost that Shemilt and colleagues specifically identified in their comparison of study identification methods when weighing the efficiency of different screening approaches against their resource requirements9. The PRISMA 2020 explanation and elaboration document provides worked examples of how the eligibility criteria, the screening process and any pilot-and-refine step should be reported in the methods section, and it is the practical reference to consult when this part of a review is written up for publication10.
Key points
Key points
Dual independent screening at both the title-and-abstract and full-text stages is not a procedural formality; a randomised trial comparing it with a single reviewer working alone found different error rates between the two approaches.
Practical application
Practical application
- Titles, abstracts and full texts should be screened by at least two independent reviewers. Each reviewer should screen without first seeing the other’s decisions, and any record flagged as potentially eligible by either reviewer should proceed to the next stage.
- The method for resolving disagreements should be pre-specified before screening begins. Reviewers should decide in advance whether disagreements will be resolved by discussion, by a third reviewer or by a fixed adjudication rule, rather than devising a resolution process during the review.
- Exclusion reasons and inter-rater agreement should be reported. The reason for excluding each full-text record should be recorded and disclosed, and a measure of inter-rater agreement such as kappa should be calculated so that the consistency of screening can be independently verified.
Frequently asked questions
The evidence base indicates that it carries a meaningfully higher risk of missing eligible studies; where resource constraints make dual screening genuinely impossible, this should be disclosed as a limitation rather than presented as equivalent to the dual-reviewer standard.
The Cochrane Handbook and the MECIR standards for Cochrane reviews specify independent dual screening of all records at the title-and-abstract and full-text stages, not of a sampled subset, because a sampled check cannot detect an error in a record that the second reviewer never examined.
Authors should report the number of disagreements that occurred at each screening stage and the method by which they were resolved (discussion, a third reviewer or a pre-specified rule), since this information enables a reader to judge how rigorously the eligibility criteria were applied.
The reviewed evidence on these tools shows variable performance across topics and no validated basis for replacing a required independent human screener entirely; they are best used to reduce screening workload, with their use and validation fully disclosed.
Title and abstract screening is a first-pass filter based on limited information and is deliberately permissive in order to avoid losing eligible studies; full-text screening applies every eligibility criterion to the complete article and produces the final inclusion decision.
References
- Gartlehner G, Affengruber L, Titscher V, Noel-Storr A, Dooley G, Ballarini N, König F. Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial. Journal of Clinical Epidemiology. 2020;121:20-28. read 10 October 2026. View on Journal of Clinical Epidemiology
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on The BMJ
- Higgins JPT, Thomas J, Chandler J, et al (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.4, Chapter 4: Searching for and selecting studies. Cochrane, 2023. read 10 October 2026. View on Cochrane Training
- Waffenschmidt S, Knelangen M, Sieben W, Bühn S, Pieper D. Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology. 2019;19:132. read 10 October 2026. View on BMC Medical Research Methodology
- Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine. 2002;21(11):1635-1640. read 10 October 2026. View on Statistics in Medicine
- Higgins JPT, Lasserson T, Chandler J, Tovey D, Thomas J, Flemyng E, Churchill R. Methodological Expectations of Cochrane Intervention Reviews (MECIR). Cochrane, 2023. read 10 October 2026. View on Cochrane Methods
- O'Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews. 2015;4:5. read 10 October 2026. View on Systematic Reviews
- Thomas J, McNaught J, Ananiadou S. Applications of text mining within systematic reviews. Research Synthesis Methods. 2011;2(1):1-14. read 10 October 2026. View on Research Synthesis Methods
- Shemilt I, Khan N, Park S, Thomas J. Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews. 2016;5:140. read 10 October 2026. View on Systematic Reviews
- Page MJ, Moher D, Bossuyt PM, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372:n160. read 10 October 2026. View on The BMJ
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening. Pubrica. https://pubrica.com/academy/selection-process-in-systematic-review/
Pubrica Academy. “How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening.” Pubrica, 10 Oct. 2026, pubrica.com/academy/selection-process-in-systematic-review/.
Pubrica Academy (2026) How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening. Pubrica. Available at: https://pubrica.com/academy/selection-process-in-systematic-review/ (Accessed: 15 October 2026).
Pubrica Academy. “How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening.” Pubrica. October 10, 2026. https://pubrica.com/academy/selection-process-in-systematic-review/
Pubrica Academy, ‘How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/selection-process-in-systematic-review/> accessed 15 October 2026.
Pubrica Academy. How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/selection-process-in-systematic-review/
Pubrica Academy, “How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/selection-process-in-systematic-review/
@misc{/la_selection_process_systematic_review_meta_analysis_2026,
author = {{Pubrica Academy}},
title = {How the Study Selection Process Should Be Conducted in a Systematic Review or Meta-Analysis: Evidence for Dual Independent Screening},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/selection-process-in-systematic-review/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Dual Screening Protocol Worksheet
Set this up before screening begins so independence and disagreement handling are fixed in advance.
[ ] Confirmed two reviewers will independently screen every title/abstract (not a sample) [ ] Confirmed two reviewers will independently screen every full text [ ] Pre-specified how disagreements are resolved (discussion / third reviewer / fixed rule) [ ] Set up a log to record the reason for every full-text exclusion [ ] Planned to calculate and report inter-rater agreement (e.g., kappa) [ ] Documented any automation/text-mining tool used, including how it was validated
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s systematic review team conducts dual independent screening with documented adjudication and reporting of inter-rater agreement, so that the study selection stage meets the scrutiny expected under PRISMA and Cochrane standards.