Contents
On this page
- Limits of single-database searching
- Language restriction as a documented source of bias
- The grey-literature gap and its consequences
- PRISMA-S: a dedicated extension for reporting the search
- Citation searching and snowballing as a complementary technique
- Empirical evidence underlying these recommendations
- Mitigation of search bias in practice
Academy · Research Data, Analytics & AI · Search Bias
Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature
A cross-sectional study of Cochrane reviews found that reliance on a single database such as PubMed would have missed a meaningful proportion of the studies those reviews included. This finding indicates that database-coverage decisions are themselves a source of bias, operating before any study is assessed for quality.
Search bias in a systematic review is a systematic distortion in which studies end up included, introduced by the design of the literature search itself rather than by anything about the studies' content or quality. It can come from restricting a search to too few databases, from excluding non-English-language studies, from omitting grey literature such as conference abstracts and dissertations, or from search terms that happen to capture one strand of a field's terminology while missing another. Because search bias operates before the screening and quality-assessment stages begin, a systematic review can apply rigorous inclusion criteria and risk-of-bias assessment to every study it finds and still produce a biased overall picture, because the body of studies it searched was not representative of the full literature on the question.
Learning objectives
- Documented evidence that reliance on a single database measurably misses studies included in other reviews
- Why language restrictions in a search are a specific, researched source of bias rather than a practical shortcut
- How grey literature and database-selection decisions together determine what a review is able to identify
Common misconceptions
- “Searching one comprehensive database like PubMed or Scopus is generally sufficient for a rigorous systematic review.” A cross-sectional study of the databases underlying a sample of Cochrane systematic reviews found that relying on PubMed alone, without also searching Embase and other databases the reviews used, would have missed a meaningful proportion of the studies those reviews ultimately included; this provides concrete evidence that single-database searches carry a measurable risk of under-coverage rather than a merely theoretical one.
Limits of single-database searching
Different bibliographic databases index different, only partially overlapping sets of journals, and no single database covers the full literature on any given biomedical or social-science question. A cross-sectional study examining the contribution of individual databases to the results of a sample of systematic reviews found that relying on only the most commonly used database would have led reviewers to miss a meaningful share of the studies those reviews ultimately included once multiple databases were searched, a direct, measured demonstration that database choice is itself a source of potential bias rather than an interchangeable detail of search methodology1. This is part of why major reporting guidelines, including PRISMA 2020, specifically require authors to report every database and source searched, along with the exact search strategy for at least one database in full, so that readers can judge how comprehensive the underlying search was2.
Language restriction as a documented source of bias
Restricting a search to studies published in English, or to languages the review team can read, is a common practical shortcut, but it is also a documented source of bias in its own right. Research examining the practical meaning of “no language restrictions” in database searches has found that even when reviewers state they imposed no language restriction, the specific databases searched may themselves have incomplete indexing of non-English-language journals, meaning the effective language coverage of a search can be narrower than its stated inclusion criteria suggest3. The direction and size of language bias is not consistent across fields; studies in some clinical areas have found that excluding non-English-language studies changes pooled effect estimates only modestly, while in other areas the excluded studies carry systematically different findings, which is itself a reason reviewers are advised to check rather than assume language restriction is harmless in their specific field3.
The grey-literature gap and its consequences
Grey literature (conference abstracts, dissertations, preprints, and unpublished or industry reports) sits outside the indexing most bibliographic databases cover, and omitting a grey-literature search compounds the database-coverage problem described above rather than operating independently of it. Because grey literature often includes studies with null or negative findings that never progressed to full journal publication, a search strategy that excludes it risks converting a database-coverage bias into something closer to a publication-bias problem, systematically missing the kind of unpublished or less-visible findings that would pull a pooled estimate back toward a more accurate value. The Cochrane Handbook treats grey-literature searching, alongside trial registry searches, as a standard component of a comprehensive search strategy for this reason, not as an optional extension reserved for reviews with unusually generous resources4.
PRISMA-S: a dedicated extension for reporting the search
Because the search strategy itself is such a significant source of potential bias, PRISMA maintains its own dedicated extension, PRISMA-S, specifically for reporting literature search methods in full. Rethlefsen and colleagues published PRISMA-S in 2021, developed with input from information specialists and librarians, providing a 16-item checklist covering exactly which databases and platforms were searched, the full search strategies used (not a paraphrased summary), the dates searches were run, any limits or filters applied, methods used to search grey literature and trial registries, and whether an information specialist was involved in designing or peer-reviewing the search5. The existence of a dedicated extension at this level of detail reflects a specific, researched concern: that search reporting in published reviews had historically been vague enough that neither a reader nor a future reviewer attempting to update the review could reliably tell what had been searched, which undermines a reader's ability to judge, and a future team's ability to replicate, the review's search coverage.
Citation searching and snowballing as a complementary technique
Because no database search strategy, however well designed, catches every relevant study, methodologists recommend supplementing database searching with citation-based techniques: checking the reference lists of all included studies for additional relevant citations (“backward” snowballing), and checking which later papers have cited each included study for further relevant work (“forward” snowballing), alongside directly searching the reference lists of any closely related systematic reviews found during the search. Greenhalgh and Peacock's analysis of how included studies were found in a large systematic review on complex evidence found that a substantial proportion of the studies ultimately included had been located through these citation-tracking and informal methods rather than through the structured database search alone, which is direct evidence that database searching, even when well executed, is not sufficient by itself for comprehensive coverage in every field7. This is a separate mitigation from the database-coverage and grey-literature steps already described, and reviews aiming for comprehensive coverage are generally advised to use both approaches together rather than treating either as sufficient on its own.
Empirical evidence underlying these recommendations
These recommendations are not merely methodological convention; they rest on a specific empirical literature examining how much comprehensive searching changes a systematic review's conclusions. Egger and colleagues' empirical study for the UK's Health Technology Assessment programme directly tested how comprehensive literature searches and formal trial-quality assessment affected the conclusions of a sample of meta-analyses, finding that less comprehensive searches and omission of quality assessment could each, in specific cases, meaningfully change a review's pooled result, which is part of the evidence base behind treating search comprehensiveness as a determinant of a review's reliability rather than only its thoroughness9. The Cochrane Database of Systematic Reviews has itself published a methodology review specifically on the value of checking reference lists as an additional source of studies, by Horsley and colleagues, concluding that this technique does meaningfully contribute additional eligible studies beyond database searching in a non-trivial proportion of reviews examined, reinforcing that citation-based and database-based search methods are complementary rather than redundant with one another8. Taken together with the Cochrane Handbook's own chapter on searching and selecting studies, authored and maintained specifically by information-retrieval methodologists within the Cochrane Collaboration, this body of evidence is why comprehensive, multi-method searching is treated as a core methodological requirement rather than an optional refinement reserved for reviews with unusually generous resources10.
Mitigation of search bias in practice
Reporting guidance and methodological literature converge on a consistent set of mitigations: search at least two, and typically more, databases with different indexing coverage (commonly PubMed/MEDLINE together with Embase, given their documented partial overlap), avoid unjustified language restrictions or explicitly test their effect through a sensitivity analysis where restriction is unavoidable, include a grey-literature and trial-registry search as a standard step rather than an optional one, and have the full search strategy for at least one database reviewed by a second person, often a trained information specialist or librarian, before it is run24. PRISMA 2020's reporting requirement to publish the complete search strategy, rather than a general description of it, exists specifically so that these choices are visible and checkable by readers and future reviewers, rather than inside a brief methods paragraph2. None of these mitigations is a one-time cost absorbed only at a review's outset; a comprehensive, well-documented search strategy also makes a review far easier for a future team to update as new evidence accumulates, which is itself one of the stated goals of maintaining rigorous, reported search methods over time.
Key points
Key points
Search bias arises from the design of the literature search itself — database selection, language restriction and grey-literature omission — and operates before screening begins. A documented cross-sectional study found that reliance on one database measurably missed studies identified by a fuller, multi-database search; comprehensive and fully reported search strategies are therefore a methodological requirement rather than a convenience.
Practical application
Practical application
- At least two databases with documented, differing coverage should be searched. PubMed/MEDLINE and Embase have documented partial overlap; searching both, rather than either alone, directly addresses the measured under-coverage risk of single-database searches.
- Any language restriction should be justified, or its effect tested. Where restriction by language is necessary for resource reasons, it should be stated explicitly as a limitation and, where feasible, a sensitivity analysis should examine whether studies in excluded languages would plausibly change the pooled estimate.
- Grey-literature and trial-registry searches should be included as a standard step. Conference abstracts, dissertations and registry searches should be treated as part of the core search strategy, not as an optional addition reserved for reviews with additional time or funding.
Frequently asked questions
Documented research on database contribution to systematic review results indicates that relying on PubMed alone carries a measurable risk of missing studies that a broader, multi-database search would find, so it is generally not considered sufficient on its own for a rigorous systematic review, even though it remains an essential starting point.
No. Research on language restriction in systematic reviews has found the direction and size of this effect varies by field and question, which is why reviewers are advised to assess its likely impact for their specific topic rather than assume it is uniformly negligible or uniformly serious.
Search bias concerns whether a review's search strategy itself fails to find studies that do exist and are findable; publication bias concerns whether studies with certain findings were ever published or made findable in the first place. The two can compound each other, since a narrow search is especially likely to miss the kind of unpublished or less-visible studies that publication bias also tends to hide.
Yes. PRISMA 2020 requires authors to present the full search strategy for at least one database, including any filters and limits used, specifically so readers and future reviewers can assess and reproduce the search rather than relying on a general methods description.
References
- Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6:245. read 10 October 2026. View on Systematic Reviews
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. read 10 October 2026. View on BMJ
- Dobrescu AI, Nussbaumer-Streit B, Klerings I, et al. Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most topics: a systematic review. J Clin Epidemiol. 2021;137:209-217. read 10 October 2026. View on Journal of Clinical Epidemiology
- Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.4. Cochrane, 2023, Chapter 4: Searching for and selecting studies. read 10 October 2026. View Cochrane Handbook
- Rethlefsen ML, Kirtley S, Waffenschmidt S, et al; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10:39. read 10 October 2026. View on Systematic Reviews
- Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R. Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Med Res Methodol. 2018;18:85. read 10 October 2026. View on BMC Medical Research Methodology
- Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064-1065. read 10 October 2026. View on BMJ
- Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2011;(8):MR000026. read 10 October 2026. View on Cochrane Database of Systematic Reviews
- Lefebvre C, Glanville J, Briscoe S, et al. Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.4. Cochrane, 2023, Chapter 4. read 10 October 2026. View Cochrane Handbook Chapter 4
- Egger M, Jüni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7(1):1-76. read 10 October 2026. View on Health Technology Assessment
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature. Pubrica. https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/
Pubrica Academy. “Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature.” Pubrica, 10 Oct. 2026, pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/.
Pubrica Academy (2026) Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature. Pubrica. Available at: https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/ (Accessed: 15 October 2026).
Pubrica Academy. “Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature.” Pubrica. October 10, 2026. https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/
Pubrica Academy, ‘Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/> accessed 15 October 2026.
Pubrica Academy. Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/
Pubrica Academy, “Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/
@misc{/la_search_bias_systematic_review_2026,
author = {{Pubrica Academy}},
title = {Search Bias in Systematic Reviews: Database Coverage, Language Restriction and Grey Literature},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/concepts-definitions/what-is-search-bias-in-a-systematic-review/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Search-Strategy Bias Checklist
The search strategy should be reviewed against this list before it is finalized.
[ ] Searched at least two databases with documented, differing coverage (e.g. PubMed/MEDLINE + Embase) [ ] Any language restriction is explicitly justified or tested with a sensitivity analysis [ ] Included a grey-literature search (conference abstracts, dissertations, preprints) [ ] Included a trial-registry search for unpublished or ongoing studies [ ] Full search strategy for at least one database is documented and included in the review [ ] A second reviewer or information specialist checked the search strategy before it was run
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s systematic review team designs and peer-checks multi-database search strategies, including grey-literature and registry searches, so that the evidence base of the review is not shaped by avoidable coverage gaps.