Contents
On this page
- Frequency of data extraction errors
- Types of discrepancy and their corresponding remedies
- Requirements of independent extraction
- Prespecification of the resolution procedure
- Purpose and later use of the discrepancy log
- Ambiguity in the source paper
- Reporting the extraction process for reader appraisal
- Machine assistance as a supplementary checker rather than a second extractor
- Key points
- Application to your study
- Frequently asked questions
Academy · Research Data Analytics & AI · data discrepancies
Handling Data Extraction Discrepancies in Systematic Reviews
Extraction errors are not rare events in systematic reviews. Data extraction or reporting errors were found in 20 out of 34 Cochrane reviews in one examination, and a minimum of seven out of 27 meta-analyses using standardised mean differences had substantial data extraction errors. Double extraction exists to identify such errors while they remain inexpensive to correct; a disagreement between two extractors therefore indicates that the process is functioning as intended.
Most review teams treat extraction disagreements as friction to be resolved quickly: one extractor is assumed to be correct and the other to have misread, an adjudication is made, and the dataset is updated. This approach discards the most useful information that a review generates about its own reliability. A disagreement indicates that at least one of two careful readers found the source ambiguous, and the appropriate response depends on the type of ambiguity. This article examines how to distinguish these types and how to handle each in a manner that leaves a documented record.
Learning objectives
- Classification of disagreements by type (transcription, interpretation, judgement or source ambiguity), each of which requires a different remedy
- A resolution procedure that resolves disagreements without one extractor simply deferring to the other
- Maintenance of a discrepancy log that remains useful at analysis and write-up rather than being discarded
- Approaches to a source paper that contradicts itself
- Wording of a methods statement on discrepancies that reviewers will accept
- Use of machine-assisted extraction as an additional checker without allowing it to become the record
Common misconceptions
- “Disagreements mean someone made a mistake.” Frequently neither extractor misread anything: the paper was ambiguous, and the disagreement is a finding about the paper. Such cases require a documented decision rule, not arbitration.
- “Resolved by consensus” is an adequate description. The phrase conveys no procedural information. A reader needs to know whether the source was re-read, whether a third person was involved and what happened when consensus could not be reached.
- “A second person checking the first person’s form is double extraction.” Cochrane’s standard is at least two people working independently, with disagreements resolved by discussion or a third assessor. A checker is anchored on the entry already recorded and verifies it rather than re-reading the source.
- “Errors get caught at peer review.” Peer review does not reliably detect them. Errors were found in 20 of 34 published Cochrane reviews examined for them, and in at least seven of 27 standardised mean difference meta-analyses, all of which had passed peer review.
- “The discrepancy log is internal paperwork.” The log is the evidence base for the review’s reliability claim, the source of sensitivity analyses on disputed values, and the record required when a reader queries a number two years after publication.
Frequency of data extraction errors
Two empirical findings should inform the design of the process, because they affect how seriously a team treats the reconciliation step.
In an examination of Cochrane reviews, data extraction or reporting errors were found in 20 out of 34 reviews 28. And in an examination of meta-analyses using standardised mean differences, a minimum of seven out of 27 had substantial data extraction errors 26. These reviews had been conducted by experienced teams, peer reviewed and published. The reasonable inference is not that those teams were careless, but that extraction is an error-prone activity, that the errors are not visible in the published output, and that a process which does not actively seek them will not detect them 27,28.
Standardised mean differences are a predictable source of error. They require the reviewer to identify the correct variability measure, recognise whether a value is a standard deviation or a standard error, determine whether the metric is a final value or a change score, and apply a conversion — a sequence of four judgements, each of which can fail without detection 26. Our pages on standardised mean difference estimation and extracting quantitative data cover the conversions themselves; this page addresses the process by which errors in them are detected.
Error frequencies from the studies of extraction accuracy cited in the Cochrane Handbook for Systematic Reviews of Interventions, Chapter 5, on collecting data; the independence requirement is Cochrane’s MECIR standard.
Types of discrepancy and their corresponding remedies
Treating all disagreements in the same way is the reason reconciliation is perceived as administrative overhead. The types of disagreement differ.
Transcription discrepancies are simple copying errors: a digit transposed, a value taken from the adjacent column, a decimal point misplaced. One extractor is correct, the remedy is to re-read the source, and the only process question is whether this type occurs often enough to indicate fatigue or a poorly laid-out form.
Interpretation discrepancies arise when both extractors read the same text and understood the field differently — one recorded the number randomised where the other recorded the number analysed, or one treated a reported value as a standard deviation where the other read it as a standard error. These are problems with the form. The remedy is to clarify the field definition and re-extract that field across all studies already completed, because the same misunderstanding will have been applied consistently.
Judgement discrepancies occur where the field genuinely requires an assessment: a risk-of-bias domain, whether an intervention counts as multimodal, whether a study’s outcome matches the review’s construct. Neither extractor is wrong. The remedy is a documented decision rule plus arbitration, with the rule recorded so that the next instance is decided in the same way.
Source ambiguity arises where the paper itself does not support a single answer — the text and the table disagree, a denominator is missing, the timepoint is not stated. Arbitration between extractors cannot resolve this, because the information is not in the source. The remedy lies outside the team: the authors should be contacted and other reports of the same study, the registry entry or the regulatory record checked; if none of these resolves the question, the ambiguity should be recorded and a sensitivity analysis planned 11,24.
Four kinds of discrepancy and the fix each one needs
| Type | What happened | The right fix | What it means for the rest of the review |
|---|---|---|---|
| Transcription | A value was copied wrongly — digit transposed, wrong column, misplaced decimal | Both extractors re-read the source independently; one entry is simply correct | A cluster suggests form layout or session length, not carelessness |
| Interpretation | Both read the same text and understood the field differently | Amend the field definition, then re-extract that field across every study already done | The misunderstanding was applied consistently, so earlier studies are affected too |
| Judgement | The field requires an assessment and two reasonable readers assessed differently | Documented decision rule plus a named arbitrator; record the rule for the next instance | The rule becomes part of the method and should be reported |
| Source ambiguity | The paper does not support a single answer — text and table disagree, timepoint absent | Look for other reports, the registry and regulatory record; contact the authors; then sensitivity analysis | Feeds the risk-of-bias assessment and the uncertainty described in the results |
Classification compiled as a working taxonomy; the two-independent-extractors requirement and the discussion-or-third-assessor resolution route are Cochrane’s methodological standard for data collection.
Requirements of independent extraction
Cochrane’s methodological standard for collecting data is to use at least two people working independently, with disagreements resolved by discussion or a third assessor 1,2. The operative word is “independently”, and three factors undermine it in practice.
Sequential visibility: the second extractor can see the first extractor’s entries, in a shared spreadsheet or in review software configured without blinding. The anchoring effect is strong, and the result is confirmation rather than extraction. The tooling should be configured to conceal the other entries until both are complete.
Role asymmetry: a senior and a junior extractor working as a pair are not two independent readers, because the junior extractor is likely to defer on any contestable point. If the pairing is unavoidable, the arbitration must be conducted by someone other than the senior extractor, and the discrepancy log should record who was overruled.
Workload asymmetry: an extractor doing all 40 studies alongside one doing only the first ten produces a review in which three quarters of the data is single-extracted 27. If capacity requires partial double extraction, this should be stated explicitly in the methods, together with the fields and studies that were double-extracted, and the double-extracted subset should be selected by rule rather than convenience. The Handbook’s own guidance on scaling the extraction team — set out on our page on what data to extract — exists precisely so that this trade-off is planned rather than absorbed.
Prespecification of the resolution procedure
A workable procedure comprises four steps and one named person, and it is agreed before extraction starts rather than devised at the first disagreement.
Step one: both extractors return to the source independently. The extractors return to the paper, not to each other. A large share of disagreements are resolved at this step, and those resolved are the transcription errors that should be eliminated before any discussion begins.
Step two: discussion, with the field definition in hand. The question under discussion is what the form field means, not which extractor is correct. If the discussion reveals that the field was ambiguous, the disagreement is an interpretation discrepancy, and the field definition is amended and reapplied retrospectively.
Step three: a third assessor, named in advance. Arbitration requires a person who makes a decision, not a meeting that reaches a general impression. The arbitrator reviews both entries and the source and records the reason for the decision in one line.
Step four: escalation for source ambiguity. If the paper cannot settle the question, the item leaves the reconciliation process and enters the author-contact and sensitivity-analysis route described below. Arbitration should not be used to arrive at a value the source does not support.
A further rule is recommended: when a disagreement is resolved in favour of one entry, the arbitrator records which entry and why, and the log retains both original values. Overwriting the rejected value destroys the only record that the number was ever in question.
The resolution sequence, and when to stop
| Step | Who does it | Stop here if | Record |
|---|---|---|---|
| 1. Independent re-read of the source | Both extractors, separately, not together | The two entries now agree | Which entry was corrected and why |
| 2. Discussion against the field definition | Both extractors | The field definition settles it | Whether the definition needed amending, and the retrospective re-extraction triggered |
| 3. Third assessor | A person named before extraction began | The arbitrator can decide from the source | The decision, the reason, and both original values |
| 4. Escalation outside the team | Whoever handles author contact | Never resolved internally — the source cannot answer | Both values, the contact attempt, the handling rule and the planned sensitivity analysis |
Sequence built on Cochrane’s standard that disagreements be resolved by discussion or a third assessor.
Purpose and later use of the discrepancy log
A discrepancy log requires one line per disagreement and serves three purposes.
During extraction, it indicates the kinds of errors being generated. A run of interpretation discrepancies on the same field indicates that the form should be corrected immediately, before another twenty studies are extracted with the same ambiguity. A rising rate of transcription errors late in a session indicates that sessions are too long.
At analysis, it identifies which values were disputed, which constitutes the natural candidate list for a sensitivity analysis. A pooled estimate that changes materially depending on how three contested values were resolved is a fragile estimate, and readers should be informed of this 14.
At write-up, it provides the evidence for the methods claims. “Disagreements were resolved by discussion, with a third assessor where discussion did not resolve them; disagreements and their resolution are listed in Supplementary File 3” is a sentence a reviewer can verify. “Resolved by consensus” is not. The log also enables a post-publication query about a specific number to be answered without re-extracting the study.
The log should be kept in the same system as the extraction itself, so that entries are made at the moment of disagreement rather than reconstructed — structured data capture makes this close to automatic.
Ambiguity in the source paper
Source ambiguity is the category teams handle least well, because the tendency is to select the more plausible value and proceed. This tendency should be resisted in favour of a fixed sequence.
First, other reports of the same study should be sought: other publications, a conference abstract, the trial registry entry, a regulatory submission, an appendix or a thesis. Studies are reported more than once far more often than teams assume, and the other report frequently resolves the ambiguity 11. Where the study is a drug trial, the regulatory record merits direct examination — approval packages are a source of outcome detail that the papers often omit, as covered on our page on sources of review data 24.
Second, the authors should be contacted with a specific question. “Table 2 reports 84 participants analysed for the primary outcome and the text reports 87; which is correct, and what accounts for the difference?” is answered more often than a general request for data. The request and the reply, or the absence of a reply, should be recorded.
Third, if the ambiguity remains unresolved, the handling should be made explicit. Both values should be recorded, the methods should state which was used and why, and a sensitivity analysis should be run using the alternative. Where a value cannot be used at all, the study should be excluded from that analysis and the exclusion reported, rather than a value being imputed without disclosure. An internal contradiction that cannot be resolved is also a legitimate input into the risk-of-bias assessment for that study, and a reason to describe the result as uncertain in the text 6,21.
Reporting the extraction process for reader appraisal
Reporting requirements in this area are modest and frequently not met. Four items belong in the methods.
The number of people who extracted data and whether they worked independently; how disagreements were resolved, naming discussion, a third assessor or both, and what happened when neither resolved them; which fields and which studies were double-extracted, if not all; and where the log of disagreements and their resolutions can be found 12.
Two items belong in the results or the supplement: the number of disagreements by field or type, and which values remained contested and how they were handled in the analysis. This is not an admission of carelessness — given that errors were found in 20 of 34 published reviews examined, a review that reports finding and correcting disagreements is more credible than one that reports none 28.
No section of the review should claim that there were no disagreements. In a double-extracted review of any size, such a claim indicates either a reporting omission or that the second extractor was not working independently. The PRISMA 2020 statement asks authors to describe the data collection process, including how many reviewers collected data and whether they worked independently; the discrepancy procedure is the part of that description reviewers read most closely 10. Reporting checklist review catches the omission before submission.
The discrepancy procedure, as a decision rule and as a methods sentence
Describes nothing a reader can check
Two reviewers extracted data using a standardised form. Any discrepancies were resolved by consensus. A third reviewer was consulted if necessary.
“Resolved by consensus” names no procedure. It is compatible with the junior extractor deferring to the senior one on every contested value.
No statement of independence. Two reviewers using “a standardised form” could be one extracting and one checking, which is a weaker method than the standard requires.
“If necessary” leaves the escalation criterion undefined. A reader cannot tell when the third reviewer was used or how often.
No handling for an ambiguous source. The procedure implicitly arbitrates its way to a number the paper may not support.
No record located. There is nothing a reader, an editor or a future reader of the review can inspect.
No coverage statement. Whether all fields and all studies were double-extracted, or only some, is left entirely open.
Describes a procedure and leaves a record
Two reviewers extracted data independently into separate records in the review software, with each reviewer’s entries hidden from the other until both were complete; all fields were double-extracted for all included studies. Disagreements were handled by a fixed sequence agreed before extraction began. Both reviewers first returned to the source independently; where the entries still differed, they discussed the disagreement against the written field definition, and where that discussion showed the field itself to be ambiguous the definition was amended and that field re-extracted for every study already completed. Disagreements surviving discussion were referred to a third reviewer, named in the protocol, who decided from the source and recorded the reason in one line. Where the source itself did not support a single value — for example where the text and a table reported different denominators — we did not arbitrate; we searched other reports of the same study, the trial registry entry and, for the drug trials, the regulatory approval package, and wrote to the corresponding author with the specific question. Values that remained unresolved were recorded with both candidate values, analysed using the value stated in the supplement, and tested in a pre-specified sensitivity analysis using the alternative. Every disagreement, its type, its resolution and the arbitrator’s reason are listed in Supplementary File 3; the number of disagreements by field is reported in the results.
Independence is stated as a mechanism — entries hidden until both are complete — rather than asserted as an adjective.
The escalation sequence is explicit and ordered, so a reader knows what happened at each stage and in what order.
Interpretation disagreements trigger retrospective re-extraction, which is the step that prevents one misunderstood field from affecting every study extracted before the error was noticed.
Source ambiguity is routed out of arbitration and into author contact and sensitivity analysis, which is the only defensible handling when the information is not in the paper.
Both candidate values are retained for contested items, so the sensitivity analysis is possible and the uncertainty is visible.
The log is located and the disagreement count is reported, which makes the whole claim verifiable rather than merely reassuring.
Given that extraction or reporting errors were found in 20 of 34 published Cochrane reviews examined for them, a reviewer reading the first version has no way to tell whether this review is one of them. The second version is longer by a paragraph and is the only one of the two that can be assessed.
Machine assistance as a supplementary checker rather than a second extractor
At least 26 studies have tested natural language processing and machine learning approaches to automating data extraction, so there is a substantive research literature in this area rather than vendor claims alone 4. The useful framing for discrepancy handling is that a tool can act as an additional, low-cost checker on the fields it handles, not as one of the two independent extractors 4,7.
Used in this way, it adds something human double extraction does not: it does not fatigue and is consistent, so it can detect the late-session transcription errors that both human extractors are most likely to make at the same time of day 15. It is also consistent in its errors, which means that a systematic tool error appears as a run of disagreements on one field rather than as scattered noise, making it easier rather than harder to detect.
Three rules apply if such a tool is used. Its output enters the reconciliation log like any other entry, with its disagreements counted. It does not resolve a disagreement between the two human extractors, because its agreement with one of them is not evidence. The review reports which fields were machine-assisted and how tool-human disagreements were handled. A tool used as an undisclosed primary extractor adds an unverifiable layer to precisely the step that the published error rates indicate requires the most rigorous verification.
Key points
Key points
Extraction disagreements indicate that the process is working. Data extraction or reporting errors were found in 20 out of 34 Cochrane reviews in one examination, and at least seven of 27 standardised mean difference meta-analyses had substantial errors; a review that finds no disagreements has therefore either not looked for them or not extracted independently. Cochrane’s standard is at least two people working independently, with disagreements resolved by discussion or a third assessor. Each disagreement should be classified before it is resolved: transcription errors require a re-read; interpretation errors require the field definition to be amended and the field re-extracted across every study already completed; judgement differences require a documented rule and a named arbitrator; and source ambiguity cannot be arbitrated at all, but is referred to other reports, the registry, the regulatory record and the authors, and then addressed in a sensitivity analysis. Every disagreement should be logged with both original values, the procedure and the counts should be reported, and machine extraction should be treated as an additional checker whose disagreements are counted, not as one of the two independent readers.
Application to your study
Application to your study
- The resolution sequence should be agreed before extraction begins. Independent re-read, discussion against the field definition, named third assessor, escalation for source ambiguity — written into the protocol with the arbitrator’s name.
- Extraction software should be configured to conceal each extractor’s entries from the other. If the second extractor can see the first extractor’s values, the process constitutes checking rather than independent extraction.
- Each disagreement should be classified when it is logged. Transcription, interpretation, judgement or source ambiguity — the type determines the fix and whether earlier studies need revisiting.
- Fields should be re-extracted retrospectively after an interpretation error is corrected. A misunderstood field was misunderstood consistently, so every study already extracted is affected.
- Both values should be retained for contested items. The rejected entry should not be overwritten; it is the input to the sensitivity analysis and the answer to any later query.
- Source ambiguity should be referred outside the team. Other reports, registry, regulatory record, then a specific question to the corresponding author — logged either way.
- The procedure and the counts should be reported. Who extracted, independently or not, how disagreements were resolved, how many there were, and where the log can be found.
Frequently asked questions
Sufficiently common that the process must assume they will occur. In an examination of Cochrane reviews, data extraction or reporting errors were found in 20 out of 34 reviews. In an examination of meta-analyses using standardised mean differences, a minimum of seven out of 27 had substantial data extraction errors. These were published, peer-reviewed reviews, which indicates two things: extraction errors are not a sign of an unusually careless team, and peer review does not reliably detect them. Standardised mean differences are a predictable source of error because they require a chain of four judgements — identifying the variability measure, distinguishing standard deviation from standard error, establishing whether the metric is a final value or a change score, and applying the conversion — each of which can fail silently.
Cochrane’s methodological standard for collecting data is to use at least two people working independently, with disagreements resolved by discussion or a third assessor. Two elements of that standard are routinely diluted. “Independently” means each person extracts from the source into their own record without seeing the other’s entries — one person extracting while a second checks the completed form is a weaker procedure, because the checker anchors on what is already written. And “a third assessor” means a named person who makes a decision from the source, not a team meeting that arrives at a general impression. Agreeing both elements in the protocol, before extraction starts, requires a single sentence and prevents the most common failure mode, which is unacknowledged deference.
Arbitration should cease, because the information is not available to the team. A fixed sequence should be followed instead. The same study should be sought in other reports: another paper, a conference abstract, the trial registry entry, a regulatory submission, an appendix or a thesis. Studies are reported more than once more often than teams expect, and the second report frequently resolves the ambiguity. The corresponding author should then be contacted with a specific question rather than a general request — a query naming the table, the text and the exact contradiction is answered far more often. If the ambiguity remains unresolved, both candidate values should be recorded, the methods should state which was used and why, a pre-specified sensitivity analysis should be run using the alternative, and the internal contradiction should inform the risk-of-bias judgement for that study.
The phrase is common and inadequate. It conveys nothing to a reader about whether the source was re-read, whether a third person was involved, what happened when consensus was not reached, or whether all fields and all studies were double-extracted. It is also compatible with the weakest form of the process, in which a junior extractor defers to a senior one on every contested value. It should be replaced with a description of the mechanism: how independence was enforced, the ordered escalation steps, the identity of the named arbitrator, how source ambiguity was handled separately, and where the log of disagreements and resolutions can be found. This requires four or five sentences and constitutes the part of the data collection description that methods reviewers read most closely.
Yes, and reporting a number makes the review more credible rather than less. Given that extraction or reporting errors were found in 20 of 34 published Cochrane reviews examined for them, a review reporting zero disagreements indicates either that disagreements were not logged or that the second extractor was not working independently. The count should be reported, ideally broken down by field or by discrepancy type, together with the values that remained contested and how they were handled in the analysis. A disagreement count is evidence that double extraction took place; its absence provides no evidence.
Yes, provided it is a declared design decision rather than an unreported shortfall. The double-extracted subset should be chosen by rule rather than by convenience: all outcome and results fields for all studies is the usual priority, since that is where errors propagate into the pooled estimate, with descriptive characteristics single-extracted and spot-checked. The methods should then state exactly this — which fields, which studies, and the spot-check rate. The Cochrane Handbook scales the extraction team to review size for this reason: fewer than 10 included studies, 2 to 3 extractors; 10 to 20 studies, 4 to 6; above that, more than 6. If a team is below that scale, partial double extraction declared openly is more defensible than full double extraction claimed imprecisely.
As an additional checker, yes; as one of the two independent extractors, no. At least 26 studies have tested natural language processing and machine learning approaches to automating data extraction, so the capability is established but narrow. The useful property for error detection is that a tool does not fatigue and is consistent, so it can detect the late-session transcription errors that two fatigued reviewers are most likely to make simultaneously, and its own errors appear as a systematic run on one field rather than as scattered noise, which makes them easier to identify. Three rules apply: its entries go into the reconciliation log and its disagreements are counted, it never adjudicates between the two human extractors, and the review states which fields were machine-assisted and how tool-human disagreements were handled.
One row per disagreement, recording the study, the field, both original values verbatim, the discrepancy type, the resolution step that settled it, the value adopted, who adopted it and the one-line reason. The rejected value should be retained — overwriting it destroys the only record that the number was ever in question, and it is the input to any sensitivity analysis on contested items. A flag should be added for items escalated outside the team, with the contact attempt and its outcome. The log should be kept in the same system as the extraction so that rows are created at the moment of disagreement rather than reconstructed before submission, and it should be filed as a supplementary file rather than discarded: it is the evidence for the methods claim and the record required if a reader queries a specific number after publication.
References
- Chapter 5: Collecting data. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
- Reddy SM, Patel S, Weyrich M, Fenton J, Viswanathan M. Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence. Systematic Reviews. 2020;9(1):243. doi:10.1186/s13643-020-01450-2 read 9 October 2026. doi.org/10.1186/s13643-020-01450-2
- Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews. 2015;4:78. read 9 October 2026. cochrane.org
- Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1):1. doi:10.1186/2046-4053-4-1 read 9 October 2026. doi.org/10.1186/2046-4053-4-1
- Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database of Systematic Reviews. 2014;10:MR000035. read 9 October 2026. cochrane.org
- Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews. 2014;3(1):74. doi:10.1186/2046-4053-3-74 read 9 October 2026. doi.org/10.1186/2046-4053-3-74
- Ip S, Hadar N, Keefe S, Parkin C, Iovin R, Balk EM, Lau J. A web-based archive of systematic review data. Systematic Reviews. 2012;1:15. read 9 October 2026. cochrane.org
- Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews. 2011;2011(8):MR000026. doi:10.1002/14651858.MR000026.pub2 read 9 October 2026. doi.org/10.1002/14651858.MR000026.pub2
- Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71 read 9 October 2026. doi.org/10.1136/bmj.n71
- Mayo-Wilson E, Li T, Fusco N, Dickersin K. Practical guidance for using multiple data sources in systematic reviews and meta-analyses (with examples from the MUDS study). Research Synthesis Methods. 2018;9:2-12. read 9 October 2026. cochrane.org
- Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. Journal of Clinical Epidemiology. 2018;94:8-18. doi:10.1016/j.jclinepi.2017.10.017 read 9 October 2026. doi.org/10.1016/j.jclinepi.2017.10.017
- Munn Z, Peters MD, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology. 2018;18(1):143. doi:10.1186/s12874-018-0611-x read 9 October 2026. doi.org/10.1186/s12874-018-0611-x
- Mayo-Wilson E, Li TJ, Fusco N, Bertizzolo L, Canner JK, Cowley T, et al. Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy. Journal of Clinical Epidemiology. 2017;91:95-110. read 9 October 2026. cochrane.org
- Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. Journal of Clinical Epidemiology. 2016;74:119-123. read 9 October 2026. cochrane.org
- Millard LAC, Flach PA, Higgins JPT. Machine learning to assist risk-of-bias assessments in systematic reviews. International Journal of Epidemiology. 2016;45:266-277. read 9 October 2026. cochrane.org
- Marshall IJ, Kuiper J, Wallace BC. RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association. 2016;23:193-201. read 9 October 2026. cochrane.org
- Rethlefsen ML, Farrell AM, Osterhaus Trzasko LC, Brigham TJ. Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews. Journal of Clinical Epidemiology. 2015;68(6):617-626. doi:10.1016/j.jclinepi.2014.11.025 read 9 October 2026. doi.org/10.1016/j.jclinepi.2014.11.025
- Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, Tierney JF; PRISMA-IPD Development Group. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. JAMA. 2015;313:1657-1665. read 9 October 2026. cochrane.org
- Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. read 9 October 2026. cochrane.org
- Dwan K, Altman DG, Clarke M, Gamble C, Higgins JPT, Sterne JAC, Williamson PR, Kirkham JJ. Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Medicine. 2014;11:e1001666. read 9 October 2026. cochrane.org
- Elliott JH, Turner T, Clavisi O, Thomas J, Higgins JP, Mavergames C, Gruen RL. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLOS Medicine. 2014;11(2):e1001603. doi:10.1371/journal.pmed.1001603 read 9 October 2026. doi.org/10.1371/journal.pmed.1001603
- Gross A, Schirm S, Scholz M. Ycasd - a tool for capturing and scaling data from graphical representations. BMC Bioinformatics. 2014;15(1):219. doi:10.1186/1471-2105-15-219 read 9 October 2026. doi.org/10.1186/1471-2105-15-219
- Turner EH. How to access and process FDA drug approval packages for use in research. BMJ. 2013;347. read 9 October 2026. cochrane.org
- Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database: update and key issues. New England Journal of Medicine. 2011;364:852-860. read 9 October 2026. cochrane.org
- Gotzsche PC, Hrobjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA. 2007;298:430-437. read 9 October 2026. cochrane.org
- Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology. 2006;59:697-703. read 9 October 2026. cochrane.org
- Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. Journal of Clinical Epidemiology. 2005;58:741-742. read 9 October 2026. cochrane.org
- Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine. 1997;127:380-387. read 9 October 2026. cochrane.org
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 9). Handling Data Extraction Discrepancies in Systematic Reviews. Pubrica. https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/
Pubrica Academy. “Handling Data Extraction Discrepancies in Systematic Reviews.” Pubrica, 9 Oct. 2026, pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/.
Pubrica Academy (2026) Handling Data Extraction Discrepancies in Systematic Reviews. Pubrica. Available at: https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/ (Accessed: 15 October 2026).
Pubrica Academy. “Handling Data Extraction Discrepancies in Systematic Reviews.” Pubrica. October 9, 2026. https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/
Pubrica Academy, ‘Handling Data Extraction Discrepancies in Systematic Reviews’ (Pubrica, 9 October 2026) <https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/> accessed 15 October 2026.
Pubrica Academy. Handling Data Extraction Discrepancies in Systematic Reviews. [Internet]. Pubrica; 2026 Oct 9 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/
Pubrica Academy, “Handling Data Extraction Discrepancies in Systematic Reviews,” Pubrica, Oct. 9, 2026. [Online]. Available: https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/
@misc{how_to_handle_discrepancies_while_you__2026,
author = {{Pubrica Academy}},
title = {Handling Data Extraction Discrepancies in Systematic Reviews},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/systematic-review/how-to-handle-discrepancies-while-you-collect-data-for-systemic-review/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Discrepancy procedure and reconciliation log
The top half should be agreed before extraction starts and the bottom half completed during extraction. The type column is what makes the log a diagnostic tool rather than a list of corrections.
DISCREPANCY PROCEDURE — REVIEW: __________________
=== AGREED BEFORE EXTRACTION BEGINS ===
Number of independent extractors: ____
Independence enforced how? [ ] separate records, entries hidden
[ ] separate software accounts
[ ] other: ______________________
Coverage: [ ] all fields, all studies double-extracted
[ ] partial — fields: ____________________
studies: ___________________
spot-check rate on the rest: ____
NAMED THIRD ASSESSOR: ______________________________
Who handles author contact: ________________________
Escalation criterion (when an item leaves arbitration):
___________________________________________________
Pre-specified sensitivity analysis on contested values? [ ] yes [ ] no
=== RESOLUTION SEQUENCE (do not skip steps) ===
1. Both extractors re-read the SOURCE independently.
2. Discuss against the WRITTEN FIELD DEFINITION.
If the definition was ambiguous -> amend it AND re-extract that
field for every study already completed. Record the re-extraction.
3. Refer to the named third assessor, who decides from the source
and records a one-line reason.
4. If the SOURCE cannot answer, do NOT arbitrate. Escalate:
[ ] other reports of the same study
[ ] trial registry entry
[ ] regulatory approval package
[ ] conference abstract / thesis / appendix
[ ] specific question to the corresponding author
Then record both values and run the sensitivity analysis.
=== RECONCILIATION LOG (one row per disagreement) ===
Study | Field | Extractor A value | Extractor B value | TYPE | Step that
| | | | | resolved it
| | | | |
... continue; TYPE = TRANSCRIPTION / INTERPRETATION / JUDGEMENT /
SOURCE AMBIGUITY
... then for each row also record:
Value adopted | Decided by | One-line reason | Still contested? (Y/N)
=== RUNNING DIAGNOSTICS (check weekly) ===
Disagreements by field — any field with a run of them? ____________
If yes, the FIELD is the problem. Amend and re-extract.
Disagreements by type:
Transcription ____ Interpretation ____ Judgement ____ Source ____
Interpretation fixes requiring retrospective re-extraction: ______
Items escalated to author contact: ____ Replies: ____
Items still contested at analysis: ____
=== MACHINE-ASSISTED FIELDS (if used) ===
Tool: ____________________ Fields it handled: ______________
Tool entries logged as a third entry? [ ] yes
Tool-human disagreements: ____ How handled: ________________
Tool used to adjudicate between humans? [ ] NO (it must not be)
Stated in the methods? [ ] yes
=== WHAT GOES IN THE MANUSCRIPT ===
[ ] Number of extractors and that they worked independently
[ ] The mechanism that made them independent
[ ] Which fields and studies were double-extracted
[ ] The ordered resolution sequence, including the third assessor
[ ] Separate handling for source ambiguity
[ ] Number of disagreements, by field or type
[ ] Contested values and how they were handled in the analysis
[ ] Location of this log as a supplementary file
[ ] The phrase "resolved by consensus" does NOT appear alone Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica conducts double extraction with enforced blinding, a classified reconciliation log and a named arbitrator, and returns the log as a supplementary file alongside the dataset. Where a disagreement proves to reflect source ambiguity rather than extractor error, Pubrica drafts the author query and prepares the sensitivity analysis rather than selecting the more plausible value without documentation.