Contents
On this page
- The abstraction form as the basis of methodological control
- Case report form design: operational definitions before fields
- Pilot testing of the abstraction form
- Measurement and adjudication of inter-abstractor reliability
- Complementary guidance from registry methodology
- Transparent reporting of retrospective chart reviews
- Data governance: de-identification, waivers and limited data sets
Academy · Research Data, Analytics & AI · Retrospective Chart Review Guidelines
Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies
The validity of a retrospective chart review depends on the abstraction form: its construction, its pilot testing and the procedures for detecting and resolving disagreements between abstractors. This article sets out guidance on each of these stages.
Retrospective chart review is often regarded as the least demanding form of clinical research, since it involves no recruitment, consent visits or follow-up, only records that already exist. This reputation reflects its procedural simplicity rather than its methodological demands. Because the data were generated for clinical care rather than for the study question, a retrospective design places the entire burden of rigour on the abstraction instrument and the people using it. Gearing and colleagues, writing on chart review methodology in child and adolescent psychiatry, describe this as the central paradox of the design: it appears procedurally simple while being methodologically unforgiving.1 This article focuses on the three decisions that determine whether a retrospective chart review produces usable evidence or unreplicable results: the design of the case report form (CRF) or abstraction form, its pilot testing before full data collection, and the measurement and adjudication of inter-abstractor reliability when more than one person abstracts data from the same records.
Learning objectives
- Why the abstraction form, rather than the chart, is the principal unit of methodological control in a retrospective study
- What a pilot test of the form should detect before full-scale abstraction begins
- How to measure and report inter-abstractor agreement, and how to resolve disagreements between abstractors
- Where registry-level guidance, such as the Registries for Evaluating Patient Outcomes guide of the Agency for Healthcare Research and Quality (AHRQ), overlaps with methodology specific to chart review
Common misconceptions
- “If the chart contains the data, extraction is simply data entry.” Clinical notes were written for patient care, not for a protocol’s operational definitions. A free-text note of “mild shortness of breath on exertion” must still be mapped to the ordinal dyspnoea scale used by the study, and two abstractors will map it differently unless the form and its instructions remove the need for judgement before abstraction begins.
- “A single well-trained abstractor is more reliable than a team, so a reliability check is unnecessary.” A single abstractor eliminates between-rater disagreement but not abstractor drift over time, and it removes the only mechanism, comparison against a second abstractor, that would reveal systematic misreading of the form. Gearing et al. specifically recommend double abstraction of at least a subset of charts for this reason.1
The abstraction form as the basis of methodological control
In a prospective trial, the protocol defines what will be measured before any patient is enrolled. In a retrospective chart review, the data already exist in whatever form clinicians happened to record them, so the abstraction form has to do the protocol's job after the fact: it has to fix, for every variable, exactly what counts as a positive finding, exactly which part of the chart is the authoritative source if two sections disagree, and exactly how missing or ambiguous entries are to be coded. Gearing et al. frame this as the defining methodological challenge of chart review — the design looks low-effort at the recruitment stage precisely because all of the effort has to move earlier, into instrument design, rather than disappearing.1
Two practical consequences follow. First, the form should be built from the study's actual research question and case definitions, not reverse-engineered from whatever fields happen to appear in the electronic health record (EHR) export. Second, every field that requires any interpretation — a symptom severity grade, a complication classification, a cause-of-admission category — needs a written operational definition sitting alongside it, because that definition is what a second abstractor will be checked against later.
Case report form design: operational definitions before fields
A workable abstraction form specifies, for each variable: the exact wording or code to be entered, the preferred source location in the chart (e.g., discharge summary over admission note when the two conflict), and the rule for "not documented" versus "documented as absent" — a distinction that is frequently collapsed into a single blank cell and then impossible to separate during analysis. Gearing and colleagues recommend building these operational definitions into a coding manual that accompanies the form rather than relying on abstractor memory or verbal instructions, precisely because verbal instructions do not survive staff turnover or a multi-month data collection window.1
The form itself should also be structured to minimize free text wherever a categorical or ordinal field will serve the analysis plan, since free text has to be re-coded later by someone, introducing exactly the kind of judgment call the form was supposed to remove.
Pilot testing of the abstraction form
Before a form is used across the full cohort, it should be tested on a small sample of charts chosen to include some of the messier records in the population — incomplete charts, transfers from another facility, cases with conflicting documentation — rather than only the cleanest ones. Gearing et al. identify piloting as a discrete step in retrospective chart review methodology, used to surface fields that are ambiguous, categories that do not cover the range of real documentation, and instructions that two trained readers interpret differently.1 Any field that produces disagreement or confusion during the pilot should be rewritten, and the revised form re-tested rather than patched mid-study, because changing definitions partway through data collection makes early and late abstractions non-comparable.
Measurement and adjudication of inter-abstractor reliability
Once more than one person is abstracting from the same set of charts, the study needs a documented way to check that they are extracting the same thing from the same note. Gearing and colleagues recommend having a subset of charts independently double-abstracted and comparing the two sets of entries, with a pre-specified process for how disagreements are adjudicated — commonly, a third reviewer or the principal investigator resolves discrepancies against the coding manual rather than by majority vote or coin flip.1
Reporting this check matters as much as running it: a retrospective chart review that states what fraction of charts were double-abstracted, what agreement level was found, and how disagreements were resolved gives a reader a basis for judging the data's reliability that a chart review without any reliability statement cannot provide.
Complementary guidance from registry methodology
When retrospective chart data feed into a structured patient registry rather than a one-off analysis, AHRQ's Registries for Evaluating Patient Outcomes: A User's Guide is the standard public reference for data element selection, data quality assurance, and the broader governance questions — data ownership, linkage, and validation plans — that sit above the abstraction form itself.2 The two sources are complementary rather than competing: Gearing et al. address the mechanics of one chart-review data collection effort, while the AHRQ guide addresses the data infrastructure a registry needs to sustain many such efforts over time.
Transparent reporting of retrospective chart reviews
A retrospective chart review is, methodologically, a form of observational study, and the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) Statement is the relevant reporting guideline for writing it up once data collection is finished. STROBE was developed by von Elm, Altman, Egger, Pocock, Gøtzsche and Vandenbroucke for the STROBE Initiative and published simultaneously across several journals in 2007, including Annals of Internal Medicine, PLoS Medicine, BMJ, and The Lancet, specifically to improve the quality of reporting of observational research — cohort, case-control, and cross-sectional designs, the exact family a retrospective chart review belongs to.3 The checklist does not prescribe how a study should be designed or run; it is explicitly a reporting tool, meant to be read alongside design-specific guidance like Gearing et al.'s chart-review methodology rather than instead of it.3
In practice, this means a completed retrospective chart review should state, using STROBE's structure, how participants (charts) were identified and selected, how key variables were defined and measured, what steps were taken to address bias (including the abstraction and reliability steps covered above), and how missing data were handled — the same methodological decisions this article has walked through, now made visible to a reader and a peer reviewer in a standard, checkable format rather than left implicit in a methods paragraph.
Data governance: de-identification, waivers and limited data sets
Before any abstraction form is finalized, a retrospective chart review built on identifiable U.S. patient records also has to clear a data-governance question that sits above study methodology: under what legal basis the records can be reviewed at all. HHS's guidance on the HIPAA Privacy Rule and research sets out the main routes. A covered entity may use de-identified health information for research without further HIPAA authorization, under the standards at 45 CFR 164.502(d) and 164.514(a)-(c).4 Where full de-identification is not practical — because the study genuinely needs dates, geographic detail, or other identifiers to do the abstraction — an Institutional Review Board or Privacy Board can instead approve a waiver or alteration of individual authorization, but only after finding that the use poses no more than minimal privacy risk (supported by a plan to protect and then destroy identifiers), that the research could not practicably be done without the waiver, and that it could not practicably be done without the identifiable data itself.4
A third option, the limited data set, strips specified direct identifiers of the patient and their relatives, employers, or household members and can be shared for research under a data use agreement rather than a full authorization or waiver.4 Which of these three routes applies changes what the abstraction team is legally allowed to record on the form — for example, whether exact admission dates or only date ranges may be captured — so this decision has to be settled before the CRF is finalized, not treated as a separate compliance step layered on afterward.
Data standards, quality dimensions and abstraction tools
The abstraction form's reliability also depends on where it sits inside the broader data-standards picture. Richesson and Nadkarni's review of data standards for clinical research data collection forms found that most form-based studies still define their own ad hoc variable sets rather than reusing standardized common data elements, which limits later pooling or comparison across studies even when each individual form is well designed.5 Where the abstracted values are pulled from an electronic health record rather than paper charts, Weiskopf and Weng's framework for EHR data quality assessment gives the abstraction team a vocabulary for naming what can go wrong — completeness, correctness, concordance across sources, plausibility, and currency — rather than treating "data quality" as a single undifferentiated concern.6
On tooling, Harris and colleagues' description of REDCap (Research Electronic Data Capture) documents a metadata-driven electronic data capture platform built specifically to support the kind of structured, auditable abstraction a retrospective chart review needs, including built-in data validation and export functions that reduce the transcription errors a paper-based abstraction form cannot catch on its own.7 Cowie and colleagues' review of how electronic health records facilitate clinical research situates this at the study level, noting that EHR-sourced abstraction still requires the same case definitions and governance decisions this article has covered, even though the record itself is digital rather than paper.8
Two further sources bear on the methodological side of a retrospective design built on existing records. Harris, McGregor and colleagues' guidance on the use and interpretation of quasi-experimental studies in medical informatics is directly relevant wherever a retrospective chart review is also being used to evaluate an intervention or policy change, since it catalogues validity threats — secular trend, regression to the mean, and selection among them — that a chart-review design built around existing records cannot rule out by design alone.9 And at the data-element level, ISO/IEC 11179, the international standard for metadata registries, gives the formal definition of what a properly specified data element requires — a registered name, a definition, a value domain, and a classification — that the operational-definitions approach recommended earlier in this article is, in effect, applying at a project scale rather than an international-registry scale.10
Finally, where free-text narrative in the chart — discharge summaries, progress notes — has to be abstracted into structured fields, Wieland-Jorna and colleagues' systematic review of natural language processing systems for extracting information from electronic health records found wide variation in extraction performance across clinical domains, a caution against assuming an NLP pipeline can replace a human abstractor without the same kind of pilot testing and inter-abstractor-style validation this article recommends for manual abstraction.11
Key points
Key points
The reliability of a retrospective chart review depends on its abstraction form, its pilot test and its documented inter-abstractor agreement; its defensibility depends on its data-governance basis and reporting structured according to the STROBE statement.
Practical application
Practical application
- Operational definitions should be written before fields are built. For every variable requiring judgement, the coding rule and preferred chart source should be specified in a written manual before the form is finalised.
- The form should be piloted on difficult charts rather than straightforward ones. Testing should target incomplete or conflicting records, since ambiguous instructions fail first in such cases.
- A pre-specified subset of charts should be abstracted twice. The proportion of charts to be independently double-abstracted should be determined before data collection begins, not after discrepancies are observed.
- The adjudication procedure should be pre-specified. The person responsible for resolving disagreements, and the reference standard (the coding manual), should be named before the first discrepancy occurs.
- Reliability should be reported in addition to results. The agreement rate and adjudication method should be stated in the methods section so that readers can assess data quality directly.
Frequently asked questions
No. Because retrospective designs rely on the interpretation of existing documentation rather than on data collection to a protocol, inter-abstractor checks are, if anything, more important than in prospective designs.
There is no fixed number; the guidance is to include enough charts, and a sufficient number of difficult ones, to reveal ambiguous fields before the form is applied at scale. A small number of straightforward charts will not achieve this.
The value should be coded as missing or unresolved under a pre-specified rule, rather than guessed, and the frequency of unresolved items should be reported.
No. STROBE is a reporting checklist, not a study-design guide: it specifies what a completed observational study should state about its methods, not how to construct the abstraction form or conduct the pilot test. It should therefore be used alongside design-specific guidance, such as the methodology of Gearing et al., rather than as a substitute for it.
De-identification can remove the need for individual patient authorization under the Health Insurance Portability and Accountability Act (HIPAA) if the data meet the de-identification standard in 45 CFR 164.514, but IRB review of the study protocol itself is typically still required; de-identification and IRB approval address different and non-interchangeable obligations.
References
- Gearing RE, Mian IA, Barber J, Ickowicz A. A methodology for conducting retrospective chart review research in child and adolescent psychiatry. J Can Acad Child Adolesc Psychiatry. 2006;15(3):126-134. read 10 October 2026. View on PubMed Central
- Gliklich RE, Leavy MB, Dreyer NA, eds. Registries for Evaluating Patient Outcomes: A User's Guide. 4th ed. Rockville, MD: Agency for Healthcare Research and Quality; 2020. AHRQ Publication No. 19(20)-EHC020. read 10 October 2026. View source
- von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP; STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573-577. read 10 October 2026. View on EQUATOR Network
- U.S. Department of Health and Human Services, Office for Civil Rights. HIPAA Privacy Rule and research: de-identification, authorization waivers, and limited data sets. read 10 October 2026. View source
- Richesson RL, Nadkarni P. Data standards for clinical research data collection forms: current status and challenges. J Am Med Inform Assoc. 2011;18(3):341-346. read 10 October 2026. View on PubMed Central
- Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association. 2013;20(1):144-151. read 10 October 2026. View on PubMed Central
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap): a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. read 10 October 2026. View on PubMed
- Cowie MR, Blomster JI, Curtis LH, et al. Electronic health records to facilitate clinical research. Clinical Research in Cardiology. 2017;106(1):1-9. read 10 October 2026. View on PubMed Central
- Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE, Finkelstein J. The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006;13(1):16-23. read 10 October 2026. View on PubMed Central
- International Organization for Standardization. ISO/IEC 11179-1:2023 Information technology — Metadata registries (MDR) — Part 1: Framework. read 10 October 2026. View source
- Wieland-Jorna Y, van Kooten D, Verheij RA, de Man Y, Francke AL, Oosterveld-Vlug MG. Natural language processing systems for extracting information from electronic health records about activities of daily living: a systematic review. JAMIA Open. 2024;7(2):ooae044. read 10 October 2026. View source
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies. Pubrica. https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/
Pubrica Academy. “Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies.” Pubrica, 10 Oct. 2026, pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/.
Pubrica Academy (2026) Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies. Pubrica. Available at: https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/ (Accessed: 15 October 2026).
Pubrica Academy. “Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies.” Pubrica. October 10, 2026. https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/
Pubrica Academy, ‘Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/> accessed 15 October 2026.
Pubrica Academy. Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/
Pubrica Academy, “Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/
@misc{/la_guidelines_for_data_collection_from_medical_records_for_use_in_retrospective_analyses_2026,
author = {{Pubrica Academy}},
title = {Guidelines for Data Collection from Medical Records in Retrospective Chart Review Studies},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/concepts-definitions/guidelines-for-data-collection-from-medical-records-for-use-in-retrospective-analyses/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Chart-review abstraction readiness checklist
These elements should be confirmed before full-scale abstraction begins.
[ ] Written operational definitions for every judgment-requiring field [ ] Coding manual distributed to all abstractors [ ] Pilot test completed on a sample including difficult/incomplete charts [ ] Ambiguous fields revised and re-tested after pilot [ ] Proportion of charts designated for double abstraction specified [ ] Adjudication process and adjudicator named in advance [ ] Plan for reporting inter-abstractor agreement in the methods section [ ] Data-governance basis (de-identification, IRB waiver, or limited data set) confirmed before CRF finalization [ ] STROBE checklist completed alongside manuscript drafting
Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s methodologists design abstraction forms and coding manuals, conduct pilot testing and establish inter-abstractor reliability checks for retrospective chart review studies.