Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · F-Scores in Data Extraction: Precision, Recall and the Evaluation of Automated Tools

F-Scores in Data Extraction: Precision, Recall and the Evaluation of Automated Tools

An F-score combines two distinct types of error in a single figure: data the tool should have identified but missed, and incorrect data the tool proposed. Interpreting an F-score therefore requires these two components to be separated. This article defines the measures and explains how reported scores should be read.

Pubrica AcademyWritten by a data-analytics reviewer Reviewed bya systematic-review methodologist Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources2 peer-reviewed

The F-score (most often reported as F1) is the harmonic mean of precision and recall, used to summarise, in a single number, how well an automated or semi-automated extraction tool performed against a set of fields whose correct values are already known.

Learning objectives

  • The exact definitions of precision, recall and F1, and why F1 uses the harmonic mean rather than a simple average.
  • Why F1 can hide very different error profiles behind the same number.
  • What F1 scores reported for text-mining and extraction tools in the systematic-review literature indicate for routine use.

Common misconceptions

  • “A higher F-score always indicates a better tool for a given task.” Two tools may report the same F1 while one misses rare but important fields (low recall) and the other proposes incorrect values (low precision); which type of error matters depends on the review workflow.
  • “F1 is equivalent to accuracy.” Accuracy counts correct predictions over all predictions, including true negatives; F1 is derived only from precision and recall and ignores true negatives entirely, which is consequential when most fields are correctly left blank.
  • “An F-score reported in a tool’s validation paper will transfer to other datasets.” F-scores are specific to the test set on which they were measured; a tool’s performance on a different document format, field type or disease area is not guaranteed to be equivalent.

Precision, recall and the use of the harmonic mean in F1

Precision is the proportion of values proposed by the tool that were correct: true positives divided by (true positives plus false positives). Recall is the proportion of correct values that the tool identified: true positives divided by (true positives plus false negatives). David Powers, in a widely cited methodological treatment of these measures, explains why F1, the harmonic mean of precision and recall, 2×(precision×recall)/(precision+recall), is used rather than a simple arithmetic mean: the harmonic mean penalises an extreme imbalance between precision and recall more heavily than an arithmetic mean would, so a tool cannot inflate its F1 by trading one measure against the other without limit1.

Different error profiles underlying identical F1 scores

Because F1 compresses two numbers into one, two tools with identical F1 scores may have reached them in very different ways: one with high precision and lower recall (conservative, missing some true values), the other with high recall and lower precision (identifying most true values but also proposing some incorrect ones). Powers states explicitly that F-measures of this kind should be read alongside their component precision and recall, not substituted for them, because the single number by construction loses information about which type of error is more common1. For a data-extraction workflow, this distinction alters the reviewer’s checking burden: low precision requires more time to reject incorrect suggestions, whereas low recall requires more time to identify manually the values that the tool missed.

Interpretation of F-scores in the extraction literature

The prospective evaluation by Gates and colleagues of a machine-learning and text-mining tool for trial data extraction reports field-level performance figures of this kind: precision and recall (and derived F-measures) that varied by field, consistent with the general pattern that simple, consistently formatted fields score higher than fields requiring interpretation2. When a vendor or methods paper reports a single overall F1 for an extraction tool, the relevant follow-up question is over which fields that average was computed, and what the precision and recall split was for the specific field types required by the review.

Key points

Key points

An F-score is a useful single-number summary for comparing tools on a fixed task, but it combines two different types of error in one figure. The underlying precision and recall, and the fields over which the score was computed, should be obtained before the score is used to judge a tool’s suitability for a specific extraction task.

Practical application

Practical application

  1. Precision and recall should be requested in addition to F1. When a tool reports an F1 score, the underlying precision and recall should be requested or calculated to establish which type of error (false positives or false negatives) predominates.
  2. The tool’s error profile should be matched to the checking workflow. If reviewers are more reliable at detecting incorrect values than missing ones, a tool with higher precision and lower recall may be more suitable than the reverse, even at the same F1.
  3. Performance should be validated on the project’s own field types before a published score is relied upon. A tool’s reported F1 was measured on a specific test set; performance should be checked on a small sample of the project’s own documents and field definitions before reliance.

Frequently asked questions

There is no universal threshold; acceptability depends on the importance of the field, the relative cost of a missed versus an incorrect value, and how the score was measured. Comparisons are meaningful only against the same test set and field definitions.

No. Inter-rater agreement (such as kappa) measures consistency between two human extractors; F1 measures a tool’s output against an established reference set of correct answers. They address related but different questions.

Accuracy treats every correctly skipped field (a true negative) as equally informative as a correctly identified one, which is misleading when most fields in a document are correctly left blank; F1 avoids this by excluding true negatives from its calculation.

References

  1. Powers DMW. Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. J Mach Learn Technol. 2011;2(1):37-63. read 10 October 2026. View preprint on arXiv
  2. Gates A, Gates M, Sim S, Elliott SA, Pillay J, Hartling L. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. BMC Med Res Methodol. 2021;21:169. read 10 October 2026. View on BMC Medical Research Methodology

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). F-Scores in Data Extraction: Precision, Recall and the Evaluation of Automated Tools. Pubrica. https://pubrica.com/academy/concepts-definitions/f-scores-in-data-extraction/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

F-score reporting checklist

For use when quoting or evaluating an F-score for an extraction or classification tool.

[ ] Precision reported alongside F1?
[ ] Recall reported alongside F1?
[ ] Field type(s) or task the score was computed over, stated
[ ] Test-set size and source stated
[ ] Score validated on your own sample before relying on it?

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

When Pubrica’s data-analytics team evaluates or reports on an extraction or classification tool for a project, precision and recall are reported alongside any F-score, and performance is validated against a sample of the project’s own data before reliance.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Interpretation of performance metrics for extraction and classification tools

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.