Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one

Home/Insights/Methodology reference/Evaluating scientific research

Evaluating scientific research

Critical appraisal asks whether a study’s results are believable, which is a different question from whether they are interesting.

Foundations

The three questions

Is it valid? Does the design support the claim, was it executed as designed, and what biases remain?

What are the results? The size of the effect and its precision, not the presence of a p-value.

Do they apply here? To these patients, this setting, this comparison.

Design-specific appraisal tools

Appraisal is not a general skill applied uniformly; the tool follows the design. Cochrane’s RoB 2 assesses risk of bias in randomised trials across domains including randomisation, deviations from intended intervention, missing outcome data, outcome measurement and selective reporting. ROBINS-I does the equivalent for non-randomised studies of interventions, and QUADAS-2 for diagnostic accuracy studies.

For a body of evidence rather than a single study, GRADE rates certainty across risk of bias, inconsistency, indirectness, imprecision and publication bias, producing a rating from high to very low.

What metrics do not tell you

Journal impact factor is a property of a journal, not of a paper, and its distribution is heavily skewed — most papers in a high-impact journal are cited far below its mean. Citation count measures attention, including critical attention.

Neither is a substitute for reading the methods. A paper is appraised by its design and conduct, not by where it appeared.

In practice

Fix, flag or query.

The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.

  • Match the tool to the design: RoB 2, ROBINS-I or QUADAS-2.
  • Rate the body of evidence, not just the study: GRADE.
  • Do not appraise by: journal impact factor or citation count.

Reporting standards

  • Cochrane RoB 2 and ROBINS-I
  • GRADE Working Group
  • QUADAS-2, for diagnostic accuracy studies

Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.