Home/Insights/Methodology reference/Evaluating scientific research
Evaluating scientific research
Critical appraisal asks whether a study’s results are believable, which is a different question from whether they are interesting.
The three questions
Is it valid? Does the design support the claim, was it executed as designed, and what biases remain?
What are the results? The size of the effect and its precision, not the presence of a p-value.
Do they apply here? To these patients, this setting, this comparison.
Design-specific appraisal tools
Appraisal is not a general skill applied uniformly; the tool follows the design. Cochrane’s RoB 2 assesses risk of bias in randomised trials across domains including randomisation, deviations from intended intervention, missing outcome data, outcome measurement and selective reporting. ROBINS-I does the equivalent for non-randomised studies of interventions, and QUADAS-2 for diagnostic accuracy studies.
For a body of evidence rather than a single study, GRADE rates certainty across risk of bias, inconsistency, indirectness, imprecision and publication bias, producing a rating from high to very low.
What metrics do not tell you
Journal impact factor is a property of a journal, not of a paper, and its distribution is heavily skewed — most papers in a high-impact journal are cited far below its mean. Citation count measures attention, including critical attention.
Neither is a substitute for reading the methods. A paper is appraised by its design and conduct, not by where it appeared.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Match the tool to the design: RoB 2, ROBINS-I or QUADAS-2.
- Rate the body of evidence, not just the study: GRADE.
- Do not appraise by: journal impact factor or citation count.
Reporting standards
- Cochrane RoB 2 and ROBINS-I
- GRADE Working Group
- QUADAS-2, for diagnostic accuracy studies
Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.
Related