Contents
On this page
- Objects of appraisal in a meta-epidemiological study
- ROBIS: risk of bias in the review process
- AMSTAR 2: a confidence rating rather than a score
- RoB 2 and ROBINS-I at the study level
- The intravenous magnesium case as a worked example
- Inferential limits of funnel plot asymmetry tests
- Incorporation of bias assessments into the synthesis
- Transparent reporting of a bias appraisal
- Unresolved methodological questions
- Key points
- Application to your study
- Frequently asked questions
Academy · Research Data Analytics & AI · Bias in meta-epidemiology
Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability
Appraising bias across a body of meta-analyses requires work at three levels simultaneously: the trials within each review, the conduct of each review, and the evidence base on which the reviews drew. RoB 2, ROBIS and AMSTAR 2 each address one of these levels, and none substitutes for another. Selection of the wrong level is the most common methodological error in meta-epidemiological research, and it is usually not apparent in the published report.
A meta-epidemiological study takes meta-analyses or trials as its units of observation and examines which study characteristics are associated with effect estimates. The appraisal question is therefore recursive: it involves judging bias in reviews that themselves judged bias in trials. Three well-validated tools exist and operate at different levels, so the first decision, namely the level at which the research question sits, determines which tool is appropriate and which conclusions are warranted. This article sets out the three levels, the tools belonging to each, and the reliability evidence for all of them.
Learning objectives
- The three levels at which bias can be appraised and the tool appropriate to each
- What ROBIS assesses, how it was developed, and what its piloting showed about reviewer agreement
- Why AMSTAR 2 is not a score, and what its critical domains are for
- How RoB 2 and ROBINS-I differ from their predecessor, and what the reliability studies found
- The interpretation of funnel plot asymmetry, with the intravenous magnesium case as the worked example
- The incorporation of bias assessments into a synthesis rather than their mere tabulation
Common misconceptions
- “AMSTAR 2 gives a quality score.” Shea and colleagues, in the BMJ, designed it to yield an overall confidence rating driven by critical domains, explicitly not an arithmetic score. Summing the items therefore defeats the purpose of the tool.
- “ROBIS and AMSTAR 2 are interchangeable.” ROBIS assesses risk of bias in the review process; AMSTAR 2 assesses methodological quality and yields a confidence rating in the results. They answer related but different questions.
- “Risk-of-bias judgements are objective.” Hartling and colleagues, in the Journal of Clinical Epidemiology, found low reliability both between individual reviewers and across consensus assessments by reviewer pairs. In the piloting of ROBIS, one reviewer pair agreed on 26 of 40 domain ratings (65%).
- “Funnel plot asymmetry means publication bias.” Sterne and colleagues, in the BMJ, set out several possible causes, and in the magnesium case both heterogeneity and a single dominant trial were relevant.
- “Reporting has improved, so older concerns no longer apply.” Dechartres and colleagues assessed 20,920 randomised controlled trials included in Cochrane reviews over time in the BMJ; improvement has been uneven, and some domains remain poorly reported.
Objects of appraisal in a meta-epidemiological study
A meta-epidemiological study treats studies, or whole meta-analyses, as its units of observation and asks whether a design or conduct characteristic is associated with the size of reported effects. Page and colleagues, in PLoS ONE, systematically reviewed this literature to assemble the empirical evidence of study design biases in randomised trials; Bafeta and colleagues, in the BMJ, used the design to examine whether single-centre status inflates estimates in trials with continuous outcomes; Lundh and colleagues’ Cochrane methodology review examined industry sponsorship and research outcome 1,7,17. Murad and Wang set out reporting guidance for meta-epidemiological methodology research in Evidence-Based Medicine, which is the appropriate starting point for researchers planning such a study 6.
Bias can enter at three distinct levels, and the research question determines which level is relevant.
The trial level. Was each included randomised trial conducted and analysed in a way that protects the estimate? The applicable tools are RoB 2 for randomised trials, ROBINS-I for non-randomised studies of interventions, and QUADAS-2 for diagnostic accuracy 2,9,21.
The review level. Was each meta-analysis conducted in a way that protects its summary? The applicable tool is ROBIS.
The confidence level. How much confidence can be placed in the results of each review, given its methods? The applicable tool is AMSTAR 2.
Meta-epidemiological studies commonly err by appraising at one level and concluding at another — for example, grading reviews with AMSTAR 2 and then making claims about the risk of bias in the underlying trials, or applying RoB 2 to primary trials and drawing conclusions about the reviews that pooled them. The conclusion must match the level appraised, and the level must match the question.
Three levels of appraisal, and the tool that belongs to each
| Level | Question it answers | Tool | Unit of assessment | Output | Source |
|---|---|---|---|---|---|
| Trial | Could the conduct or analysis of this trial have biased its result? | RoB 2 | One result within one randomised trial, not the trial as a whole | Domain judgements plus an overall judgement, via signalling questions | Sterne et al, BMJ 2019 |
| Trial, non-randomised | Same question, for non-randomised studies of interventions | ROBINS-I | One result, framed against a hypothetical target trial | Domain and overall judgements | Sterne et al, BMJ 2016 |
| Trial, diagnostic accuracy | Could this accuracy study’s design have biased its estimates? | QUADAS-2 | One accuracy study | Risk of bias and applicability judgements by domain | Whiting et al, Annals of Internal Medicine 2011 |
| Review process | Could the way this review was conducted have biased its summary? | ROBIS | The review, across four domains in phase 2 | Risk of bias in the review, via four signalling questions in phase 3 | Whiting et al, Journal of Clinical Epidemiology 2016 |
| Confidence in the review | How much confidence do this review’s methods warrant in its results? | AMSTAR 2 | The review, with designated critical domains | An overall confidence rating — not a numerical score | Shea et al, BMJ 2017 |
| Certainty of the body of evidence | How certain is the effect estimate for this outcome, across studies? | GRADE | One outcome across the body of evidence | Certainty rating with reasons for downgrading, including risk of bias | Guyatt et al, BMJ 2008 and the Journal of Clinical Epidemiology series |
Characterisations taken from each tool’s own development publication. The tools are not interchangeable and none substitutes for another; a conclusion must be stated at the level that was actually appraised.
ROBIS: risk of bias in the review process
Whiting and colleagues, in the Journal of Clinical Epidemiology, developed ROBIS to assess risk of bias in a systematic review as a process rather than a product 10. It runs in three phases: assessing relevance, identifying concerns with the review process across four domains in phase 2, and judging the overall risk of bias in the review using four signalling questions in phase 3. The four domains cover study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings.
The development process is informative because it indicates what the tool is anchored to. The team drew on 80 MECIR items, of which 46 were classed as bias items, and reviewed 40 existing quality tools 10. The Delphi process ran three rounds with at most 29 respondents per round. In piloting, one reviewer pair agreed on 26 of 40 domain ratings, which is 65%.
This last figure is directly relevant to the methods of any study using the tool. Agreement in the tool’s own development work was moderate; consequently, two independent assessors and a documented resolution process are not optional refinements but are what make a ROBIS assessment reportable. A meta-epidemiological study in which a single person rated every review has an unmeasured and probably substantial source of variation in its principal exposure variable.
Development and piloting of ROBIS. The Delphi process ran three rounds with at most 29 respondents per round. Source: Whiting P, Savovic J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology. 2016;69:225-234.
AMSTAR 2: a confidence rating rather than a score
Shea and colleagues published AMSTAR 2 in the BMJ as a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Its design decision that is most often ignored is that it does not produce a total. Instead, certain items are designated critical — among them protocol registration before commencement, adequacy of the literature search, justification for excluding individual studies, risk of bias from individual studies being included in the synthesis, appropriateness of the meta-analytical methods, and consideration of risk of bias when interpreting results — and the pattern of critical and non-critical weaknesses determines an overall rating of confidence in the review’s results 3.
Converting that rating into a percentage or a count is not a simplification; it produces a different and misleading quantity, because a review can satisfy most items while failing a single critical one. Where AMSTAR 2 serves as an exposure or outcome variable in a meta-epidemiological analysis, authors should use the confidence rating it was designed to produce and should state the critical-domain pattern behind it rather than only the category.
The relationship to ROBIS is complementary. ROBIS asks whether the review process could have biased the result; AMSTAR 2 asks how much confidence the review’s methods warrant in its findings 3,10. Many meta-epidemiological studies use one tool and report conclusions appropriate to the other. Where both questions require an answer, both tools should be applied and reported separately. Our systematic review page covers the conduct side.
RoB 2 and ROBINS-I at the study level
Sterne and colleagues published RoB 2 in the BMJ as a revision of the Cochrane tool that Higgins and colleagues had set out there in 2011. Two changes are important for meta-epidemiological work. The first is that RoB 2 is applied per result rather than per study, so a trial can carry different judgements for different outcomes; consequently, the statement “we assessed risk of bias in the included trials” no longer constitutes a complete description of the assessment 2. The second is that it works through signalling questions leading to domain-level judgements and then an overall judgement, which makes the reasoning visible but also makes the assessment longer. ROBINS-I, also from Sterne and colleagues in the BMJ, extends the approach to non-randomised studies of interventions by framing them against a hypothetical target trial, and QUADAS-2, from Whiting and colleagues in the Annals of Internal Medicine, covers diagnostic accuracy studies 9.
The reliability evidence warrants more attention than it usually receives. Hartling and colleagues, in the Journal of Clinical Epidemiology, found low reliability between individual reviewers and across consensus assessments of reviewer pairs when testing the original risk-of-bias tool 15,23. Savović and colleagues, in Systematic Reviews, evaluated that tool and reported on its application; Jørgensen and colleagues, in the same journal, examined it again. Page and Higgins, also in Systematic Reviews, re-examined how risk of bias due to selective reporting should be assessed at all.
The implication for a meta-epidemiological design is important: if the exposure variable is a risk-of-bias judgement, it carries measurement error, and that error attenuates associations. Inter-rater agreement should be reported, and investigators should consider whether a reliability substudy belongs in the design. Mansournia and colleagues’ exchange in Epidemiology between trialists and epidemiologists explains why the two traditions label these biases differently, which is itself a source of inconsistency across this literature 4.
The intravenous magnesium case as a worked example
The clearest published illustration of how bias assessment at the evidence-base level should operate is the intravenous magnesium case, reported by Sterne and colleagues in the BMJ. Early small trials suggested benefit in acute myocardial infarction, and the pooled result appeared favourable. The subsequent large trials, however, did not agree.
Four figures summarise the case. Heterogeneity in the meta-analysis was substantial, with I-squared of 68% 22. The single ISIS-4 trial received about 90% of the fixed-effect weight, meaning the fixed-effect summary was largely one trial. Tests for funnel plot asymmetry were strongly positive — the Harbord test gave P less than 0.001 and the Peters test P equal to 0.002 — though the two tests disagreed in magnitude because they model the effect-precision relationship differently 22. And the subsequent MAGIC trial reported an odds ratio of 1.0 with a 95% confidence interval of 0.8 to 1.1, contradicting the earlier pooled benefit 22.
The value of this case for teaching lies in the fact that no single diagnostic would have provided the warning. The asymmetry tests flagged a problem, but asymmetry has several possible causes. The heterogeneity statistic indicated that the studies disagreed, but not why. The weight distribution indicated that the summary rested on one trial. Considered together, these signals pointed in the same direction; considered individually, each could be dismissed. A meta-epidemiological appraisal should examine all of them, together with the funnel plot itself: Sterne and colleagues note that if no bias or heterogeneity is present, about 95% of studies should lie inside the triangle defined by the fixed-effect estimate plus or minus 1.96 standard errors.
The intravenous magnesium case. Source: Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002.
Inferential limits of funnel plot asymmetry tests
Because meta-epidemiological studies often use asymmetry test results as a variable, two constraints from Sterne and colleagues’ BMJ recommendations must be respected rather than merely acknowledged.
First, tests for funnel plot asymmetry should generally not be used when a meta-analysis contains fewer than 10 studies, and the minimum may be substantially higher when heterogeneity is substantial 22. When asymmetry test results are coded across a sample of published meta-analyses, a large proportion of those meta-analyses should not have run the test at all, and treating their non-significant results as evidence of no bias imports their error into the meta-epidemiological study 22.
Second, the choice of test depends on heterogeneity: the Harbord, Peters and Rücker tests are advised when the estimated between-study variance of log odds ratios is below 0.1, and above that only the random-effects arcsine test performs reasonably well 22,27. Harbord, Egger and Sterne, and Rücker, Schwarzer and Carpenter, both published their tests in Statistics in Medicine; the original regression test is Egger and colleagues in the BMJ, and Begg and Mazumdar’s rank correlation test is in Biometrics.
Beyond the statistical methods, direct evidence on reporting bias is stronger than any funnel plot. Turner and colleagues, in the New England Journal of Medicine, compared published antidepressant trial results against the regulatory record 26. Chan and colleagues, in JAMA, compared published outcomes against protocols. Those designs identify the bias rather than inferring it, and a meta-epidemiological study that can access registries or regulatory submissions should prefer them 26,34. Our statistical tests in meta-analysis page covers the tests themselves in more detail.
Incorporation of bias assessments into the synthesis
The most common finding about risk-of-bias assessment is that it is performed and then disregarded. Hopewell and colleagues, in BMJ Open, examined how assessments of risk of bias in primary studies are incorporated into systematic reviews of randomised trials 14. Katikireddi, Egan and Petticrew, in the Journal of Epidemiology and Community Health, asked the same question about how reviews bring those assessments into the synthesis of evidence. The pattern both describe is a table of judgements followed by a synthesis that proceeds as though every study were sound 12,14.
Only a few defensible approaches to using the assessments exist, and all of them must be pre-specified. Restricting the primary analysis to studies at low risk of bias is one. Stratifying by risk of bias and comparing the strata is another, and it is the most informative in a meta-epidemiological context, since the comparison constitutes the finding. Using risk of bias to downgrade certainty in the conclusion is the GRADE approach, set out by Guyatt and colleagues in the BMJ and developed across their Journal of Clinical Epidemiology series 19,28. A results section in which the appraisal has no consequence is not defensible.
Wood and colleagues’ BMJ analysis of whether the impact of bias depends on outcome type is relevant here, because it indicates that the effect of a given bias is not constant and that stratification should take this into account 25. Deeks, Macaskill and Irwig, in the Journal of Clinical Epidemiology, make the analogous point for diagnostic test accuracy reviews, where the funnel plot behaves differently again.
Transparent reporting of a bias appraisal
The methods paragraph below determines whether an appraisal functions as evidence or merely as decoration. The weak version names tools without levels, assessors or consequences; the strong version is longer and makes explicit commitments.
The same appraisal, reported two ways
Decoration
The quality of the included systematic reviews was assessed using AMSTAR 2 and risk of bias was assessed using the Cochrane tool. Reviews scoring below 70% were considered low quality. Publication bias was assessed using Egger’s test. Results are presented in Table 2.
Which level? AMSTAR 2 and a trial-level tool are mixed in one sentence, so the conclusion cannot be matched to what was appraised.
“Scoring below 70%” — AMSTAR 2 does not produce a score. This threshold is an artefact of summing items, and the tool’s critical domains are lost in it.
Who assessed, and did they agree? Unstated, in a design where the judgement is the exposure variable and is known to be unreliable.
What did the appraisal change? Nothing: it went into Table 2 and the synthesis continued unaffected.
Egger’s test on how many studies? If the source reviews were small, their non-significant results do not constitute evidence of no bias, and that error is now incorporated into the dataset.
Evidence
Appraisal was conducted at two levels and reported separately. Each included review was assessed with ROBIS for risk of bias in the review process, using the four phase 2 domains and the phase 3 signalling questions, and with AMSTAR 2 for confidence in its results; AMSTAR 2 is reported as the overall confidence rating it was designed to produce, with the critical-domain pattern behind each rating given in full, and no numerical score was computed. Two assessors rated every review independently and disagreements were resolved by a third; agreement before resolution is reported as a percentage by domain, because the tool’s own piloting found one reviewer pair agreeing on 26 of 40 domain ratings. Risk of bias in the primary trials was not re-assessed and no conclusion about trial-level bias is drawn. Pre-specified analyses stratify the association of interest by ROBIS judgement, since that comparison is the study’s object rather than a sensitivity check. Where a review reported a funnel plot asymmetry test on fewer than 10 studies, that result was coded as not interpretable rather than as absence of asymmetry.
The two levels are named, kept separate, and the trial level is explicitly excluded from the conclusions.
AMSTAR 2 produces the rating it was built to produce, with the critical-domain pattern reported so a reader can see what drove it.
Two independent assessors with a resolution rule, and agreement reported — which is what the reliability evidence requires.
The bias judgement is used as a stratifying variable in a pre-specified analysis, so it has a consequence.
Uninterpretable asymmetry tests in the source reviews are coded as such, preventing an imported error.
The second version makes fewer claims and reports more. Three additions account for most of the improvement: separating the levels so that the conclusion matches the appraisal, reporting inter-rater agreement because these judgements are known to be unreliable, and giving the appraisal a pre-specified role in the analysis. The deletion of the “70%” threshold is the single most consequential change, because a score computed from AMSTAR 2 items is a different quantity from the confidence rating the tool produces.
Unresolved methodological questions
Reliability. This is the principal weakness of the entire enterprise. Hartling and colleagues found low reliability between individual reviewers and across consensus assessments of reviewer pairs; ROBIS’s own piloting had one pair agreeing on 26 of 40 domain ratings 10. No tool in current use has demonstrated the reliability expected of a measurement instrument, and meta-epidemiological studies that use these judgements as variables inherit that imprecision. Agreement should be reported rather than assumed.
Whether reporting has improved. Dechartres and colleagues assessed reporting quality and methodological adequacy over time across 20,920 randomised controlled trials included in Cochrane reviews, published in the BMJ 5. Improvement has been evident in some domains and slow in others; consequently, a meta-epidemiological study pooling across decades should treat publication year as a variable rather than a nuisance 5.
Selective reporting. Page and Higgins, in Systematic Reviews, argued that the standard approach to assessing bias due to selective reporting needs rethinking, and the question of how to assess it when protocols are unavailable remains open 11. Where a protocol or registry entry cannot be accessed, the domain should be reported as not assessable rather than rated as low risk.
Terminology. Mansournia and colleagues’ conversation between trialists and epidemiologists in Epidemiology documents genuine disagreement about what the biases are called and how they are individuated. Across a sample of published meta-analyses, the same phenomenon is often labelled differently, and harmonising those labels is a judgement that should be described rather than performed silently. Our biostatistics team most often addresses this issue at the stage at which a coding frame is being built.
Key points
Key points
Bias can be appraised at the trial level, the review-process level or the confidence level, and the tools are not interchangeable: RoB 2 and ROBINS-I apply to studies, ROBIS to the review process, AMSTAR 2 to confidence in the results of a review, and GRADE to certainty in a body of evidence. Conclusions should be stated at the level that was actually appraised. AMSTAR 2 yields a confidence rating driven by critical domains, never a score. Because these judgements are imperfectly reliable (in the piloting of ROBIS, a reviewer pair agreed on 26 of 40 domain ratings), two independent assessors should be used and their agreement reported. The appraisal should also have a pre-specified consequence in the analysis rather than being presented only as a separate table.
Application to your study
Application to your study
- The level to which the research question relates should be recorded. The level may be the trial, the review process or confidence in review results; this single decision determines the tool and delimits the conclusion.
- ROBIS and AMSTAR 2 should be used together only when both questions require an answer, and should be reported separately. The two tools are complementary rather than redundant.
- An AMSTAR 2 score should never be computed. The overall confidence rating should be reported together with the pattern of critical-domain weaknesses that produced it.
- Two independent assessors should be used, and agreement should be reported by domain. The reliability evidence makes this practice a requirement rather than a refinement.
- RoB 2 should be applied to each result rather than to each study. A trial may receive different judgements for different outcomes, so the outcome assessed should be specified.
- The role of the appraisal in the analysis should be pre-specified. Options include restricting the primary analysis, stratifying or downgrading certainty; one option should be selected and stated before the data are examined.
- Uninterpretable asymmetry tests in source reviews should be recoded. When fewer than 10 studies are available (a threshold that rises where heterogeneity is substantial), a non-significant test does not constitute evidence of the absence of bias.
- Publication year should be treated as a variable. Reporting has changed unevenly across 20,920 Cochrane-included trials, so pooling across decades without accounting for publication year conceals a genuine gradient.
- Comparison with registry or regulatory data is preferable where such data are available. These designs identify reporting bias directly rather than inferring it from a plot.
Frequently asked questions
They answer related but distinct questions, and the distinction is the one most often collapsed. ROBIS, from Whiting and colleagues in the Journal of Clinical Epidemiology, assesses risk of bias in the review process: it runs in three phases, with four domains of concern in phase 2 and four signalling questions in phase 3, and asks whether the way a review was conducted could have biased its summary. AMSTAR 2, from Shea and colleagues in the BMJ, assesses methodological quality and yields an overall rating of confidence in the review’s results, driven by designated critical domains. If the question is whether a review’s process was sound, ROBIS is appropriate; if it is how much confidence its findings warrant, AMSTAR 2 is appropriate. Using both is reasonable provided that they are reported separately.
No; doing so changes what is being measured. AMSTAR 2 was deliberately designed not to produce an arithmetic total: certain items are designated critical — protocol registration before commencement, adequacy of the search, justification for excluding studies, risk of bias from individual studies being included in the synthesis, appropriateness of the meta-analytical methods, and consideration of risk of bias when interpreting results — and the pattern of critical and non-critical weaknesses determines an overall confidence rating. A review can satisfy most items while failing a single critical one, which a score obscures entirely. Thresholds such as “below 70% was considered low quality” are therefore not a simplification of the tool but a departure from it.
They are less reliable than their use as data implies, and this is the principal limitation of the field. Hartling and colleagues, in the Journal of Clinical Epidemiology, found low reliability between individual reviewers and across consensus assessments of reviewer pairs when testing the Cochrane risk-of-bias tool. ROBIS’s own piloting had one reviewer pair agreeing on 26 of 40 domain ratings, which is 65%. Savović and colleagues and Jørgensen and colleagues, both in Systematic Reviews, examined the tool further. For a meta-epidemiological study this is important because measurement error in an exposure variable attenuates the association under investigation; agreement should therefore be reported, and a reliability substudy should be considered.
It indicates a relationship between estimated effect and precision in the set of studies, which has several possible causes besides reporting bias — genuine heterogeneity, differences in conduct between small and large trials, or mathematical properties of the effect measure. Sterne and colleagues, in the BMJ, note that if no bias or heterogeneity is present, about 95% of studies should lie inside the triangle defined by the fixed-effect estimate plus or minus 1.96 standard errors. They also constrain its use: tests should generally not be applied below 10 studies, and the minimum may be substantially higher when heterogeneity is substantial. The appropriate report names asymmetry and its candidate explanations, not publication bias.
Because it shows that no single diagnostic would have provided the warning, whereas several considered together would have. Sterne and colleagues report that heterogeneity in the meta-analysis was substantial at an I-squared of 68%, that the single ISIS-4 trial received about 90% of the fixed-effect weight, and that asymmetry tests were strongly positive, with the Harbord test at P below 0.001 and the Peters test at P equal to 0.002. The subsequent MAGIC trial then reported an odds ratio of 1.0 with a 95% confidence interval of 0.8 to 1.1, contradicting the earlier pooled benefit. Individually, each signal could be dismissed; collectively, they pointed in one direction, and the large trial confirmed it.
The approach should be decided before the data are examined, and one of three defensible approaches should be selected. The primary analysis may be restricted to studies at low risk of bias. Alternatively, studies may be stratified by risk-of-bias judgement and compared — in meta-epidemiological work this comparison is usually the study’s object rather than a sensitivity check. Finally, risk of bias may be used to downgrade certainty via GRADE, following Guyatt and colleagues. Hopewell and colleagues in BMJ Open and Katikireddi and colleagues in the Journal of Epidemiology and Community Health both document the alternative pattern: a table of judgements followed by a synthesis that proceeds as though every study were sound. This outcome should be avoided.
To each result. This change from the original Cochrane tool most often goes unnoticed in reporting: RoB 2, as published by Sterne and colleagues in the BMJ, is applied per result, so one trial can carry a different judgement for its primary outcome than for a secondary one, and for an intention-to-treat effect than for a per-protocol effect. The practical consequence is that “risk of bias was assessed in the included trials” is not a complete description of the assessment; authors must state which result in each trial was assessed, and it should be the result used in the synthesis.
Only partly and unevenly. Dechartres and colleagues, in the BMJ, assessed reporting quality and methodological adequacy over time across 20,920 randomised controlled trials included in Cochrane reviews. Improvement has been evident in some domains and slow in others, which has two implications for a meta-epidemiological design. First, publication year should enter the model as a variable rather than being treated as a nuisance, because it is correlated with both reporting and conduct. Second, poor reporting is not equivalent to poor conduct, and where a domain is unreported the appropriate coding is that it could not be assessed, not that it was at low risk.
References
- Lundh A, Lexchin J, Mintzes B, Schroll JB, Bero L. Industry sponsorship and research outcome. Cochrane Database of Systematic Reviews. 2017;2:MR000033. read 9 October 2026. eyes.cochrane.org
- Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Hernan MA, Hopewell S, Hrobjartsson A, Junqueira DR, Juni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. doi:10.1136/bmj.l4898 read 9 October 2026. doi.org/10.1136/bmj.l4898
- Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. doi:10.1136/bmj.j4008 read 9 October 2026. doi.org/10.1136/bmj.j4008
- Mansournia MA, Higgins JPT, Sterne JAC, Hernan MA. Biases in randomized trials: a conversation between trialists and epidemiologists. Epidemiology. 2017;28:54-59. doi:10.1097/EDE.0000000000000564 read 9 October 2026. doi.org/10.1097/EDE.0000000000000564
- Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, Perrodeau E, Altman DG, Ravaud P. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews. BMJ. 2017;357:j2490. doi:10.1136/bmj.j2490 read 9 October 2026. doi.org/10.1136/bmj.j2490
- Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence-Based Medicine. 2017;22(4):139-142. read 9 October 2026. equator-network.org
- Page MJ, Higgins JPT, Clayton G, Sterne JAC, Hrobjartsson A, Savovic J. Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies. PLoS ONE. 2016;11:e0159267. doi:10.1371/journal.pone.0159267 read 9 October 2026. doi.org/10.1371/journal.pone.0159267
- Jorgensen L, Paludan-Muller AS, Laursen DR, Savovic J, Boutron I, Sterne JAC, Higgins JPT, Hrobjartsson A. Evaluation of the Cochrane tool for assessing risk of bias in randomized clinical trials. Systematic Reviews. 2016;5:80. doi:10.1186/s13643-016-0259-8 read 9 October 2026. doi.org/10.1186/s13643-016-0259-8
- Sterne JAC, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hrobjartsson A, Kirkham J, Juni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schunemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JPT. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. doi:10.1136/bmj.i4919 read 9 October 2026. doi.org/10.1136/bmj.i4919
- Whiting P, Savovic J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, Davies P, Kleijnen J, Churchill R. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. Journal of Clinical Epidemiology. 2016;69:225-234. doi:10.1016/j.jclinepi.2015.06.005 read 9 October 2026. doi.org/10.1016/j.jclinepi.2015.06.005
- Page MJ, Higgins JPT. Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Systematic Reviews. 2016;5:108. doi:10.1186/s13643-016-0289-2 read 9 October 2026. doi.org/10.1186/s13643-016-0289-2
- Katikireddi SV, Egan M, Petticrew M. How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? Journal of Epidemiology and Community Health. 2015;69:189-195. doi:10.1136/jech-2014-204711 read 9 October 2026. doi.org/10.1136/jech-2014-204711
- Savovic J, Weeks L, Sterne JAC, Turner L, Altman DG, Moher D, Higgins JPT. Evaluation of the Cochrane Collaboration's tool for assessing the risk of bias in randomized trials. Systematic Reviews. 2014;3:37. doi:10.1186/2046-4053-3-37 read 9 October 2026. doi.org/10.1186/2046-4053-3-37
- Hopewell S, Boutron I, Altman DG, Ravaud P. Incorporation of assessments of risk of bias of primary studies in systematic reviews of randomised trials. BMJ Open. 2013;3:e003342. doi:10.1136/bmjopen-2013-003342 read 9 October 2026. doi.org/10.1136/bmjopen-2013-003342
- Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. Journal of Clinical Epidemiology. 2013;66:973-981. doi:10.1016/j.jclinepi.2012.07.005 read 9 October 2026. doi.org/10.1016/j.jclinepi.2012.07.005
- Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One. 2012;7:e35621. read 9 October 2026. trialsjournal.biomedcentral.com
- Bafeta A, Dechartres A, Trinquart L, Yavchitz A, Boutron I, Ravaud P. Impact of single centre status on estimates of intervention effects in trials with continuous outcomes: meta-epidemiological study. BMJ. 2012;344:e813. doi:10.1136/bmj.e813 read 9 October 2026. doi.org/10.1136/bmj.e813
- Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, Als-Nielsen B, Balk EM, Gluud C, Gluud LL, Ioannidis JPA, Schulz KF, Beynon R, Welton NJ, Wood L, Moher D, Deeks JJ, Sterne JAC. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine. 2012;157(6):429-438. doi:10.7326/0003-4819-157-6-201209180-00537 read 9 October 2026. doi.org/10.7326/0003-4819-157-6-201209180-00537
- Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schunemann HJ. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology. 2011;64:383-394. doi:10.1016/j.jclinepi.2010.04.026 read 9 October 2026. doi.org/10.1016/j.jclinepi.2010.04.026
- Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. 2011;342:d549. doi:10.1136/bmj.d549 read 9 October 2026. doi.org/10.1136/bmj.d549
- Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MMG, Sterne JAC, Bossuyt PMM. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155:529-536. doi:10.7326/0003-4819-155-8-201110180-00009 read 9 October 2026. doi.org/10.7326/0003-4819-155-8-201110180-00009
- Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rucker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JPT. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. doi:10.1136/bmj.d4002 read 9 October 2026. doi.org/10.1136/bmj.d4002
- Higgins JPT, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JAC. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi:10.1136/bmj.d5928 read 9 October 2026. doi.org/10.1136/bmj.d5928
- Hopewell S, Dutton S, Yu LM, Chan AW, Altman DG. The quality of reports of randomised trials in 2000 and 2006: comparative study of articles indexed in PubMed. BMJ. 2010;340:c723. read 9 October 2026. trialsjournal.biomedcentral.com
- Wood L, Egger M, Gluud LL, Schulz KF, Juni P, Altman DG, et al. BMJ. 2008;336:601-605. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. New England Journal of Medicine. 2008;358:252-260. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Rucker G, Schwarzer G, Carpenter J. Statistics in Medicine. 2008;27:746-763. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schunemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336:924-926. doi:10.1136/bmj.39489.470347.AD read 9 October 2026. doi.org/10.1136/bmj.39489.470347.AD
- Leeflang MMG, Deeks JJ, Gatsonis C, Bossuyt PMM; Cochrane Diagnostic Test Accuracy Working Group. Systematic reviews of diagnostic test accuracy. Annals of Internal Medicine. 2008;149(12):889-897. doi:10.7326/0003-4819-149-12-200812160-00008 read 9 October 2026. doi.org/10.7326/0003-4819-149-12-200812160-00008
- Harbord RM, Egger M, Sterne JA. Statistics in Medicine. 2006;25:3443-3457. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, Gaboury I. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Medical Journal of Australia. 2006;185:263-267. read 9 October 2026. trialsjournal.biomedcentral.com
- Le Henanff A, Giraudeau B, Baron G, Ravaud P. Quality of reporting of noninferiority and equivalence randomized trials. JAMA. 2006;295:1147-1151. read 9 October 2026. trialsjournal.biomedcentral.com
- Deeks JJ, Macaskill P, Irwig L. Journal of Clinical Epidemiology. 2005;58:882-893. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. JAMA. 2004;291:2457-2465. read 9 October 2026. lshtm-eprints-repo-prod.lshtm-eprints.cdl.cosector.com
- Steichen TJ. Nonparametric trim and fill analysis of publication bias in meta-analysis (metatrim). Stata Technical Bulletin. 2003. read 9 October 2026. link.springer.com
- Steichen TJ. Tests for publication bias in meta-analysis (metabias). Stata Journal. 2003. read 9 October 2026. link.springer.com
- Higgins JPT, Thompson SG. Quantifying heterogeneity in a meta-analysis. Statistics in Medicine. 2002;21(11):1539-1558. doi:10.1002/sim.1186 read 9 October 2026. doi.org/10.1002/sim.1186
- Eysenck HJ. Meta-analysis and its problems. BMJ. 1994;309:789-792. read 9 October 2026. link.springer.com
- Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50(4):1088-1101. doi:10.2307/2533446 read 9 October 2026. doi.org/10.2307/2533446
- DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7(3):177-188. doi:10.1016/0197-2456(86)90046-2 read 9 October 2026. doi.org/10.1016/0197-2456(86)90046-2
- Fleiss JL. Measuring nominal scale agreement among many raters. Psychological Bulletin. 1971;76:378-382. read 9 October 2026. trialsjournal.biomedcentral.com
- Cochran WG. The combination of estimates from different experiments. Biometrics. 1954;10(1):101-129. doi:10.2307/3001666 read 9 October 2026. doi.org/10.2307/3001666
- Veroniki AA, Jackson D, Bender R, Kuss O, Langan D, Higgins JPT, Knapp G, Salanti G. Methods to calculate uncertainty in the estimated overall effect size from a random-effects meta-analysis. doi:10.1002/jrsm.1319 read 9 October 2026. doi.org/10.1002/jrsm.1319
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 9). Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability. Pubrica. https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/
Pubrica Academy. “Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability.” Pubrica, 9 Oct. 2026, pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/.
Pubrica Academy (2026) Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability. Pubrica. Available at: https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/ (Accessed: 15 October 2026).
Pubrica Academy. “Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability.” Pubrica. October 9, 2026. https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/
Pubrica Academy, ‘Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability’ (Pubrica, 9 October 2026) <https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/> accessed 15 October 2026.
Pubrica Academy. Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability. [Internet]. Pubrica; 2026 Oct 9 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/
Pubrica Academy, “Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability,” Pubrica, Oct. 9, 2026. [Online]. Available: https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/
@misc{how_to_evaluate_bias_in_meta_analysis__2026,
author = {{Pubrica Academy}},
title = {Evaluating Bias in Meta-Analysis Within Meta-Epidemiological Studies: Levels, Tools and Reliability},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/meta-analysis/how-to-evaluate-bias-in-meta-analysis-within-meta-epidemiological-studies/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Bias appraisal specification for a meta-epidemiological study
This specification should be completed before any review or trial is rated. Section 1 prevents the most common error in this literature, namely appraising at one level and concluding at another.
BIAS APPRAISAL SPECIFICATION
Study: ______________________ Version / date: ______________
1. LEVEL AND CONCLUSION BOUNDARY
Our research question is about bias at the level of:
[ ] individual trials
[ ] the review process
[ ] confidence in review results
[ ] certainty in a body of evidence for an outcome
The conclusion we will be entitled to draw, written out:
____________________________________________________
Levels we will NOT draw conclusions about: ____________
2. TOOLS
[ ] RoB 2 applied per RESULT; result assessed: ______
[ ] ROBINS-I target trial specified: _______________
[ ] QUADAS-2
[ ] ROBIS phase 2 domains and phase 3 questions used as published
[ ] AMSTAR 2 reported as the overall confidence rating
[ ] no numerical score will be computed
[ ] critical-domain pattern reported per review
[ ] GRADE outcome(s): __________________________
3. ASSESSORS AND AGREEMENT
Number of independent assessors per unit: ____
Resolution rule for disagreements: ___________________
Agreement to be reported: [ ] overall [ ] by domain
Statistic used: ______________________________________
Reliability substudy planned? [ ] yes [ ] no
4. CODING RULES FOR SOURCE REVIEWS
If a review reported no risk-of-bias assessment: ______
If a domain was unreported: code as NOT ASSESSABLE
(not as low risk) [ ] agreed
If a review ran an asymmetry test on fewer than 10
studies: code as NOT INTERPRETABLE [ ] agreed
If a review computed an AMSTAR score: _________________
Terminology harmonisation across source reviews, and
who decides: _______________________________________
5. HOW THE APPRAISAL WILL BE USED (pre-specify ONE primary route)
[ ] restrict primary analysis to low risk of bias
[ ] stratify the association by bias judgement
[ ] downgrade certainty via GRADE
Secondary / sensitivity uses: _________________________
[ ] the appraisal will not appear only as a table
6. TIME AND CONTEXT VARIABLES
[ ] publication year included as a variable
Other context variables (centre count, funding source,
registration status): ______________________________
7. DIRECT EVIDENCE OF REPORTING BIAS, WHERE AVAILABLE
[ ] registry entries compared with publications
[ ] protocols obtained and compared
[ ] regulatory submissions obtained
If none available, state so as a limitation: __________
8. REPORTING CHECKS BEFORE SUBMISSION
[ ] every conclusion is stated at the level appraised
[ ] inter-rater agreement reported
[ ] no AMSTAR 2 score anywhere in the manuscript
[ ] asymmetry described as asymmetry, with candidate causes
[ ] reliability of these judgements named as a limitation
[ ] reporting follows the guidance for meta-epidemiological
methodology research Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s review and biostatistics teams establish the coding frame for meta-epidemiological research before rating begins: the level at which the question sits, the tool that follows from it, the coding of uninterpretable results in the source reviews, and the role of the appraisal in the analysis. The teams conduct dual independent assessment with documented resolution and report agreement. For completed manuscripts, they check for the error most frequently identified by reviewers in this field: a conclusion drawn at a level that was never appraised.