Home/Insights/Methodology reference/Delphi study methodology
Delphi study methodology
Structured, anonymous, iterative expert consensus. The design stands or falls on decisions that are too often left unreported: who the panel were, what consensus meant, and when to stop.
The method
A Delphi gathers judgement from a panel across successive rounds. Responses are anonymous to one another, each round is fed back in aggregate, and panellists may revise in light of that feedback. It is used where empirical evidence is absent, contested or impossible to obtain.
Four features define it: anonymity between panellists, iteration across rounds, controlled feedback of the group response, and statistical aggregation of the final position.
Anonymity is the point of the design. It removes the dominance effects that shape face-to-face consensus, so a senior voice does not carry a room.
The decisions that must be pre-specified
This is where published Delphi studies most often fail, and where the method is most criticised. Each of the following should be fixed in a protocol before round one and reported in the paper:
Panel definition. What made someone an expert, how they were identified, how many were invited and how many took part. There is no single standard for expertise, which is exactly why the operational definition used must be stated.
Panel size and composition. Whether the panel is homogeneous or deliberately mixed, since heterogeneous panels produce different — and often more defensible — results.
The consensus threshold, defined in advance. Commonly a stated percentage of panellists rating an item in the top band of a Likert scale, frequently in the region of 70–80%, but the figure is a choice and must be justified rather than discovered after the fact.
The stopping rule. Either a fixed number of rounds, or stability — the absence of meaningful change in responses between successive rounds, assessed by a stated criterion. Stopping when the desired level of agreement happens to be reached is not a stopping rule.
Attrition handling. Panellist dropout between rounds changes the group, and response rates per round should be reported.
On the question of quality standards
It is sometimes said that no quality criteria exist for Delphi studies. That is no longer accurate. Guidance for conducting and reporting Delphi work has been published, including CREDES for palliative care research, and the Delphi process used to develop reporting guidelines is itself governed by published method.
What remains true is that no single universally adopted standard governs all Delphi studies across all fields, and that reporting quality in the published literature is uneven. Those are different statements, and the second does not license the first.
Where consensus is and is not evidence
A Delphi produces agreement among the people asked. Agreement is not accuracy: a panel can converge on a wrong answer, and feedback can amplify a majority view rather than improve it.
Reporting the dispersion alongside the central tendency, and recording the items that did not reach consensus, is what keeps the result interpretable.
In practice
Fix, flag or query.
The distinction that keeps an edit defensible: what an editor may change silently, what must be shown to the author, and what only the author can answer.
- Pre-specify and report: consensus threshold, stopping rule, panel definition.
- Report: per-round response rates and attrition, not just the final panel.
- Report: items that failed to reach consensus, not only those that did.
Reporting standards
- CREDES, Guidance on Conducting and REporting DElphi Studies
- EQUATOR Network, for consensus-method reporting guidance
Reporting guidelines named here are hosted by the EQUATOR Network. Writing to the checklist for your design before submission is the cheapest available improvement to a methods section.
Related