What Is a Pilot Study? From Concept to Execution
A pilot study asks whether a planned trial can be run, not whether a treatment works. Almost every misuse of the design follows from blurring those two questions: a small study reports a promising effect, the effect is read as preliminary evidence, and a full trial is designed around a number that was never precise enough to design anything.
- How pilot studies and feasibility studies relate, in the definition the CONSORT extension actually uses
- How many participants a pilot needs, what the published rules of thumb say, and what UK studies in practice do
- How to write progression criteria before the data arrive, and how to report the study so a reviewer can use it
- “A pilot study gives you a preliminary effect size to power the main trial with.” This is the single most consequential error in the area. Such estimates “will be imprecise in typically small pilot studies” and “should not therefore play a part in any formal decision-making”.12 The guidance is explicit that using a pilot to estimate the treatment effect is “a practice that should be avoided in general”.10
What a pilot study is
A pilot study is a small version of a planned study, run first, to find out whether the planned study can be carried out as designed. The definition used by the UK National Institute for Health Research states it compactly: pilot studies are “a version of the main study that is run in miniature”, in order “to test whether the components of the main study can all work together”.5 The emphasis falls on process. A pilot “is focused on the processes of the main study”,5 which means recruitment, randomisation, delivery of the intervention, data collection and retention.
Whether a pilot is worth running at all, and what the evidence says it is worth, is the subject of the companion article on why a pilot study is important in research. This article covers how to carry one out.
Two consequences follow, and both are routinely lost.
First, the outcome a pilot reports is a process outcome, not a clinical one. The same guidance is blunt about the boundary: “crucially, feasibility studies do not evaluate the outcome of interest; that is left to the main study”.5 A pilot that reports whether an intervention worked has answered a question it was not built to answer.
Second, a pilot is preparation rather than evidence. Arain and colleagues give the standard formulation: a pilot study is “a version of the main study that is run in miniature to test whether the components of the main study can all work together”.3 The test is of the machinery.
The Medical Research Council guidance on complex interventions places feasibility and piloting as a distinct phase between development and evaluation. Under that phase it lists “testing procedures”, “estimating recruitment and retention” and “determining sample size”.6 The guidance also states what the pilot is for at that point: to “examine the key uncertainties that have been identified during development”.6 An uncertainty that was never identified will not be examined, which is why the development work before the pilot matters as much as the pilot itself.
Pilot and feasibility: one contains the other
The relationship between these two terms is stated differently in almost every methods textbook. The authoritative answer comes from the consensus work that produced the CONSORT extension, and it is not the symmetrical “can it be done versus how will it be done” formulation often repeated.
Eldridge and colleagues define a feasibility study as one that asks “whether the future trial can be done, should be done, and, if so, how”.1 Against that, they place the pilot study inside it: “pilot studies are a subset of feasibility studies”.1 The companion framework paper puts the containment beyond doubt: “all pilot studies are feasibility studies but not all feasibility studies are pilot studies”.2
What distinguishes the subset is scale rather than subject matter. The defining feature of a pilot is that “a future study or part of a future study, is conducted on a smaller scale”.2 Feasibility studies that do not do this “cannot be called pilot studies since no part of the future trial is being conducted on a smaller scale”.2
| Feasibility study | Pilot study | |
|---|---|---|
| Question asked | Whether the future trial “can be done, should be done, and, if so, how”1 | The same question, approached by running the future trial in miniature2 |
| Relationship | The overarching category2 | A subset of it1 |
| Defining feature | May examine single components or parameters without running the trial2 | “A future study or part of a future study, is conducted on a smaller scale”2 |
| Evaluates the clinical outcome? | No — “that is left to the main study”5 | No — studies of either kind “should not be focusing on effectiveness”2 |
| Reporting standard | The same one. The extension states that “we make no distinction in this extension between pilot and feasibility randomised trials”1 | |
The last row carries a practical consequence. Whichever label a study takes, the same checklist applies to it, so there is nothing to be gained by choosing the word that sounds less committal.
This is measured rather than asserted. Reviewing published practice and journal policy, Arain and colleagues found “no clear distinction in use of the terms ‘pilot’ and ‘feasibility’”.3 They observed a pattern rather than a rule: “studies labeled ‘feasibility’ were conducted with more flexible methodology compared to those labeled ‘pilot’”.3 A later systematic review of guidance documents reached the same place, reporting that researchers “view feasibility as an overarching concept” “with pilot studies as a subset of feasibility studies”.8
Internal and external pilots
A second distinction matters more for the analysis than the first, and it concerns what happens to the data afterwards.
In an external pilot, the pilot is a separate study. Its participants do not contribute to the main trial's results. The NIHR guidance describes this ending directly: “at the end of the pilot study the data may be analysed and set aside, a so-called external pilot”.5 Avery and colleagues give the same design in one line: “an ‘external pilot’ is a rehearsal of the main study”.7
In an internal pilot, the first phase of the definitive trial is designated as the pilot. Recruitment continues without a break if the trial proceeds, and “the outcome data generated may contribute to the final analyses”.7 The attraction is efficiency: an internal pilot “provides a valuable opportunity to formally review complex trial processes” without discarding the participants already recruited.7
The cost of that efficiency is that the decision point must be specified in the protocol before recruitment opens, because the data will be reused. That requirement is the subject of progression criteria below. Note also that the conceptual framework discussed above sets internal pilots aside: its authors “do not specifically cover internal pilot studies which are designed as the early stage of a definitive RCT”.2
What a pilot study sets out to establish
The objectives of a pilot are feasibility objectives, and they should be written as such. Thabane and colleagues state the governing principle plainly: “the main goal of pilot studies is to assess feasibility”.4
| Area | The question it answers | What it yields for the main trial |
|---|---|---|
| Recruitment | Can eligible participants be found and enrolled at the rate assumed? | A recruitment rate, and the number of sites or months the main trial will need |
| Randomisation | Does the allocation procedure work in practice, and is it acceptable to staff and participants? | Confirmation that the procedure runs, or the specific point at which it fails |
| Retention | Do participants stay to the follow-up point? | A retention rate, which feeds the main trial's sample size inflation |
| Intervention delivery | Can the intervention be delivered as specified, by the staff who will deliver it? | Fidelity data, and the protocol amendments the main trial will need |
| Outcome measurement | Can the outcome be collected, and is the instrument understood by participants? | Completeness rates, and an estimate of the standard deviation9 |
| Data systems | Do the case report forms, databases and transfers function end to end? | A tested data pipeline before the expensive phase begins |
One entry in the last column deserves separate treatment. A pilot can inform the standard deviation used in the main trial's sample size calculation, and Sim describes this as “an empirical matter” that pilots can legitimately address.12 The treatment effect is not such a matter, and the distinction between the two is the subject of a later section.
Figures: defining and evaluating a pilot study
Figure 1 shows how pilot studies relate to the wider category of feasibility studies. Figure 2 illustrates progression criteria that are set before a pilot begins. Together they help authors describe the study accurately and pre-specify the criteria for moving to a definitive trial.
Feasibility is the overarching category. Pilot studies are the subset in which the future trial, or part of it, is run in miniature.
What to look for. The containment is the point. A feasibility study asks “whether the future trial can be done, should be done, and, if so, how”, and “pilot studies are a subset of feasibility studies”.1 The subset is defined by scale: in a pilot “a future study or part of a future study, is conducted on a smaller scale”, and studies that do not do this “cannot be called pilot studies”.2 The split inside the pilot box is about what happens to the data: an external pilot’s data “may be analysed and set aside”,5 whereas in an internal pilot “the outcome data generated may contribute to the final analyses”.7
Schematic representation of the definitions, not of relative frequency.
Three bands rather than a single threshold. The observed rate falls where it falls; what it means was decided before the study opened.
What to look for. The graded form is the recommendation, not an embellishment: “a traffic light system of green (go), amber (amend) and red (stop)” “might be preferable to a simple stop/go approach”.7 The reporting standard requires the bands to exist beforehand, as “prespecified criteria used to judge whether, or how, to proceed with future definitive trial”,1 and warns against being too precise about where they sit, advising caution over “setting definitive thresholds that could be missed simply due to chance variation”.1 Meeting them “does not automatically guarantee continuation to and throughout the main trial”.7
Schematic. The band widths are illustrative and carry no recommended values.
How to run one, step by step
The sequence below follows the order in which decisions become irreversible. Each step is anchored to the guidance that requires it.
| Step | What it involves | What makes it adequate |
|---|---|---|
| 1. Name the uncertainties | List what is actually unknown about running the trial: the recruitment rate, the acceptability of the intervention, whether the outcome can be collected | The pilot examines “the key uncertainties that have been identified during development”6 |
| 2. Convert them into objectives | Write each uncertainty as a feasibility objective with a measurable outcome attached | The checklist requires “specific objectives or research questions for pilot trial”1; an objective to test effectiveness “is not recommended”1 |
| 3. Set progression criteria | State in advance what result would mean proceed, amend or stop | The extension adds an item for “prespecified criteria used to judge whether, or how, to proceed with future definitive trial”1 |
| 4. Write a protocol that mirrors the main trial | Reproduce the eligibility criteria, allocation procedure, intervention and follow-up schedule at smaller scale | The design is only informative about the main trial to the extent that it resembles it3 |
| 5. Justify the number of participants | Derive the sample size from the feasibility objectives, not from convenience | Item 7a requires a “rationale for numbers in the pilot trial”, since “the criterion of congruency between the objectives and the sample size holds as true for a pilot trial as for any study”1 |
| 6. Obtain ethical approval | Submit for review, and state the preliminary nature of the study in the participant materials | Transparency about feasibility intent in consent documents is currently inadequate and “needs to be specifically addressed by research ethics guidelines”15 |
| 7. Run it, and record what went wrong | Collect the feasibility outcomes, and log protocol deviations, refusals and withdrawals with reasons | The reasons are the finding; a rate without them cannot be acted on14 |
| 8. Analyse descriptively | Report rates and estimates with intervals; do not test hypotheses about effectiveness | Estimates should be reported “without P values” because “pilot trials are not powered for testing hypotheses about effectiveness”1 |
| 9. Decide, and say what changes | Apply the progression criteria, and specify the amendments the main trial will carry | Item 22a requires “implications for progression from pilot to future definitive trial, including any proposed amendments”1 |
How many participants
There is no single answer, and the published recommendations disagree with one another by a factor of three. The disagreement is itself informative, because the recommendations are derived from different objectives.
Bell, Whitehead and Julious survey the field and report the range directly: “there are several rules of thumb for the size of a pilot study, ranging from 12 to 35 individuals per arm”.10 The lower end of that range originates with Julious, the upper end with simulation work on how precisely a standard deviation can be estimated.
The precision argument is the most developed. Teare and colleagues simulated how well key design parameters can be recovered from an external pilot, and recommend “that an external pilot study has at least 70 measured subjects (35 per group)”.9 Their reason is specific: “a total of at least 70 may be necessary for estimating the standard deviation”.9 They also quantify the bias at smaller sizes. The standard deviation “tends to be underestimated”, although “once the total sample size is above 50 the average bias becomes negligible”.9 The warning they attach is the part most often dropped: “considerable sampling variation remains even with a large sample size”.9
A second family of recommendations scales the pilot to the main trial rather than fixing it. Whitehead and colleagues “proposed stepped rules of thumb for pilot studies based on the target effect size and the size of the future trial”, under which “the number of patients per arm for the pilot study should be 25 for 90% power” in the worked case Bell and colleagues report.10 Cocks and Torgerson “also recommend basing the pilot study size on the future trial's size”.10
Totton and colleagues reviewed 761 UK pilot and feasibility studies registered on the ISRCTN registry between 2013 and 2020.11 The median target sample size was 30 per arm, with an interquartile range of 20 to 50; pilot and feasibility studies had the same median.11 Binary outcomes attracted a higher median, 50 per arm, against 30 for continuous outcomes.11
Set against the recommendations, the picture divides. Only 11 studies, 1%, fell below the most permissive threshold of ten per arm.11 Against the Teare recommendation, 436 studies, 57%, did not meet it.11 The authors summarise it as follows: most studies “satisfy the earlier and more lenient recommendations however do not satisfy the most recent largest recommendation”.11
A separate finding bears on how much weight any of these numbers can carry. Among studies with recruitment data, “173/301 (57%) did not reach their target total sample size”.11 The authors warn that studies falling short “may not be providing suitable information” for designing a definitive trial.11
The practical position that follows is not a number. It is a requirement to derive one. Thabane and colleagues note that “in general, sample size calculations may not be required for some pilot studies”,4 which is consistent with the reporting standard: the extension asks not for a power calculation but for a “rationale for numbers in the pilot trial”.1 A pilot sized to estimate a retention rate to a stated precision has a rationale. A pilot sized at twenty because twenty was available does not.
Progression criteria, written before the data arrive
A progression criterion is a statement, made in advance, of what result would justify proceeding to the main trial. The CONSORT extension added an item for exactly this: item 6c covers “prespecified criteria used to judge whether, or how, to proceed with future definitive trial”.1 Thabane and colleagues make the same requirement in stronger terms: “it is always important to state the criteria for success of a pilot study”, and “the criteria should be based on the primary feasibility objectives”.4
The form the criteria take matters. Avery and colleagues, reporting a workshop on internal pilots, describe the alternative to a binary rule: “a traffic light system of green (go), amber (amend) and red (stop)”, which “might be preferable to a simple stop/go approach”.7 The middle band is the useful part, because most pilots land there.
Two cautions attach to this. The first concerns how firmly a threshold should be drawn. The extension advises being “cautious about setting definitive thresholds that could be missed simply due to chance variation”.1 A recruitment target missed by two participants in a study of forty is within the noise of the design. The second concerns what meeting the criteria entitles a team to conclude. Avery and colleagues state that meeting them “does not automatically guarantee continuation to and throughout the main trial”.7 They add a reporting requirement: “it is recommended that progression criteria are reported clearly in a trial protocol paper”.7
A systematic review of guidance documents found this to be the weakest-supported part of the whole design. Hallingberg and colleagues report that “little guidance was provided on what should be considered when formulating progression criteria”, and that “guidance is lacking on the evidence needed to inform when to proceed to an effectiveness study”.8 Their own position is the one the traffic-light system implements: “progression criteria should not be judged as strict thresholds but as guidelines”.8
What a pilot study cannot tell you
Of all the errors in this area, one does the most damage: taking the treatment effect observed in a pilot and using it to plan, justify or power the main trial.
Sim sets out the objection in a commentary devoted to the question. Pilot estimates of a treatment effect “will be imprecise in typically small pilot studies”, and they “may be biased if the pilot or feasibility study is unrepresentative of the main trial”.12 His conclusion is unambiguous: such estimates “should not therefore play a part in any formal decision-making” about progression, and treatment effects “should not be the basis of a sample size calculation for a main trial”.12 What should set the target instead is judgement about what would matter clinically: “a targeted treatment effect should be established in terms of clinical judgement”.12
Bell, Whitehead and Julious reach the same conclusion from the statistical side, calling the practice “a practice that should be avoided in general, as estimation is poor due to the small sample size”, and adding that “preliminary efficacy evidence from a pilot study should not be overstated”.10 Thabane and colleagues put it in one clause: “it can be dangerous to use pilot studies to estimate treatment effects”.4
Because pilots are small, they will usually fail to reach conventional significance whatever is true. Arain and colleagues state the arithmetic consequence: “pilot studies, by their nature, will not produce ‘significant’ (i.e P < 0.05) results”.3 Thabane and colleagues name the inferential error that follows: “underpowered studies (with no statistically significant results) are inconclusive, not negative”, because “‘no evidence of effect’ is not ‘evidence of no effect’”.4 That distinction is developed further in our article on the null hypothesis and its importance in research.
If an effect estimate is reported at all, the question becomes how to present it. Lee and colleagues examined whether the conventional threshold is the right one for pilot work, observing that “pilot studies are usually underpowered to achieve statistical significance at the commonly used 5% level” and that “they are not designed to formally assess evidence of benefit”.13 Their recommendation is to change the form of the report rather than the threshold of the test: “the focus should be on descriptive statistics and estimation, using confidence intervals”, with the interval “interpreted with regards to the minimum clinically important difference”.13
The reporting standard takes the same line and makes it a rule. Estimates should be reported “without P values”, because “pilot trials are not powered for testing hypotheses about effectiveness”.1 Hallingberg and colleagues record the general view in the guidance literature, noting that some sources “condemned statistically testing for effectiveness” because it leads “to imprecise and potentially misleading estimates of effect sizes”.8
A worked example
The SuMMiT-D pilot illustrates what a well-specified feasibility randomised trial looks like when it reports against its own objectives rather than against a clinical outcome.
The trial tested whether brief text messages supporting medicine use could be delivered to people with type 2 diabetes alongside routine primary care.16 Its objectives were explicitly about process. The stated primary objective was “to test the recruitment and randomisation”, with “a planned rate of 22 participants randomised per month” and “an aim of retaining more than 80% of participants”.16
| Feasibility objective | What was observed |
|---|---|
| Recruitment and randomisation | 294 assessed for eligibility; 209 randomly allocated, 103 to the intervention and 106 to usual care16 |
| Retention above 80% at six months | Self-report data complete for 177 participants, 84.7%16 |
| Routine data capture | Medical record data available for 207 participants, 99.0%; a new HbA1c measurement for 140, 67%16 |
Two features of the report are worth copying. First, the power statement names the right quantity: “the study was powered to estimate retention to the study over a period of 6 months”, and the authors state directly that “the trial was not designed to establish a signal of efficacy”.16 Second, the conclusion is conditional rather than triumphant. Sending the messages “is feasible”, but “further work needs to be carried out prior to a trial to ensure routine data on cholesterol measurements is collected”.16 A gap in the data pipeline was identified at the stage where it was cheap to fix.
Ethics and what participants are told
A pilot study involving human participants requires ethical approval on the same basis as any other study. The smaller scale does not reduce the obligation, and the preliminary purpose creates an additional one: participants should be told that the study is preparatory and that procedures may change.
How well that is done has been measured, and the answer is poorly. Khan and colleagues audited the participant materials of pilot and feasibility studies at a single centre and found that “less than half stated intent to assess feasibility in the informed consent documentation”.15 In figures, 42.4% of studies stated that intent in their consent documents; only 12% of final consent documents defined what a pilot or feasibility study is; and a single study, 0.5% of the sample, met all five transparency criteria the authors assessed.15
Their recommendation is addressed to the system rather than to individual investigators: transparency in these documents “needs to be specifically addressed by research ethics guidelines”.15 Until it is, the practical step is for the protocol and the information sheet to say three things explicitly: that the study is preparatory to a larger one, what the feasibility objectives are, and that the participant's data may be used to decide whether the larger study goes ahead.
Qualitative work inside a feasibility study
A rate tells you that something failed. It does not tell you why, and the why is what the main trial has to be redesigned around. This is the gap qualitative work fills, and there is formal guidance on using it.
O'Cathain and colleagues set out what qualitative research contributes: it “can be used to examine and address key uncertainties prior to a full trial”.14 The specific contribution is visibility. “Some aspects of the intervention in practice may be hidden or not understood, and these hidden aspects may be the key to delivering outcomes.”14
It also bears directly on the progression decision discussed above. Where a study “falls just short of a quantitative criterion”, the authors argue that “a sufficient qualitative understanding of why this happened and how to improve it” may justify proceeding.14 That is the amber band of the traffic-light system, with evidence behind it rather than optimism.
Two of their recommendations are concrete. On staffing: “it is important that an expert in qualitative methods be included in both the planning and delivery teams”.14 On dissemination: where others could learn from a feasibility study, “we recommend publishing them in peer-reviewed journal articles”.14
Reporting: the CONSORT extension
Pilot and feasibility randomised trials have their own reporting standard, and it is the CONSORT 2010 extension published in 2016.1 The extension applies to both labels, since its authors “make no distinction in this extension between pilot and feasibility randomised trials”.1
| Item | What the extension requires | Why it differs |
|---|---|---|
| 1a — Title | “Identification as a pilot or feasibility randomised trial in the title”1 | Standard CONSORT asks only for identification as a randomised trial |
| 1b — Abstract | A structured summary that “focuses on the aims and objectives of the pilot trial and not on the future definitive RCT”1 | Abstracts are where pilot findings are most often overstated |
| 2b — Objectives | “Specific objectives or research questions for pilot trial”1. An objective to test effectiveness “is not recommended”1 | The design is not built to answer an effectiveness question |
| 6c — Outcomes | “If applicable, prespecified criteria used to judge whether, or how, to proceed with future definitive trial”1 | A new item with no equivalent in standard CONSORT |
| 7a — Sample size | “Rationale for numbers in the pilot trial”1 | A power calculation for effectiveness would be inappropriate, but a rationale is still required |
| 12a — Analysis | Estimates reported “without P values”1 | “Pilot trials are not powered for testing hypotheses about effectiveness”1 |
| 12b — Subgroup analysis | Marked “not applicable”1 | “The primary focus is not on determining treatment effects”1 |
| 21 — Generalisability | “Generalisability (applicability) of pilot trial methods and findings to future definitive trial and other studies”1 | What transfers is the method, not the result |
| 22a — Interpretation | “Implications for progression from pilot to future definitive trial, including any proposed amendments”1 | A new item; the discussion should state whether the definitive trial is likely to go ahead1 |
The extension does not forbid reporting an effect estimate outright, but it confines it. If underpowered tests are performed, “they should certainly be viewed as secondary and a caveat included in the discussion”.1 The framework paper adds a labelling requirement that costs nothing: reports should “give clear confirmation that their study is in preparation for a future randomised controlled trial”.2
The problem the extension was written to address was documented first. Arain and colleagues concluded that “pilot studies are still poorly reported, with inappropriate emphasis on hypothesis-testing”, and that “any testing of an intervention needs to be reported cautiously”.3 They also surveyed journal policy and found little of it: “four of the journals did not have a specified policy about publishing pilot studies”, and in general editors “were loathe to publish studies described as ‘pilot’”.3 The publication difficulty is one reason a study's feasibility purpose sometimes disappears from its title.
When a pilot study is not needed
A pilot is justified by uncertainty. Where the uncertainty is not there, the study adds cost and delay without adding information.
- The procedures are already established in this setting. If the recruitment pathway, the outcome instrument and the data systems have all been used before with the same population, the key uncertainties the guidance asks about have already been examined.6
- The parameters are available from existing data. A standard deviation or a retention rate that can be taken from a completed study in the same population does not need to be re-estimated, and a pilot would estimate it less precisely.9
- The question is about effectiveness. A study designed to answer whether an intervention works is a definitive trial, which the extension defines as “an appropriately powered study focusing on effectiveness or efficacy”.1 Labelling it a pilot does not make its effect estimate more defensible.
- An internal pilot would serve better. Where the main trial is funded and the uncertainty concerns recruitment rate rather than whether the trial should exist, an internal pilot reviews the processes without discarding participants.7
Conversely, a pilot earns its place where the intervention is new, the population has not been studied with this design before, the trial is complex, or recruitment is expected to be difficult. Those are precisely the conditions the Medical Research Council guidance has in mind when it warns that evaluations are often undermined by “recruitment and retention” problems.6
Common mistakes
| Mistake | Stage | What it costs | What to do instead |
|---|---|---|---|
| Powering the main trial from the pilot's effect estimate | Design of the main trial | The estimate is imprecise and may be biased if the pilot was unrepresentative12 | Set the target effect by clinical judgement; use the pilot for the standard deviation12 |
| Testing a hypothesis of effectiveness | Objectives | The trial is underpowered, so the test is uninformative either way1 | Write feasibility objectives; formal hypothesis testing for effectiveness “is not recommended”1 |
| Reading a non-significant pilot as a negative result | Interpretation | A promising intervention is abandoned on no evidence4 | Report it as inconclusive; “no evidence of effect” is not “evidence of no effect”4 |
| No progression criteria | Protocol | The decision to proceed becomes retrospective and unauditable1 | Prespecify them, and report them in the protocol paper7 |
| Progression criteria set as hard thresholds | Protocol | A target missed by chance variation stops a viable trial1 | Use green, amber and red bands rather than stop/go7 |
| Sample size chosen by convenience | Design | Item 7a cannot be satisfied, and the estimates may be too imprecise to use9 | Give a “rationale for numbers in the pilot trial” tied to the feasibility objectives1 |
| Recording rates without reasons | Conduct | The main trial cannot be redesigned, because nothing explains the failure14 | Collect qualitative data alongside the rates14 |
| Omitting the feasibility purpose from consent materials | Ethics | Participants are not told what they are contributing to; currently the norm rather than the exception15 | State the preparatory purpose and the feasibility objectives in the information sheet15 |
| Dropping “pilot” or “feasibility” from the title | Reporting | Readers and reviewers assess the study against the wrong standard1 | Item 1a requires identification in the title1 |
A pilot study answers whether the planned trial can be run, and it answers that by running the trial in miniature. It does not answer whether the intervention works, and the effect it happens to observe is too imprecise to plan anything with. Write feasibility objectives rather than effectiveness ones. Prespecify what result would mean proceed, amend or stop, as bands rather than as hard thresholds. Justify the number of participants against those objectives. Collect qualitative data alongside the rates, because a rate without a reason cannot be acted on. Then report the study against the CONSORT extension, with the word “pilot” or “feasibility” in the title and estimates given without p-values.
How to apply this to your work
- State the uncertainties the pilot exists to examine, taken from the development work rather than invented afterwards.6
- Write feasibility objectives, not effectiveness objectives, and attach a measurable outcome to each one.1
- Prespecify progression criteria as green, amber and red bands rather than as single thresholds, and report them in the protocol.7
- Justify the number of participants against those objectives, since item 7a asks for a rationale rather than a power calculation.1
- Say in the consent materials that the study is preparatory, and what the data will be used to decide.15
- Plan the qualitative component and name who will run it, so the reasons behind the rates are collected rather than reconstructed.14
- Plan a descriptive analysis, reporting estimates with intervals and no p-values for effectiveness.1
- Set the main trial's target effect by clinical judgement, not from whatever the pilot observed.12
Common questions
What is a pilot study in simple terms?
It is a small version of a planned study, run first to check that the planned study can be carried out. The standard definition describes it as “a version of the main study that is run in miniature”, in order “to test whether the components of the main study can all work together”.5 Its focus is on process rather than on the clinical outcome.5
What is the difference between a pilot study and a feasibility study?
They are not two parallel categories. A feasibility study asks “whether the future trial can be done, should be done, and, if so, how”, and “pilot studies are a subset of feasibility studies”.1 What makes a feasibility study a pilot is that “a future study or part of a future study, is conducted on a smaller scale”.2 For reporting purposes the distinction does not matter, because the same checklist covers both.1
How many participants does a pilot study need?
Published rules of thumb range “from 12 to 35 individuals per arm”.10 Simulation work recommends “at least 70 measured subjects (35 per group)” where the aim is to estimate a standard deviation.9 In practice, UK studies registered between 2013 and 2020 had a median target of 30 per arm.11 What the reporting standard requires is not a particular number but a “rationale for numbers in the pilot trial”.1
Can I use my pilot results to calculate the sample size for the main trial?
The standard deviation, yes, with caution. The treatment effect, no. Pilot effect estimates “will be imprecise” and “should not be the basis of a sample size calculation for a main trial”.12 Even the standard deviation “tends to be underestimated” in small samples, and “considerable sampling variation remains even with a large sample size”.9 A sensitivity analysis is the usual remedy.10
Should a pilot study report p-values?
Not for effectiveness. The CONSORT extension asks for estimates “without P values”, because “pilot trials are not powered for testing hypotheses about effectiveness”.1 Where an effect is reported, the recommended form is “descriptive statistics and estimation, using confidence intervals”, read against the minimum clinically important difference.13
What are progression criteria, and when are they set?
They are prespecified statements of what result would justify proceeding to the main trial, and they belong in the protocol before recruitment opens.1 A graded form is usually preferable to a binary one: “a traffic light system of green (go), amber (amend) and red (stop)”.7 Meeting them “does not automatically guarantee continuation to and throughout the main trial”.7
Does a pilot study need ethical approval?
A pilot involving human participants is subject to the same ethical review requirements as any other study; its smaller size does not change that. The specific obligation it adds is disclosure of the preparatory purpose, and audit evidence shows this is commonly missed: “less than half stated intent to assess feasibility in the informed consent documentation”.15
What is the difference between an internal and an external pilot?
In an external pilot, “the data may be analysed and set aside”, and the study is “a rehearsal of the main study”.57 In an internal pilot, the first phase of the definitive trial is designated as the pilot and “the outcome data generated may contribute to the final analyses”.7 Because those data are reused, the decision point has to be fixed in the protocol in advance.
Declared interest: Pubrica provides research design and manuscript support commercially. Nothing on this page should be read as suggesting that a pilot study, however well analysed, can supply evidence of effectiveness; that is a property of the design, not of the analysis.
References and sources
- Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, Lancaster GA. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ. 2016;355:i5239. doi:10.1136/bmj.i5239 read 20 October 2026 from an institutional repository copy of the publisher PDF.View the full text
- Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, Bond CM. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS ONE. 2016;11(3):e0150205. doi:10.1371/journal.pone.0150205 read 20 October 2026.View on PLOS ONE
- Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology. 2010;10:67. doi:10.1186/1471-2288-10-67 read 20 October 2026 from an institutional repository copy.View the full text
- Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Medical Research Methodology. 2010;10:1. doi:10.1186/1471-2288-10-1 read 20 October 2026.View the full text
- National Institute for Health Research, Health Technology Assessment programme. HTA Clinical Evaluation and Trials: an Open Call — Specification Document. 2012. read 20 October 2026. This programme document carries the NIHR definitions of feasibility and pilot studies that the methodological literature cites.View the specification document
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi:10.1136/bmj.a1655 read 20 October 2026 from an institutional repository copy.View the full text
- Avery KNL, Williamson PR, Gamble C, O’Connell Francischetto E, Metcalfe C, Davidson P, Williams H, Blazeby JM. Informing efficient randomised controlled trials: exploration of challenges in developing progression criteria for internal pilot studies. BMJ Open. 2017;7(2):e013537. doi:10.1136/bmjopen-2016-013537 read 20 October 2026.View on BMJ Open
- Hallingberg B, Turley R, Segrott J, Wight D, Craig P, Moore L, Murphy S, Robling M, Simpson SA, Moore G. Exploratory studies to decide whether and how to proceed with full-scale evaluations of public health interventions: a systematic review of guidance. Pilot and Feasibility Studies. 2018;4:104. doi:10.1186/s40814-018-0290-8 read 20 October 2026 from an institutional repository copy.View the full text
- Teare MD, Dimairo M, Shephard N, Hayman A, Whitehead A, Walters SJ. Sample size requirements to estimate key design parameters from external pilot randomised controlled trials: a simulation study. Trials. 2014;15:264. doi:10.1186/1745-6215-15-264 read 20 October 2026 from an institutional repository copy.View the full text
- Bell ML, Whitehead AL, Julious SA. Guidance for using pilot studies to inform the design of intervention trials with continuous outcomes. Clinical Epidemiology. 2018;10:153–157. doi:10.2147/CLEP.S146397 read 20 October 2026 from an institutional repository copy. The 12-per-arm rule of thumb originates with Julious (Pharmaceutical Statistics. 2005;4(4):287–291) and the stepped rules with Whitehead and colleagues; neither original was reachable, so both are cited here as this paper reports them.View the full text
- Totton N, Lin J, Julious S, Chowdhury M, Brand A. A review of sample sizes for UK pilot and feasibility studies on the ISRCTN registry from 2013 to 2020. Pilot and Feasibility Studies. 2023;9:188. doi:10.1186/s40814-023-01416-w read 20 October 2026 from an institutional repository copy.View the full text
- Sim J. Should treatment effects be estimated in pilot and feasibility studies? Pilot and Feasibility Studies. 2019;5:107. doi:10.1186/s40814-019-0493-7 read 20 October 2026 from an institutional repository copy.View the full text
- Lee EC, Whitehead AL, Jacques RM, Julious SA. The statistical interpretation of pilot trials: should significance thresholds be reconsidered? BMC Medical Research Methodology. 2014;14:41. doi:10.1186/1471-2288-14-41 read 20 October 2026 from an institutional repository copy.View the full text
- O’Cathain A, Hoddinott P, Lewin S, Thomas KJ, Young B, Adamson J, Jansen YJFM, Mills N, Moore G, Donovan JL. Maximising the impact of qualitative research in feasibility studies for randomised controlled trials: guidance for researchers. Pilot and Feasibility Studies. 2015;1:32. doi:10.1186/s40814-015-0026-y read 20 October 2026 from an institutional repository copy.View the full text
- Khan MIU, Mbuagbaw L, Holek M, Bdair F, Durrani ZH, Mellor K, Eddy S, Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Lancaster GA, Thabane L. Transparency of informed consent in pilot and feasibility studies is inadequate: a single-center quality assurance study. Pilot and Feasibility Studies. 2021;7:96. doi:10.1186/s40814-021-00828-w read 20 October 2026 from an institutional repository copy.View the full text
- Farmer A, Allen J, Bartlett Y, Bower P, Chi Y, French D, Gudgin B, Holmes E, Horne R, Hughes D, Jones L, Kenning C, Locock L, McSharry J, Miles L, Newhouse N, Rea R, Robinson S, Tarassenko L, Velardo C, Williams V, Yu L-M. Supporting people with type 2 diabetes in effective use of their medicine through mobile health technology integrated with clinical care (SuMMiT-D pilot): results of a feasibility randomised trial. Pilot and Feasibility Studies. 2024;10(1):15. doi:10.1186/s40814-023-01429-5 read 20 October 2026 from the version of record in an institutional repository.View the version of record
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Work through these before the protocol goes for review.
[ ] Key uncertainties listed, taken from the development work [ ] Each uncertainty written as a feasibility objective [ ] No objective stated in terms of effectiveness or efficacy [ ] Measurable feasibility outcome attached to every objective [ ] Progression criteria prespecified as green, amber and red bands [ ] Progression criteria recorded in the protocol, not decided later [ ] Internal or external pilot chosen, and the reason recorded [ ] Rationale given for the number of participants [ ] Protocol mirrors the main trial it is preparing for [ ] Ethical approval sought on the same basis as any study [ ] Information sheet states the study is preparatory [ ] Information sheet states what the data will be used to decide [ ] Qualitative component planned, with a named lead [ ] Protocol deviations, refusals and withdrawals to be logged with reasons [ ] Analysis plan descriptive, with intervals and no effectiveness p-values [ ] Title will identify the study as a pilot or feasibility trial
We help you turn the uncertainties in a trial design into feasibility objectives, set progression criteria that will survive review, justify the number of participants, and report the study against the CONSORT extension so a reviewer does not have to correct it.