Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Services · Experimental design · Benefits of an experimental plan

Benefits of an Experimental Plan in Research and Design

Four ordinary analytic choices, made after the data are in, take a nominal 5% false-positive rate to 61% for an effect that does not exist. An experimental plan prevents this by fixing those choices in advance.

Pubrica AcademyWritten by a research writing specialist Reviewed bya research methodology reviewer Published15 October 2026 Last checked15 October 2026 Reading time15 minutes Sources14 primary

An experimental plan is not paperwork produced for a funder. It is the set of decisions that determine what your experiment will be able to conclude — how many units, allocated how, measuring what, analysed how — and every one of them is cheaper to make now than to defend later.

Why Experimental Planning Matters

Table 1. Six things a plan gets you

You can tell whether the result means anythingA study too small to detect the effect you are looking for produces an inconclusive result, not a cautious one — and when it does find something, that finding is often an overestimate.
You get more answers from the same materialA factorial design answers two questions with the run size one-factor-at-a-time spends on one, at the same power for the treatment effect.
Known nuisance variation stops inflating your error termBlocking removes batch, day, litter or operator effects from the error, which raises precision without adding units.
You are protected from what you did not anticipateRandomisation guards against latent variables and subjective bias, and is what makes the statistical inference valid.
The analysis cannot drift toward the answer you wantedDeciding the outcome, the analysis and the stopping rule in advance removes the degrees of freedom that turn a nominal 5% false-positive rate into 61%.
The work is reportable, and therefore publishableReporting guidelines ask for the design decisions directly. If they were never made, there is nothing to report — sample size justification appears in fewer than 10% of animal-research publications.

The Three Principles of Experimental Design

Table 2. Fisher’s three principles of experimental design

PrincipleWhat it isWhat it achieves
ReplicationRunning each treatment on more than one experimental unit.Enables the estimation of experimental error. Without it there is nothing to compare a treatment difference against.
RandomisationAllocating units to treatments by chance.Protects against latent variables and subjective biases, and is the basis for a valid estimate of a treatment effect.
Blocking, also called local controlGrouping units that resemble each other, and randomising within each group.Removes a known source of variation from the error term, which raises precision. It is a deliberate restriction on randomisation.

The attribution, and the principles as the source states them: “Sir Ronald A. Fisher laid down the three” principles — “replication, randomization, and blocking (local control) in the 1920's.”1 “Fisher (1935) first presented the three principles of experiment design.”1

Blocking and Local Control Are One Principle

This naming point often causes confusion. Blocking and local control are two names for the same principle, so there are three principles, not four. Some textbooks use one term, some the other, and some give both — a page listing randomisation, replication, blocking and local control as four separate principles has double-counted.

What each one is for, in short: “Enable the estimation of experimental error.”2 for replication; “Randomization is the basis for obtaining a valid estimate of a treatment effect.”1 for randomisation; and for blocking, “The blocking principle implies that extraneous, non-treatment variation”1 is controlled.

On attribution: the three principles are R. A. Fisher's, from his work at Rothamsted and set out in The Design of Experiments (1935). Sources differ on the year to attach to them — the paper cited here says Fisher laid them down in the 1920s and first presented them in 1935, while other teaching sources say 1930 — so no single date is asserted. Two of the sources below are university course notes rather than published texts, and are cited as such.

The Distinct Roles of Blocking and Randomisation

How Blocking and Randomisation Complement Each Other

The roles of blocking and randomisation are complementary, and this is central to practical design. Blocking handles nuisance factors you know about and can measure — batch, litter, day, operator, site. Randomisation protects against the ones you do not know about: where “the nuisance factor is unknown and uncontrollable (a 'lurking' variable), we hope that randomization balances out its impact across the experiment”. Hence the standard maxim: block what you can and randomise what you cannot.

In the sources' own terms: “Blocking is a technique for dealing with nuisance factors”3, and “A block represents a restriction on randomization”3 — deliberately so. Where you cannot identify the nuisance factor, “If the nuisance factor is unknown and uncontrollable (a 'lurking' variable), we hope that randomization balances out its impact across the experiment”3 Randomisation's other function is the one most often forgotten: it “Ensure the validity of statistical inferences.”2

That maxim is standard in the design literature and appears unattributed in the course notes we read, so it is given here as a convention rather than credited to anyone in particular.

Factorial Designs Versus One-Factor-at-a-Time Experiments

A full factorial design is one “A design in which every setting of every factor appears with every setting of every other factor is a full factorial design”4 “If there are k factors, each at 2 levels, a full factorial design has 2k runs.”4 — which is also its limit: “when the number of factors is 5 or greater, a full factorial design requires a large number of runs”4

Why One-Factor-at-a-Time Experiments Are Inefficient

The alternative most researchers reach for instinctively is one factor at a time: hold everything constant, vary one thing, repeat. It is intuitive and it is wasteful. Varying factors together in a factorial design “allows us to address twice the questions with half the animal numbers”, and the same total run size gives “the same statistical power for testing the treatment effect” while also answering the second question. One-factor-at-a-time “is highly inefficient as it uses more resources and provides less-accurate effect size estimates” — and it cannot detect an interaction at all, because it never varies two factors together.

The quantified claim: “a factorial design allows us to address twice the questions with half the animal numbers”5, and a factorial run of the same size “has thus the same statistical power for testing the treatment effect”5 as the one-factor-at-a-time alternative, while also answering the second question. “This leads to increased precision and, thus, increased power in the factorial analysis compared with the OFAT setting.”5

Sample Size as a Design Decision

The requirement is to calculate it in advance: “Provide details of any a priori sample size calculation, if done.”7, and “The primary outcome measure should be identified in the planning stage of the experiment”7 The analysis belongs to the same stage — the statistical method “should be determined at the design stage”8, and methods should be “planned at the same time as their experiments.”8

The cost of underpowering. “Sample sizes that are too small (i.e., underpowered studies) produce inconclusive results”7 “Low power has three effects: first, within the experiment, real effects are more likely to be missed”7; and “when an effect is detected, this will often be an overestimation of the true effect size”7 In animal work there is a further cost: you “risk wasting animals used in inconclusive research”7

The second effect is counter-intuitive and deserves attention. An underpowered study does not simply fail more often; the positive results it does produce are systematically too large. As Button and colleagues put it, “low power also reduces the likelihood that a statistically significant result reflects a true effect”9 In the neuroscience literature they examined, “Our results indicate that the median statistical power in neuroscience is 21%.”9

Limits of the 21% Median Power Figure

This figure is often quoted beyond its original context. The 21% median was estimated within the neuroscience literature. It is not a figure for research in general, and a page quoting it as “the median power of published studies” has overreached. What generalises is the mechanism rather than the number: low power does not merely risk missing a real effect, it “reduces the likelihood that a statistically significant result reflects a true effect”, and inflates the apparent size of the effects it does detect.

The structural version of the same argument: “Small sample size means smaller power”10, and “Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true.”10 Their recommendation is simply to “Perform an a priori power calculation”9 — “Use the existing literature to estimate the size of effect you are looking for and design your study accordingly.”9

Risks of Deciding the Analysis After Data Collection

If the outcome, the covariates and the stopping rule are chosen once the data are in, each choice can be defended individually, but in combination they seriously inflate the false-positive rate. “The culprit is a construct we refer to as researcher degrees of freedom.”11 That “flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates”11

Table 3. Probability of at least one significant result at p < .05 for an effect that does not exist

Analytic flexibility exploitedFalse-positive rate
Two dependent variables (r = .50)9.5%
Adding 10 more observations per cell7.7%
Controlling for gender, or gender × treatment11.7%
Dropping, or not dropping, one of three conditions12.6%
The first two combined14.4%
The first three combined30.9%
All four combined60.7%

Interpreting the 61% False-Positive Figure

The last row of the table is widely misquoted. It is not a claim that 61% of published findings are false. It is the simulated probability of obtaining at least one result at p < .05 for an effect that does not exist, when four ordinary-looking analytic choices are exploited together — against a nominal rate of 5%. The choices are each defensible in isolation, which is the point: no single one looks like misconduct, and in combination they make a false positive more likely than not. The paper also reports that optional stopping alone, testing after each new observation per condition, “finds a significant effect 22% of the time”.

The authors' remedy is a design rule rather than a statistical one: “Authors must decide the rule for terminating data collection before data collection begins”11 And the mechanism in Ioannidis's terms: “Flexibility increases the potential for transforming what would be 'negative' results into 'positive' results”10

The figures in the table above were read from a proof version of the paper hosted by a university, because the publisher's full text was not reachable; the citation itself was confirmed on the publisher's page. If you need to quote the table exactly, check it against the published version.

Pre-Registration and the Evidence on Its Effects

Pre-registration and Registered Reports are “two methods for creating a permanent record of a research plan prior to conducting data collection”14. A Registered Report goes further: “Registered Reports are a form of empirical publication in which study proposals are peer reviewed and pre-accepted”13 — the journal commits before the results exist, which removes the incentive to produce a particular result. Its authors describe the format as one where “Registered Reports offer a remedy for a range of reporting and publication biases”13

Interpreting the Evidence on Registered Reports

What this comparison does and does not show. Of 152 randomly sampled hypothesis-testing studies in the standard psychology literature, 96% reported a positive result for their first hypothesis. Of 71 published Registered Reports, 44% did — or 50% if direct replications are excluded. That is a striking difference, and it is observational: the two sets of papers differ in more than their format, and the authors themselves describe reduced publication bias or Type I error inflation as a “plausible explanation” rather than a demonstrated one. The honest reading is that Registered Reports report positive results at roughly half the rate of the conventional literature — not that pre-registration halves false positives.

The figures: “we found 96% positive results in standard reports, but only 44% positive results in Registered Reports”12, and excluding direct replications, “(96% vs 50% positive results)”12

Reporting Guideline Requirements for the Experimental Plan

The most practical argument for planning is that the reporting guideline for your field will ask for these decisions directly, and if they were never made there is nothing to report. ARRIVE 2.0’s Essential 10 are “These ten items are the basic minimum that must be included in any manuscript describing animal research.”6:

Table 4. The ARRIVE 2.0 Essential 10

ItemWhat it asks for
1Study designThe groups compared, and the experimental unit.
2Sample sizeThe number of units, and the justification for it.
3Inclusion and exclusion criteriaCriteria, exclusions and numbers analysed.
4RandomisationWhether it was used, and how confounders were addressed.
5Blinding or maskingWho was aware of group allocation, and when.
6Outcome measuresAll measures, and which was primary.
7Statistical methodsThe methods for each analysis, and their assumptions.
8Experimental animalsSpecies-appropriate detail.
9Experimental proceduresWhat, when, where and why.
10ResultsSummary statistics per group, with effect sizes and intervals.

Seven of those ten are design decisions. The guideline's stated purpose includes “helping the design of rigorous and reliable in vivo experiments”6, not merely describing them afterwards, and the reason for the minimum is “to enable others to scrutinise the work adequately”6

How often the hardest of them is actually supplied: “sample size justification (reported in less than 10% of publications)”6 This is the gap that an experimental plan addresses.

ARRIVE 2.0 was developed by an international working group and is hosted by the NC3Rs. None of the pages we read states who maintains it, so no maintainer is asserted here.

Recommended Working Practices

How to Apply Experimental Planning to Your Own Work

  1. Write the analysis plan before collecting data: the primary outcome, the comparison, the statistical method and its assumptions.
  2. Calculate the sample size a priori from the effect size you are looking for, and record the calculation. An underpowered study is not a cautious study.
  3. Fix the stopping rule before data collection begins. Optional stopping alone produces a significant result about a fifth of the time when nothing is there.
  4. Block the nuisance factors you can identify — batch, day, litter, operator, site — and randomise within blocks.
  5. Randomise allocation, and conceal it. This is what makes the inference valid, not just what makes it fair.
  6. Use a factorial design when you have more than one factor. One factor at a time costs more, estimates effects less precisely, and cannot detect an interaction.
  7. Pull up the reporting guideline for your field — ARRIVE for animal work — and treat its checklist as the design agenda rather than a submission chore.
  8. Consider pre-registration, or a Registered Report where the journal offers it, particularly for a confirmatory study.
  9. Record every deviation as it happens. A documented change of plan is a finding; an undocumented one is a degree of freedom.

Take-Home Message

Replication, randomisation and blocking are three principles, not four, because blocking and local control are the same principle. Block the factors you can measure and randomise the ones you cannot. Factorial designs answer more questions from the same material and are the only way to detect an interaction. Sample size and the analysis method are design decisions: underpowered studies produce inconclusive results and overestimated effects. Deciding the analysis after data collection has serious consequences, because four ordinary analytic choices combined take a nominal 5% false-positive rate to 61% for an effect that does not exist.

Replication, randomisation, and blocking — also called local control. They are R. A. Fisher’s, from his work at Rothamsted and set out in The Design of Experiments (1935). Replication enables the estimation of experimental error; randomisation is the basis for a valid estimate of a treatment effect; blocking removes known nuisance variation from the error term.

No — blocking and local control are two names for the same principle. Some textbooks use one term, some the other, and some give both. A list of randomisation, replication, blocking and local control as four separate principles has counted one of them twice.

Blocking handles nuisance factors you know about and can measure — batch, litter, day, operator, site. Randomisation protects against the ones you cannot: where the nuisance factor is unknown and uncontrollable, you rely on randomisation to balance its impact across the experiment. Hence the standard maxim: block what you can and randomise what you cannot.

Because it answers more questions from the same material and can detect interactions. One group reports that “a factorial design allows us to address twice the questions with half the animal numbers”, with the same statistical power for the treatment effect. Varying one factor at a time “is highly inefficient as it uses more resources and provides less-accurate effect size estimates” — and it cannot find an interaction at all.

Before the experiment. The ARRIVE guidance asks authors to “provide details of any a priori sample size calculation”, and notes that “the primary outcome measure should be identified in the planning stage of the experiment”. Underpowered studies “produce inconclusive results”, and when they do detect an effect, that estimate “will often be an overestimation of the true effect size”.

Dramatically, and the headline figure is often misquoted. Simulations combining four ordinary analytic choices produced “a stunning 61% false-positive rate” — that is, a 60.7% chance of finding at least one significant result for an effect that does not exist, against a nominal 5%. It is not a claim that 61% of published findings are false.

The strongest available comparison found 96% positive results among 152 studies sampled from the standard psychology literature, against 44% among 71 published Registered Reports — or 50% excluding direct replications. That is observational rather than experimental, and the authors describe reduced publication bias or Type I error inflation as a “plausible explanation” rather than a demonstrated one.

“A form of empirical publication in which study proposals are peer reviewed and pre-accepted” — the journal commits to publishing the paper on the strength of the plan, before the results exist. Its authors describe the format as offering “a remedy for a range of reporting and publication biases”.

Animal Research: Reporting of In Vivo Experiments, currently version 2.0 (2020). Its “Essential 10” are “the basic minimum that must be included in any manuscript describing animal research”, and most of them are design decisions — study design, sample size, inclusion criteria, randomisation, blinding, outcome measures and statistical methods.

References and Sources

  1. Federer WT. Principles of Statistical Design with Special Reference to Experiment and Treatment Design. Biometrics Unit Paper BU-813-M, Cornell University, May 1983 (revised July 1983). read 15 October 2026. View on Cornell eCommonsThe citable source for the three classical principles and their attribution. Note it dates Fisher's statement of them to the 1920s and to his 1935 book, where other sources say 1930, so no single year is asserted here.
  2. Statistics 514: Design of Experiments, Lecture 1: Overview and Fundamental Principles. Department of Statistics, Purdue University, 2005. Readings: Montgomery ch. 1; Wu & Hamada §1.1–1.3. read 15 October 2026. View on Purdue University (course notes)University course notes rather than a published text, and cited as such. Used for the compact statements of what each principle achieves, and for the standard maxim about blocking and randomising.
  3. Design of Engineering Experiments: The Blocking Principle. STAT 502 lecture notes, Department of Statistics, University of Washington (after Montgomery, ch. 4). read 15 October 2026. View on University of Washington (course notes)Course notes, cited as such. The clearest available statement of how blocking differs from randomisation: one handles nuisance factors you know about, the other the ones you do not.
  4. NIST/SEMATECH. e-Handbook of Statistical Methods, §5.3.3.3: Full factorial designs. National Institute of Standards and Technology. read 15 October 2026. View on NISTCited only for the definition of a full factorial design and the run-count arithmetic. This page does not discuss interaction, and no NIST page we opened compares factorial with one-factor-at-a-time designs.
  5. Grüninger SL, Frommlet F. Half the price, twice the gain: how to simultaneously decrease animal numbers and increase precision with good experimental design. Laboratory Animals. 2024;58(5):411–418. doi:10.1177/00236772241260905 read 15 October 2026. View on Laboratory Animals (SAGE)A peer-reviewed methods paper on why factorial designs beat varying one factor at a time. Read on the publisher's own site.
  6. Percie du Sert N, Hurst V, Ahluwalia A, et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. PLoS Biology. 2020;18(7):e3000410. doi:10.1371/journal.pbio.3000410 read 15 October 2026. View on PLOS BiologyAnimal Research: Reporting of In Vivo Experiments, version 2.0. Developed by an international working group and hosted by the NC3Rs; none of the pages we read states who maintains it, so no maintainer is asserted.
  7. ARRIVE guidelines 2.0, item 2b: Sample size justification — explanation and elaboration. UK EQUATOR Centre, after Percie du Sert et al., PLoS Biology 2020;18(7):e3000410. read 15 October 2026. View on UK EQUATOR CentreThe source for sample size being determined in advance, and for what underpowering costs. Useful because it is a reporting guideline rather than a clinical-trial guideline, so it applies to laboratory work.
  8. Festing MFW. The design and statistical analysis of animal experiments. ILAR Journal. 2002;43(4):191–193. read 15 October 2026. View on a university-hosted copyCited for the point that the analysis method belongs to the design stage, not the end of the study. Read from a university copy rather than the publisher.
  9. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, Munafò MR. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience. 2013;14:365–376. doi:10.1038/nrn3475 read 15 October 2026. View on a third-party full-text mirrorThe source of the 21% median power figure, which is specific to neuroscience. Read from a mirror; the volume and page range were confirmed separately against a publisher-side reference list.
  10. Ioannidis JPA. Why most published research findings are false. PLoS Medicine. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124 read 15 October 2026. View on a university-hosted copyCited for the relationship between study size, flexibility and the probability that a finding is true. Read from a university journal-club copy rather than the publisher.
  11. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011;22(11):1359–1366. doi:10.1177/0956797611417632 read 15 October 2026. View on Psychological Science (SAGE)The source of the 61% figure. The citation and abstract were confirmed on the publisher's page; the simulation table was read from a proof version hosted by a university, which is noted where those numbers are used.
  12. Scheel AM, Schijen MRMJ, Lakens D. An excess of positive results: comparing the standard Psychology literature with Registered Reports. Advances in Methods and Practices in Psychological Science. 2021;4(2):1–12. doi:10.1177/25152459211007467 read 15 October 2026. View on Association for Psychological ScienceThe source of the 96% versus 44% comparison. Read on the publishing society's own page and corroborated against two repository records; the publisher's article page was not reachable.
  13. Chambers CD, Tzavella L. The past, present and future of Registered Reports. Nature Human Behaviour. 2022;6:29–42. doi:10.1038/s41562-021-01193-7 read 15 October 2026. View on Cardiff University repositoryCited for what a Registered Report is. Read as the repository record rather than the full text, so nothing is quoted from the body and no figures are taken from it.
  14. Center for Open Science. Preregistration and Registered Reports. read 15 October 2026. View on Center for Open ScienceCited for the bare definition only. The page does not define the two practices separately or discuss publication bias.

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each carries the date it was last checked.

Related: principles of experimental design · variables and what each implies · biostatistics in clinical research.

How Pubrica Can Help

Our biostatisticians and research designers prepare experimental plans before data collection, covering the design and its blocking structure, a priori sample size and power, the statistical analysis plan, and the reporting-guideline checklist completed at the design stage rather than at submission.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.