Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data, Analytics & AI · Analyzing Research Data

Analyzing Research Data: Key Steps for Manuscript Preparation

A widely cited 2011 demonstration showed that sufficient undisclosed flexibility in data cleaning and analysis can raise the false-positive rate for a simple comparison well above the nominal 5% threshold. For this reason, the sequence of analytical decisions made before writing begins is now the subject of documented best practices.

Pubrica AcademyWritten by a research data methodologist Reviewed bya reproducibility reviewer Published10 October 2026 Last checked10 October 2026 Reading time8 minutes Sources11 peer-reviewed

Before a manuscript's results section can be written, the underlying data has to be organized, cleaned, explored, and formally analyzed, and the order and discipline of those steps matter independently of which statistical test eventually gets reported. This stage carries its own well-documented risks, chief among them the undisclosed flexibility in analytical choices that a landmark 2011 demonstration showed can inflate a nominal 5% false-positive rate far beyond what researchers or readers typically assume, simply through small, individually reasonable-seeming decisions made during analysis rather than through any single dramatic error.

Learning objectives

  • What a 2011 demonstration study showed about how analytical flexibility inflates false-positive rates
  • The practical distinction between exploratory and confirmatory data analysis, and why conflating them is a documented problem
  • What 'tidy data' means as a formal concept, and why its originator argues it is a prerequisite for reliable analysis

Common misconceptions

  • “Provided that nothing is fabricated or falsified, the normal iterative exploration of data before an analysis is settled is harmless.” Joseph Simmons, Leif Nelson, and Uri Simonsohn demonstrated in 2011 that researcher degrees of freedom (undisclosed, individually defensible choices such as when to stop collecting data, which covariates to include, or which of several similar outcome measures to report) can combine to raise the true false-positive rate for a comparison well above the conventional 5% significance threshold, even when every individual choice appeared reasonable and no outright fabrication occurred.

Inflation of false-positive rates by undisclosed analytical flexibility

Simmons, Nelson, and Simonsohn’s 2011 paper, titled “False-Positive Psychology,” used both simulation and a deliberately absurd real experiment to demonstrate that a researcher with multiple small, flexible decisions available (when to stop collecting data, which of several measured outcomes to analyze and report, which covariates to control for, and which comparison groups to combine) can produce a “statistically significant” result from essentially any dataset, including one with no real underlying effect1. Their central point was that no single one of these choices appears to constitute misconduct in isolation; each is individually defensible, which is precisely why the practice had remained largely unaddressed before their paper made the cumulative effect concrete and quantifiable1. A broader methodological response to this and related findings, published as “A manifesto for reproducible science” by Marcus Munafò and colleagues, catalogued a set of structural remedies, including pre-registration of analysis plans and blinded analysis, intended specifically to prevent this kind of undisclosed flexibility in advance rather than to detect it after publication2. The central implication of this finding is that no single step in the chain needs to be dishonest for the cumulative effect to appear. A researcher who tries, in good faith, two reasonable ways of handling an ambiguous outlier, then two reasonable choices of covariate, and then reports whichever combination reached significance without disclosing the others, has produced a misleading result through a sequence of individually ordinary decisions rather than through any single act of misconduct. For this reason, structural safeguards applied before the data are seen are more reliable than reliance on individual researchers’ good intentions after the fact, and pre-registration is recommended as a default practice rather than reserved for projects in which a researcher suspects that their own judgment might be compromised.

Exploratory and confirmatory analysis and the importance of their sequence

John Tukey’s foundational work on exploratory data analysis, first formalized in his 1977 text, drew a deliberate distinction between analysis aimed at discovering unexpected patterns in data and analysis aimed at formally testing a pre-specified hypothesis, arguing that both have genuine value but serve different purposes and carry different evidentiary weight3. The problem that Simmons and colleagues identified concerns less exploratory analysis itself than the presentation of an exploratory, flexible process as if it had been a single pre-specified confirmatory test1. Brian Nosek and colleagues, writing on what they termed “the preregistration revolution,” argued that formally separating these two modes, by registering a confirmatory analysis plan before the outcome data are seen while still permitting, and clearly labeling, additional exploratory analysis afterward, gives readers a transparent basis for judging how much confidence a given reported result deserves4. For secondary analyses of data collected for another purpose, a preregistration template and tutorial developed by Van den Akker and colleagues extends the same logic to situations in which the data already exist before the confirmatory question is formally specified5.

Data organization as a formal prerequisite

Hadley Wickham’s “tidy data” framework defines a standard way of structuring a dataset (each variable in its own column, each observation in its own row, and each type of observational unit in its own table) because inconsistent, ad hoc data structures are a major and underappreciated source of analysis errors and wasted effort before any statistical test is run6. Karl Broman and Kara Woo’s practical guide to organizing data in spreadsheets catalogues the specific recurring mistakes (inconsistent date formats, merged cells, the use of color alone to encode meaning, and the absence of a clear distinction between raw and processed data) that make a dataset harder to analyze correctly and harder for others to check later7. The FAIR Guiding Principles, published by Mark Wilkinson and a large international group of co-authors, extend the same concern for careful organization to entire datasets that are shared or archived, specifying that data should be Findable, Accessible, Interoperable, and Reusable as a formal standard for research data management rather than treating good organization as a matter of individual preference8.

Version control in research data workflows

A version control system, software originally developed for tracking changes to programming code, records every change made to a file over time, together with who made it and when, and allows any prior version to be recovered exactly; this capability is as important for a research data-analysis pipeline as it is for software. Karthik Ram, writing specifically on applying the version control system Git to scientific workflows, argued that the practice directly addresses two common and avoidable research failures: losing track of which version of an analysis script produced a specific reported result, and being unable to reconstruct what changed between an earlier and a later version of an analysis when a collaborator or reviewer asks why a number shifted11. This concern relates directly to the “good enough practices” framework discussed below, since version control of both the data-processing code and the manuscript text is one of the low-overhead practices that the framework recommends every research team adopt as a baseline, rather than an advanced technical skill reserved for computational specialists9. In practice, when a reviewer or co-author asks exactly how a reported figure was produced, a research team with disciplined version control can answer precisely, down to the specific script and commit, rather than reconstructing the answer from memory or from an unlabeled folder of similarly named file versions.

Good-enough computing practices for a verifiable analysis trail

Greg Wilson and a large group of co-authors published a widely cited set of “good enough practices in scientific computing,” arguing that a research team does not need an elaborate software-engineering setup but does need a minimum, specific discipline: keeping raw data unaltered and separate from processed data, recording every step taken to derive one from the other (ideally as a script rather than as manual spreadsheet edits), and maintaining version control of both the data-processing code and the manuscript itself9. Jeffrey Leek and Roger Peng, in a short but widely cited article on the importance of correctly matching the type of question being asked (descriptive, exploratory, inferential, predictive, causal, or mechanistic) to the analysis actually performed, argue that a substantial share of avoidable analytical errors in published research stems not from a flawed statistical technique but from applying a technique suited to one kind of question to a dataset intended to answer a different kind10. Together, these sources describe a consistent discipline for the data-analysis stage of manuscript preparation: organizing data to a recognized standard, keeping exploratory and confirmatory analyses explicitly separate and labeled, and maintaining a complete, reproducible record of every processing and analytical step. None of these practices requires specialized computational training beyond that of a typical quantitative researcher; the barrier is usually habit and project-management discipline rather than technical skill, which is also why each of the sources cited here frames its recommendations as “good enough” baseline practice rather than as an aspirational standard reserved for computational specialists.

Key points

Key points

The data-analysis stage of manuscript preparation carries documented risks of its own, chief among them undisclosed analytical flexibility that can inflate false-positive rates well beyond the nominal threshold. Structured data organization, labeled separation of exploratory and confirmatory analyses, and a reproducible processing trail are specifically designed to prevent this.

Practical application

Practical application

  1. The confirmatory analysis plan should be decided and recorded before the data are explored freely. The primary outcome, covariates and planned comparisons should be pre-specified, and any additional analysis performed afterward should be clearly labeled as exploratory in the manuscript itself.
  2. The dataset should be organized to a recognized tidy-data standard before analysis begins. This entails one variable per column, one observation per row, and a clear, documented separation between raw and processed versions of the data.
  3. A scripted, version-controlled record of every data-processing step should be maintained. Manual, undocumented spreadsheet edits between the raw data and the dataset entered into statistical software should be avoided, so that the full analytical path can be reconstructed and checked.

Frequently asked questions

No. Tukey’s foundational framework treats exploratory analysis as genuinely valuable for discovering patterns; the documented problem is the presentation of an exploratory, flexible process as if it had been a single pre-specified confirmatory test, which misrepresents the evidentiary weight that the result deserves.

Pre-registration of the primary analysis plan, including the outcome measure, covariates, and planned comparisons, before the outcome data are examined is the structural remedy proposed in the reproducibility literature cited here.

No. It is a formally defined structure (each variable in its own column, each observation in its own row, and each type of unit in its own table), developed because inconsistent structures are a major source of downstream analysis errors, not merely an aesthetic preference.

Yes, and the preregistration literature cited here explicitly allows for it; the key requirement is to label clearly which reported results derive from the pre-specified confirmatory plan and which derive from additional exploratory analysis conducted afterward.

References

  1. Simmons JP, Nelson LD, Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science. 2011;22(11):1359-1366. read 10 October 2026. View on Psychological Science
  2. Munafò MR, Nosek BA, Bishop DVM, et al. A manifesto for reproducible science. Nature Human Behaviour. 2017;1:0021. read 10 October 2026. View on Nature Human Behaviour
  3. Tukey JW. Exploratory Data Analysis. Addison-Wesley; 1977. read 10 October 2026. View publisher record
  4. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences. 2018;115(11):2600-2606. read 10 October 2026. View on PNAS
  5. Van den Akker O, Peters G, Bakker C, et al. Preregistration of secondary data analysis: a template and tutorial. Meta-Psychology. 2021;5. read 10 October 2026. View on Meta-Psychology
  6. Wickham H. Tidy data. Journal of Statistical Software. 2014;59(10):1-23. read 10 October 2026. View on Journal of Statistical Software
  7. Broman KW, Woo KH. Data organization in spreadsheets. The American Statistician. 2018;72(1):2-10. read 10 October 2026. View on The American Statistician
  8. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data. 2016;3:160018. read 10 October 2026. View on Scientific Data
  9. Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal TK. Good enough practices in scientific computing. PLOS Computational Biology. 2017;13(6):e1005510. read 10 October 2026. View on PLOS Computational Biology
  10. Leek JT, Peng RD. What is the question? Science. 2015;347(6228):1314-1315. read 10 October 2026. View on Science
  11. Ram K. Git can facilitate greater reproducibility and increased transparency in science. Source Code for Biology and Medicine. 2013;8:7. read 10 October 2026. View on Source Code for Biology and Medicine

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 10). Analyzing Research Data: Key Steps for Manuscript Preparation. Pubrica. https://pubrica.com/academy/concepts-definitions/research-data-key-steps-for-manuscript-preparation/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Data Analysis Stage Checklist

This checklist should be completed before the manuscript’s methods and results sections are written.

[ ] Pre-specified the primary confirmatory analysis plan before freely exploring the outcome data
[ ] Labeled all additional exploratory analyses clearly as exploratory in the manuscript
[ ] Organized the dataset to a tidy-data standard (one variable per column, one observation per row)
[ ] Kept raw data untouched and separate from any processed/cleaned version
[ ] Scripted every data-cleaning and transformation step rather than editing manually
[ ] Matched the analytical technique used to the actual type of question being asked

Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.

How Pubrica can help

Pubrica’s data analytics team organizes and cleans research data to a tidy, reproducible standard and assists in separating confirmatory from exploratory analysis before the methods and results sections of a manuscript are drafted.

Find something else

Search the Pubrica Academy

Access is free and requires no account. Enter a topic, or open the full library to filter by category, sub-topic or resource type.

Identifying undisclosed analytical flexibility in research results

A scoping assessment is free and takes two business days. It provides a written assessment of what the work requires and which specialist would undertake it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.