Free scoping read. A written assessment of your manuscript in two business days, at no cost.

Start one
Contents

Academy · Research Data Analytics & AI · reviewing a web tool

NetworkAnalyst and the Problem of Reviewing a Web Tool

A systematic review of a bioinformatics web tool such as NetworkAnalyst is not an intervention review and cannot be made into one. The object under review is a versioned piece of software, the primary evidence about it is written by the people who built it, and the secondary evidence is a body of application papers that mostly do not report which version or which parameters they used. A review that acknowledges all three constraints is useful; one that pretends to be a review of effects is not.

Pubrica AcademyWritten by the medical writing and biostatistics teams Reviewed byPubrica biostatistics team Published9 October 2026 Last checked9 October 2026 Reading time16 minutes Sources36, including 3 NetworkAnalyst papers

Tools for integrative visualisation of expression data sit at the point in an analysis where decisions become invisible. A researcher uploads a count matrix, accepts a set of defaults, and gets a network diagram that will be published. Reviewing a tool systematically means reconstructing those decisions across a literature that rarely records them — which turns out to be both the central difficulty and the most useful finding such a review can produce.

NetworkAnalyst at a glance

What it is
A free web-based platform for gene expression profiling, meta-analysis and network-based visual analytics36
Key papers
Xia et al. 2014 and Zhou et al. 2019 in Nucleic Acids Research; Xia et al. 2015 protocol in Nature Protocols34,35,36
Inputs accepted (version 3.0)
Gene lists, a single expression table, multiple expression tables, raw RNA-seq reads and network files36
Reported use
More than 220,000 analysis jobs from more than 14,000 users in a 12-month period, as reported by the developers36
This article’s question
How to review a tool like this systematically, when the object changes between versions
Reading time
About 16 minutes, with 3 figures and 4 tables

What you will learn

  • Identify what kind of object a tool is, and why it needs a different review frame from an intervention
  • Write a tool review question that is answerable from the evidence that actually exists
  • Find the evidence about a tool — the original paper, update papers, application papers, documentation and repositories
  • Extract a tool version record and an application record, and keep them separate
  • Appraise application papers on the analytical choices they report, and treat silence as a finding
  • Report a tool review so that it stays interpretable after the tool changes, which it will

What to unlearn

  • “A tool review is just a systematic review with software as the intervention.” There is no outcome in a patient, no comparator arm and no effect to estimate. Borrowing the intervention-review frame produces a review with an empty synthesis section.
  • “The tool paper tells you how well the tool works.” The original and update papers are written by the developers and report performance on examples they chose. That is legitimate and it is not independent evaluation.
  • “Citation count measures a tool’s validity.” It measures adoption. A widely used tool with defaults nobody examines is more consequential than a rarely used one, not better validated.
  • “The version does not matter.” A web tool updates without notice: the method behind a button can change between two papers that cite the same reference. Without a version and a date, a reported result is not attributable to anything specific.
  • “If the application papers do not report their parameters, that is a gap in the papers, not the review.” It is both, and quantifying it is the review’s most transferable output. A field-level account of what is not reported is something only a reviewer can produce.

What NetworkAnalyst is and what it does

NetworkAnalyst is a web-based tool for interpreting gene expression data in the context of biological networks. Its first description, by Xia, Benner and Hancock in Nucleic Acids Research in 2014, set out the problem it addressed: network analysis requires three consecutive steps — identifying genes or proteins of interest, constructing a network, and analysing and visualising it — and researchers previously had to combine several tools to complete them. NetworkAnalyst integrated the three steps in a browser, mapping significant genes to a manually curated protein–protein interaction database and supporting human and mouse data 34.

A 2015 protocol in Nature Protocols extended its purpose to statistical, visual and network-based meta-analysis of gene expression data, describing the tool as designed to be accessible to biologists rather than to specialist bioinformaticians and the complete protocol as executable in about 1.5 hours 35. Version 3.0, described by Zhou and colleagues in 2019, added expression profiling for 17 species, raw RNA-seq processing, tissue-specific and regulatory networks, gene set enrichment analysis, 3D and virtual-reality network views, and saved analysis projects 36.

2014First releaseXia, Benner and Hancock, Nucleic Acids Research: protein–protein interaction (PPI) network analysis and visual exploration for human and mouse34
2015Version 2.0 and protocolRevamped interface and workflow for statistical meta-analysis of multiple expression studies; step-wise protocol by Xia, Gill and Hancock in Nature Protocols35,36
2019Version 3.0Zhou et al., Nucleic Acids Research: 17 species, raw RNA-seq processing, new network types, 3D and virtual-reality views, saved projects36
Figure 1. Published versions of NetworkAnalyst. Each paper describes a different tool: an application paper citing the 2014 article may have used modules that did not exist until 2019.
1Upload one of five input types; raw reads are processed on a Galaxy server with HISAT2 or kallistoInput
2Quality check and normalisation, with box, density and PCA plotsPreparation
3Differential expression with limma, edgeR or DESeq2; enrichment by over-representation or GSEAStatistics
4Build PPI, gene regulatory, co-expression or chemical and drug networks and explore them in 2D, 3D or VR, or compare gene lists by heatmap, Venn or chord diagramNetworks and visual analytics
Figure 2. The NetworkAnalyst 3.0 workflow as described by its developers. Every step contains choices (method, threshold, network order, database) that a published result depends on.36

What each published version added, and what it means for a review

VersionAdded, according to the developersConsequence for a tool review
2014 (first release)Gene annotation and differential expression; mapping of significant genes to a manually curated PPI database; network topology and module analysis; export as PNG, PDF or GraphML; human and mousePapers citing only this article may describe a much smaller tool than the one they used
2015 (version 2.0, protocol)Workflow for meta-analysis of multiple gene expression datasets, designed for biologists rather than specialist bioinformaticians; the protocol runs in about 1.5 hoursMeta-analysis results need the version and the meta-analysis method stated
2019 (version 3.0)17 species; tissue-specific PPI from 42 tissues; transcription factor and miRNA regulatory networks; co-expression, toxicogenomic and drug networks; GSEA; 3D and VR views; saved projectsNetwork type and database source become extraction items in their own right

Sources: Xia et al. (2014), Xia et al. (2015) and Zhou et al. (2019). Descriptions are the developers’ own; none is an independent evaluation.

Capacity limits and design boundaries stated for NetworkAnalyst 3.0

FeatureStated limitWhy a reviewer should record it
Raw RNA-seq processing (Galaxy server)100 GB disc space per user by default, about 30–50 samplesLarge studies may have been processed elsewhere and only partly analysed in the tool
Single expression tableMaximum 50 MB, about 200 samples with about 25,000 genes eachSets the study size the tool was designed for: small to medium
Meta-analysisUp to 1,000 samples in totalDefines the scale of pooled analyses that could have been run on the public server
Venn diagram comparisonUp to four gene listsLarger comparisons must have used heatmaps or another method
Saved projectsUp to 10 per account; an account is optionalAnalysis steps are tracked only if the user created a project
Large-scale studiesDevelopers recommend local processing, then uploading gene listsPart of the analysis may sit outside the tool and must be extracted separately

Source: Zhou et al. (2019), Implementation section. Limits apply to version 3.0 and may have changed since.

Species supported by four web-based expression tools, as tabulated by NetworkAnalyst’s developers

g:Profilerfirst released 2007213
NetworkAnalyst 3.02019 release17
WebGestaltfirst released 200512
Enrichrfirst released 20136
Figure 3. Number of species supported, from Table 1 of Zhou et al. (2019); bar lengths are proportional to the largest value. The comparison was compiled by the developers of one of the four tools, which is exactly the kind of evidence a systematic tool review must label as non-independent.36

The developers report that over a 12-month period the public server performed more than 220,000 analysis jobs submitted by more than 14,000 users, while noting that many jobs were probably re-analyses of the same datasets by the same users 36. That figure measures adoption, not validity, a distinction this article returns to below.

From describing the tool to reviewing it

The three papers above are the natural starting point for anyone who wants to know what NetworkAnalyst does. They are also the reason a systematic review of it is unusual. They are written by the developers; they describe three different versions; their comparison with other tools is self-compiled; and the usage figure counts jobs rather than correct results. A review of NetworkAnalyst therefore has to treat the tool as a changing object with a version history, and the papers that use it as the evidence. The rest of this article sets out how.

What you are actually reviewing

Be precise about the object, because four different things get called “the tool” and they have different evidence bases.

There is the method — the statistical or algorithmic procedure the tool implements, which usually predates the tool and has its own literature. There is the implementation: the code that realises the method, with its defaults, its thresholds and its bugs. There is the interface, which in a web tool is a set of decisions about what the user is asked, what is and what is pre-selected — and defaults are consequential, because most users accept them. And there is the deployed instance at a particular version on a particular date, which is the only thing a published result was actually produced by.

A review has to say which of the four it is about 25. A review of the method is a methods review and belongs in the statistical literature. A review of the implementation requires reading or running code. A review of the interface and its defaults is a usability and reporting question. A review of the deployed instance is a snapshot with an expiry date. Most published “systematic review of tool X” papers conflate all four, which is why they read as descriptive catalogues rather than as reviews.

For a tool like NetworkAnalyst, whose role is integrative visualisation and analysis of expression data, the most defensible object is usually the implementation-plus-interface as used in the published literature: what the tool offers, what users of it actually selected, and what they reported. See our page on reviewing developments in bioinformatics for the same problem at field scale.

Why PICO does not fit, and what to use instead

PICO requires a population, an intervention delivered to it, a comparator and an outcome measured in it. A tool review has none of these in the required sense. The “population” is a set of papers or datasets, the “intervention” is an analysis choice, there is usually no comparator, and the “outcome” is a result whose correctness is generally unknown.

A frame that does fit has four elements. The object: the tool, module and version under review. The use case: the analytical task, such as differential expression from bulk RNA sequencing counts, meta-analysis across expression datasets, or network-based enrichment. The claim: what is being asserted — that the tool implements a named method correctly, that it is usable by a given audience, that results obtained with it are reproducible, or that its use in the literature is adequately reported. And the evidence type that could support or undermine that claim.

Writing these four down forces the useful question early: is there any evidence that could answer this, and who produced it? If the only available evidence is developer-authored, say so in the protocol and scope the review to what that evidence can support 5. Our page on framing review questions covers why forcing an unfitting framework is worse than naming a different one.

Four questions a tool review can actually answer

Each of these is answerable from existing literature. Mixing them in one undeclared question is what produces an unstructured review.

What does it do, and how has that changed? Answerable from the original paper, the update papers and the documentation, read as a version history. This is descriptive and it is useful, because users routinely cite the original paper while using a module introduced years later.

Has anyone evaluated it independently? Answerable from benchmarking studies and comparison papers, where they exist. Often the honest answer is that independent evaluation is sparse or absent, and reporting that plainly is more valuable than assembling developer-reported performance into a table that looks like a comparison.

How has it been used, and was that use reported adequately? Answerable from application papers, and this is where a systematic approach earns its keep. A structured sample of papers using the tool, assessed against a fixed list of items — version, module, input type, normalisation, statistical method, thresholds, multiple-testing correction, date of access — produces a quantified account of reporting practice in that field.

Can published analyses using it be reproduced? Answerable only by attempting it, with the original data, which turns the project into a reproducibility study with a systematic sampling frame rather than a review. That is a legitimate and valuable design, and it must be labelled as what it is.

Four answerable questions about a tool, and the evidence each one needs

QuestionEvidence that answers itWho produced that evidenceWhat the review may conclude
What does it do, and how has it changed?Original paper, update papers, documentation, release historyThe developersA version history and a description of current capability, date-stamped
Has it been independently evaluated?Benchmarking and comparison studiesThird parties, where they exist — often sparse or absentWhat independent evaluation exists, or that it does not
How has it been used, and was that use reported adequately?A structured sample of application papers, assessed against a fixed item listThe review itselfQuantified reporting practice in the field — usually the most transferable finding
Can analyses using it be reproduced?Re-analysis of published data with the toolThe review team, which makes this a reproducibility studyWhether sampled analyses reproduced, and why failures could not be resolved

Compiled as a working framework for reviews whose object is software rather than an intervention.

Where the evidence about a tool lives

A conventional bibliographic search will find the tool papers and some application papers, and will miss much of what matters. Plan for five source types.

Bibliographic databases for the original paper, update papers, benchmarking studies and application papers. Because tools are cited by name rather than indexed as subjects, the search needs the tool name, its variants and common misspellings, alongside the method and task terms 1,12. See sources of review data for the general structure.

Citation tracking on the tool papers, which is the single highest-yield method for finding application papers and is frequently the primary identification route in a tool review. Report it as such, with the index used and the date 3.

Software repositories and release histories, which carry the version record that the papers do not: when modules appeared, when methods changed, when defaults were altered. This is primary evidence about the object under review and it belongs in the search, with the date you accessed it.

Documentation, tutorials and help pages, which define the defaults. Defaults are the single most consequential thing about a web tool, because they are what most users accept, and they are usually documented nowhere else.

Data repositories referenced by the application papers, which is what makes any reproducibility component possible and which also tells you whether the field’s raw data is available at all.

Two extraction records, kept separate

Tool reviews go wrong at extraction when one form tries to capture both the software and the papers using it. Build two.

The tool version record has one row per release or documented change: version identifier or date, modules available, methods implemented per module, defaults for each consequential parameter, accepted input types, and what changed from the previous version. Where the information does not exist — which is common for web tools without a public changelog — record that it does not, because an undocumented change history is a finding about reproducibility in the field.

The application record has one row per paper, with the items a reader would need to reproduce the analysis: tool and version as stated, module used, date of access, input data type and source, preprocessing and normalisation, the statistical method selected, effect-size and significance thresholds, multiple-testing correction, annotation or pathway database and its version, and what was reported as the result. Each item gets three possible states: reported, reported ambiguously, or not reported. The third state is data, not a blank.

The same discipline applies as in any review: extract what the paper says rather than what you can infer, record where you found it, and have two people do it independently. The general principles are on our pages on what data to extract and handling discrepancies.

Tool version record: minimum fields per release

  • Version identifier or release date, and the paper or changelog that documents it.
  • Modules available, and the input types each accepts.
  • Statistical methods offered per module, for example limma, edgeR or DESeq2 for differential expression.
  • Default value of each consequential parameter, including thresholds and network order.
  • Interaction and annotation databases used, with their versions where stated.
  • Stated capacity limits, such as maximum upload size or sample number.
  • What changed from the previous version, or a note that no change history is available.

Appraising application papers on what they chose

There is no risk-of-bias instrument for “paper that used a web tool”, and inventing a scored one is worse than using a transparent item list. Assess against items whose absence has known consequences, and report item by item rather than as a composite score.

Four items carry most of the weight in expression analysis. Normalisation and preprocessing, because the same count matrix analysed under different normalisation gives different gene lists. Statistical method and thresholds, because a tool typically offers several methods and the choice changes the output; a paper that names neither is not reproducible even in principle. Multiple-testing correction, which in a genome-wide analysis is the difference between a result and a list of coincidences — see our checklist for reviewing genome-wide literature for the same issue in association studies. And batch and confounding structure, particularly where the analysis integrates datasets from different sources, which is exactly the use case an integrative tool exists to serve.

Two further items are specific to web tools and almost never reported: the version used, and the date of access. Without them, no result is attributable to a defined piece of software. Count how many papers in your sample report each, and that count is the review’s headline finding about reporting practice.

Resist converting these into a quality score. A composite score hides which item failed, and the point of the exercise is to say which items the field does and does not report.

Reporting items for a paper that used a web-based expression analysis tool

ItemWhy its absence mattersRecord as
Tool name and versionA web tool changes without notice, so a result with no version is not attributable to any defined softwareReported / ambiguous / not reported
Date of accessThe only proxy for version when no version is givenReported / ambiguous / not reported
Module or workflow usedAuthors frequently cite the original paper while using a module added years laterReported / ambiguous / not reported
Input data type and source accessionDetermines whether the analysis could be repeated at allReported / ambiguous / not reported
Preprocessing and normalisationThe same count matrix under different normalisation yields a different gene listReported / ambiguous / not reported
Statistical method selectedTools offer several; the choice changes the output and the default is not always the one usedReported / ambiguous / not reported
Effect-size and significance thresholdsWithout them the reported list cannot be regeneratedReported / ambiguous / not reported
Multiple-testing correctionIn a genome-wide analysis, its absence is the difference between a result and a list of coincidencesReported / ambiguous / not reported
Batch and confounding handlingCritical where the analysis integrates datasets from different sources, which is the integrative use caseReported / ambiguous / not reported
Annotation or pathway database and its versionEnrichment results depend on the annotation release, which changes continuouslyReported / ambiguous / not reported

Compiled as a transparent item list; report item by item rather than as a composite quality score, so that a reader can see which item the field omits.

The methods sentence in a paper that used the tool

Attributable to nothing specific

Differentially expressed genes were identified and functional enrichment and network analysis were performed using NetworkAnalyst with default settings.

No version and no access date. A web tool’s behaviour changes between releases, so this sentence does not identify the software that produced the result.

“Default settings” is not a parameter record. Defaults change, are not always documented historically, and a reader cannot recover what they were on the day the analysis ran.

No module named. Authors routinely cite a tool’s original paper while using a workflow added long afterwards, so the citation misdirects the reader.

No normalisation or statistical method. The same input under different normalisation and a different test produces a different gene list; nothing here constrains which was produced.

No thresholds and no multiple-testing statement. The reported gene list cannot be regenerated, and in a genome-wide analysis the correction is what separates a result from a coincidence.

No batch or platform handling. This is an integrative analysis across datasets, which is precisely the situation in which unadjusted batch structure produces spurious differential expression.

No annotation versions. Enrichment and network results depend on database releases that change continuously, so the result has no fixed referent.

Reproducible by a stranger

Raw count data for the two datasets were obtained from the repository accessions listed in Table 1 and analysed in NetworkAnalyst, stated version, accessed on a stated date, using the named module for multi-dataset expression analysis. Counts were filtered at the stated low-abundance threshold and normalised by the named method as implemented in that module; the differential expression test used was the named method with the stated design matrix, and genes were called differentially expressed at the stated absolute log fold-change threshold and the stated adjusted p value threshold after Benjamini–Hochberg correction. Because the two datasets came from different platforms and institutions, dataset was included as a covariate in the model, and the alternative of combining datasets without adjustment is reported as a sensitivity analysis in the supplement. Functional enrichment used the named pathway database at its stated release version, with the stated background gene set rather than the whole annotated genome. Network construction used the named interaction database at its stated version, with the stated confidence filter and the stated node limit, and the layout algorithm is named. All parameter selections that departed from the module defaults are listed in Supplementary Table 2, and the defaults as they stood on the date of access are recorded alongside them.

Version and access date are given, so the result is attributable to a defined piece of software at a defined moment.

Every consequential parameter is stated, and departures from defaults are listed separately alongside the defaults as they then stood — which is what makes the analysis repeatable after the tool changes.

The module is named, so the method citation points at the thing actually used.

Normalisation, test, design and thresholds are all specified, so the gene list can be regenerated rather than merely believed.

Multiple-testing correction is named, which turns the reported list into a statistical claim with a defined error rate.

Cross-dataset batch structure is modelled and the unadjusted alternative reported as a sensitivity analysis, which is the honest handling for an integrative analysis.

Annotation and interaction database versions are recorded, so enrichment and network findings are anchored to a release rather than to a moving target.

A review that counts how many papers in a field write the first version and how many write the second has produced a quantified, reusable statement about reproducibility in that field — which is the one thing a tool review can establish that nobody else is positioned to.

Running the tool yourself, and what that makes the project

At some point in a tool review the temptation arrives to re-run a published analysis and see whether the result comes out. This is worth doing and it changes the nature of the project, so decide and declare rather than drifting.

If you re-analyse published data with the tool, you are conducting a reproducibility study. It needs its own design: a sampling frame for which published analyses you attempt, a pre-specified definition of what counts as reproduced — identical gene list, substantially overlapping list, same top-ranked features, same biological conclusion — and a protocol for how you resolve the unreported parameters, which is the central difficulty. Record the version and date you ran, because your own result is as version-dependent as theirs.

Done well, a systematically sampled reproducibility study is more informative than any descriptive review of the same tool, and it is honest about its own limits: a failure to reproduce may reflect an unreported parameter rather than an error, and you usually cannot tell which. Report attempts, outcomes and the reason each failure could not be resolved.

Done casually — a couple of re-runs mentioned in the discussion of a review — it is an anecdote that implies more than it establishes. If you cannot give it a design, leave it out and report instead the reporting items that would have to exist for reproduction to be possible.

Reporting a review whose object will change

Report the review process in accordance with the PRISMA 2020 statement — the search, the selection, the flow of records and the synthesis approach are all ordinary review reporting, and the fact that the subject is software changes none of it 10. If the design is a mapping exercise rather than an effect estimate, report it as a scoping review and say so 25.

Then add the items a tool review specifically needs. Date-stamp everything: the version of the tool examined, the dates on which the tool, its documentation and its repository were accessed, and the date range of the application papers. State explicitly that the tool may have changed since, because it will have. Publish the version record and the per-item reporting counts as supplementary tables; they are the parts of the review that other people will reuse 13.

Say plainly who generated the performance evidence you report. “Performance as reported by the developers” and “performance reported in independent benchmarking” are different claims and should not appear in the same column of the same table without labels.

For a fast-moving tool, consider a living review format with a stated update cycle, or write a state-of-the-art review that does not claim completeness at a fixed date. Either is more honest than a one-off systematic review of a moving target.

Weak methods statement (illustrative)

“Differentially expressed genes were identified and a PPI network was constructed using NetworkAnalyst.”

No version, no access date, no statistical method, no thresholds, no network order or interaction database. Nothing in it can be extracted or reproduced.

Strong methods statement (illustrative)

“Counts were analysed in NetworkAnalyst 3.0 (accessed on a stated date) using DESeq2, with an adjusted p-value below 0.05 and an absolute log2 fold change above 1. A first-order, tissue-specific PPI network was built from the significant genes and enrichment was tested with GSEA.”

Every item in the application record can be marked as reported. The thresholds are examples, not recommendations.

What a tool review cannot settle

Four limits, worth writing into the discussion rather than leaving a reader to infer.

It cannot establish that the tool is correct. Correctness is a property of the implementation against the method, and establishing it requires reading or testing code against known inputs with known expected outputs, which is a software verification exercise rather than a literature review.

It cannot establish that one tool is better than another. Developer-reported comparisons are not independent, independent benchmarks are sparse in most tool areas, and benchmark results depend heavily on the datasets chosen. A review can report which comparisons exist and who ran them; a review that ranks tools on assembled developer claims has produced a ranking of marketing material.

It cannot tell a reader what the right parameters are. Documenting what users chose is not the same as establishing what they should choose, and a review that slides from the first to the second has substituted popularity for evidence.

And it expires. Everything in a tool review is true of a version at a date. That is not a defect to be apologised for; it is a property of the object, and the right response is to date-stamp the review and state the limitation specifically rather than writing a generic sentence about the pace of the field. Support for this kind of work sits with our bioinformatics team and research data service.

Take-home message

Take-home message

Reviewing a bioinformatics web tool means reviewing a versioned implementation, an interface whose defaults most users accept, and a literature of application papers that largely does not record which version or parameters were used. Say which of those you are reviewing, drop PICO in favour of an object-use case-claim-evidence frame, and pick one of four answerable questions: what the tool does and how it has changed, whether it has been evaluated independently, how it has been used and whether that use was reported adequately, or whether published analyses using it can be reproduced. Keep two extraction records — a version record from the release history and documentation, and an application record per paper with every reporting item coded as reported, ambiguous or not reported. Assess item by item rather than as a quality score, label developer-reported performance as such, and date-stamp the whole review, because its object will have changed by the time anyone reads it.

How to apply this to your study

How to apply this to your own study

  1. Name the object in one sentence. Method, implementation, interface and defaults, or deployed instance at a version — pick one and write it into the protocol.
  2. Replace PICO with four elements. Object, use case, claim, evidence type — then check that evidence of that type exists before committing to the review.
  3. Make citation tracking a primary identification method. Tools are cited by name rather than indexed, so report the citation index used and the date as part of the search.
  4. Build the version record from the repository, not the papers. Releases, module additions, method changes, default changes — and record where no changelog exists.
  5. Code every reporting item three ways. Reported, reported ambiguously, not reported — and publish the per-item counts, which are the review’s most reusable output.
  6. Label who produced each performance claim. Developer-reported and independently benchmarked results never share a column without labels.
  7. Date-stamp everything and say the review expires. Tool version, documentation access date, repository access date, and the date range of application papers.

Common questions

You can do a systematic review about a tool, provided you are precise about what the question is. What you cannot do is an intervention review, because there is no population receiving the tool, no comparator arm and no outcome to estimate an effect on. The workable questions are descriptive or methodological: what the tool does and how that has changed across versions; whether independent benchmarking exists; how the tool has been used in the published literature and whether that use was reported well enough to be reproducible; and whether sampled published analyses can actually be reproduced. The third of these benefits most from systematic methods, because it requires an unbiased sample of application papers and a fixed assessment item list — which is exactly what a systematic approach provides.

Because a web tool is a moving target in a way that a published method is not. Modules get added, the statistical method behind a button can be replaced, defaults change, and the underlying annotation and interaction databases are updated continuously. Two papers citing the same tool reference, published two years apart, may have used materially different software. Without a version and a date of access, a published result is not attributable to any defined piece of software, which means it cannot be reproduced even in principle and cannot be compared with another paper’s result. For the review, this has two consequences: the version record has to be built from the release history and documentation rather than from the papers, and the proportion of papers reporting a version is itself a finding worth quantifying.

It is good evidence of what the developers built and intended, and it is not independent evaluation. Tool papers and update papers are authored by the people who wrote the software, and the performance they report is performance on examples they selected, under settings they chose. That is normal practice and not a criticism; the error is a review that assembles developer-reported performance figures from several tool papers into a comparison table, because the resulting ranking compares selection choices rather than tools. Report developer-reported and independently benchmarked performance in clearly separate, labelled places, and where independent evaluation of a tool does not exist, state that plainly — it is a more useful finding than a synthesised table of self-reports.

Keep an application record with one row per paper and a fixed item list: tool name and version as stated, date of access, module or workflow used, input data type and repository accession, preprocessing and normalisation, the statistical method selected, effect-size and significance thresholds, multiple-testing correction, batch and confounding handling where datasets were combined, and the annotation or pathway database with its release version. Code each item as reported, reported ambiguously, or not reported, and treat the third state as data rather than as an empty cell. Extract what the paper says rather than what the tool’s defaults imply, note the location of each item, and have two people extract independently — the items that are most often ambiguous are also the ones where extractors most often disagree.

Use a transparent item list and report it item by item rather than inventing a scored instrument. The items that carry real consequences in expression analysis are normalisation and preprocessing, the statistical method and its thresholds, multiple-testing correction, and batch or confounding structure where datasets from different sources were combined — plus the two web-tool-specific items, version and date of access. A composite score would hide which of these failed, and the useful output of the exercise is precisely which items a field does and does not report. Where an item is not reported, that is a limitation of the paper’s reproducibility, and it should be recorded as such rather than converted into a judgement about the paper’s conclusions, which may well be correct.

You can, and it changes what the project is, so declare it. Re-analysing published data with the tool makes this a reproducibility study with a systematic sampling frame, not a literature review, and it needs its own design: which published analyses you attempt and why those, a pre-specified definition of what counts as reproduced — identical gene list, substantial overlap, same top-ranked features, same biological conclusion — and a stated protocol for handling the unreported parameters, which is usually the hard part. Record the version and date you ran, because your result is as version-dependent as theirs. Report failures with the reason each one could not be resolved, and be clear that a failure may reflect an unreported parameter rather than an error in either the paper or the tool.

By date-stamping rather than apologising. State the version examined, the dates on which the tool, its documentation and its repository were accessed, and the date range of the application papers, and say explicitly that the tool may have changed since. Publish the version record and the per-item reporting counts as supplementary tables, since those are the parts others will reuse. Where the tool is developing quickly, a living review format with a stated update cycle, or a state-of-the-art review that does not claim completeness at a fixed date, is more honest than a one-off systematic review presented as definitive. A reader who knows exactly what was true on a given date can work with that; a reader given an undated description cannot.

Rarely, and it should resist the temptation. Three things stand in the way. Independent head-to-head benchmarking is sparse in most tool areas, and where it exists the result depends heavily on the datasets chosen, so a single benchmark does not generalise. Developer-reported comparisons are not independent evidence. And documenting which parameters users have chosen records popularity, not correctness — a widely used default is consequential precisely because it is rarely examined. What a review can legitimately say is which comparisons exist and who ran them, which capabilities each tool offers at a stated version, what reporting a reproducible analysis would require, and what independent evaluation would have to be done for a recommendation to be possible.

References and sources

  1. Chapter 4: Searching for and selecting studies. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
  2. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane. 2024;6.5. read 9 October 2026. cochrane.org
  3. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Systematic Reviews. 2021;10(1):1-19. read 9 October 2026. doaj.org
  4. Jonnalagadda SR, Goyal P, Huffman MD. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews. 2015;4:78. read 9 October 2026. cochrane.org
  5. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Systematic Reviews. 2015;4(1):1. doi:10.1186/2046-4053-4-1 read 9 October 2026. doi.org/10.1186/2046-4053-4-1
  6. Tsafnat G, Glasziou P, Choong MK, Dunn A, Galgani F, Coiera E. Systematic review automation technologies. Systematic Reviews. 2014;3(1):74. doi:10.1186/2046-4053-3-74 read 9 October 2026. doi.org/10.1186/2046-4053-3-74
  7. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews. 2011;2011(8):MR000026. doi:10.1002/14651858.MR000026.pub2 read 9 October 2026. doi.org/10.1002/14651858.MR000026.pub2
  8. Bolanos F, Salatino A, Osborne F, Motta E. Artificial intelligence for literature reviews: opportunities and challenges. Artificial Intelligence Review. 2024;57(10):259. doi:10.1007/s10462-024-10902-3 read 9 October 2026. doi.org/10.1007/s10462-024-10902-3
  9. Karim MR, Beyan O, Zappa A, Costa IG, Rebholz-Schuhmann D, Cochez M, Decker S. Deep learning-based clustering approaches for bioinformatics. Briefings in Bioinformatics. 2021;22(1):393-415. doi:10.1093/bib/bbz170 read 9 October 2026. doi.org/10.1093/bib/bbz170
  10. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71 read 9 October 2026. doi.org/10.1136/bmj.n71
  11. Gauthier J, Vincent AT, Charette SJ, Derome N. A brief history of bioinformatics. Briefings in Bioinformatics. 2019;20(6):1981-1996. doi:10.1093/bib/bby063 read 9 October 2026. doi.org/10.1093/bib/bby063
  12. Bramer WM, de Jonge GB, Rethlefsen ML, Mast F, Kleijnen J. A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association. 2018;106(4):531-541. doi:10.5195/jmla.2018.283 read 9 October 2026. doi.org/10.5195/jmla.2018.283
  13. Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. Journal of Clinical Epidemiology. 2018;94:8-18. doi:10.1016/j.jclinepi.2017.10.017 read 9 October 2026. doi.org/10.1016/j.jclinepi.2017.10.017
  14. Yang Y, Gao J, Wang J, Heffernan R, Hanson J, Paliwal K, Zhou Y. Sixty-five years of the long march in protein secondary structure prediction: the final stretch? Briefings in Bioinformatics. 2018;19(3):482-494. doi:10.1093/bib/bbw129 read 9 October 2026. doi.org/10.1093/bib/bbw129
  15. Bramer WM, Giustini D, de Jonge GB, Holland L, Bekhuis T. De-duplication of database search results for systematic reviews in EndNote. Journal of the Medical Library Association. 2016;104(3):240-243. doi:10.3163/1536-5050.104.3.014 read 9 October 2026. doi.org/10.3163/1536-5050.104.3.014
  16. Angermueller C, Parnamaa T, Parts L, Stegle O. Deep learning for computational biology. Molecular Systems Biology. 2016;12(7):878. doi:10.15252/msb.20156651 read 9 October 2026. doi.org/10.15252/msb.20156651
  17. Jelicic Kadic A, Vucic K, Dosenovic S, Sapunar D, Puljak L. Extracting data from figures with software was faster, with higher interrater reliability than manual extraction. Journal of Clinical Epidemiology. 2016;74:119-123. read 9 October 2026. cochrane.org
  18. Libbrecht MW, Noble WS. Machine learning applications in genetics and genomics. Nature Reviews Genetics. 2015;16(6):321-332. doi:10.1038/nrg3920 read 9 October 2026. doi.org/10.1038/nrg3920
  19. Welch L, Lewitter F, Schwartz R, Brooksbank C, Radivojac P, Gaeta B, Schneider MV. Bioinformatics Curriculum Guidelines: Toward a Definition of Core Competencies. PLOS Computational Biology. 2014;10(3):e1003496. doi:10.1371/journal.pcbi.1003496 read 9 October 2026. doi.org/10.1371/journal.pcbi.1003496
  20. Gross A, Schirm S, Scholz M. Ycasd - a tool for capturing and scaling data from graphical representations. BMC Bioinformatics. 2014;15(1):219. doi:10.1186/1471-2105-15-219 read 9 October 2026. doi.org/10.1186/1471-2105-15-219
  21. Favrin B. esyN: Network Building, Sharing and Publishing. PLOS ONE. 2014;9(9):e106035. doi:10.1371/journal.pone.0106035 read 9 October 2026. doi.org/10.1371/journal.pone.0106035
  22. Langfelder P, Mischel PS, Horvath S, Ravasi T. When Is Hub Gene Selection Better than Standard Meta-Analysis? PLOS ONE. 2013;8(4):e61505. doi:10.1371/journal.pone.0061505 read 9 October 2026. doi.org/10.1371/journal.pone.0061505
  23. Shabalin AA. Matrix eQTL: ultra fast eQTL analysis via large matrix operations. Bioinformatics. 2012;28(10):1353-1358. doi:10.1093/bioinformatics/bts163 read 9 October 2026. doi.org/10.1093/bioinformatics/bts163
  24. Langfelder P, Luo R, Oldham MC, Horvath S. Is my network module preserved and reproducible? PLOS Computational Biology. 2011;7(1):e1001057. doi:10.1371/journal.pcbi.1001057 read 9 October 2026. doi.org/10.1371/journal.pcbi.1001057
  25. Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Information and Libraries Journal. 2009;26(2):91-108. doi:10.1111/j.1471-1842.2009.00848.x read 9 October 2026. doi.org/10.1111/j.1471-1842.2009.00848.x
  26. Koschutzki D, Schreiber F. Centrality Analysis Methods for Biological Networks and Their Application to Gene Regulatory Networks. Gene Regulation and Systems Biology. 2008;2:193-201. doi:10.4137/grsb.s702 read 9 October 2026. doi.org/10.4137/grsb.s702
  27. Benson DA, Karsch-Mizrachi I, Lipman DJ, Ostell J, Wheeler DL. GenBank. Nucleic Acids Research. 2008;36(Database issue):D25-D30. doi:10.1093/nar/gkm929 read 9 October 2026. doi.org/10.1093/nar/gkm929
  28. Langfelder P, Horvath S. WGCNA: an R package for weighted correlation network analysis. BMC Bioinformatics. 2008;9:559. doi:10.1186/1471-2105-9-559 read 9 October 2026. doi.org/10.1186/1471-2105-9-559
  29. Yip AM, Horvath S. Gene network interconnectedness and the generalized topological overlap measure. BMC Bioinformatics. 2007;8:22. doi:10.1186/1471-2105-8-22 read 9 October 2026. doi.org/10.1186/1471-2105-8-22
  30. Larranaga P, Calvo B, Santana R, Bielza C, Galdiano J, Inza I, et al. Machine learning in bioinformatics. Briefings in Bioinformatics. 2006;7(1):86-112. doi:10.1093/bib/bbk007 read 9 October 2026. doi.org/10.1093/bib/bbk007
  31. Zhang B, Horvath S. A general framework for weighted gene co-expression network analysis. Statistical Applications in Genetics and Molecular Biology. 2005;4:17. doi:10.2202/1544-6115.1128 read 9 October 2026. doi.org/10.2202/1544-6115.1128
  32. Ouzounis CA, Valencia A. Early bioinformatics: the birth of a discipline - a personal view. Bioinformatics. 2003;19(17):2176-2190. doi:10.1093/bioinformatics/btg309 read 9 October 2026. doi.org/10.1093/bioinformatics/btg309
  33. Ravasz E, Somera AL, Mongru DA, Oltvai ZN, Barabasi AL. Hierarchical organization of modularity in metabolic networks. Science. 2002;297(5586):1551-1555. doi:10.1126/science.1073374 read 9 October 2026. doi.org/10.1126/science.1073374
  34. Xia J, Benner MJ, Hancock REW. NetworkAnalyst--integrative approaches for protein-protein interaction network analysis and visual exploration. Nucleic Acids Research. 2014;42(Web Server issue):W167-W174. read 13 November 2026. View via DOI
  35. Xia J, Gill EE, Hancock REW. NetworkAnalyst for statistical, visual and network-based meta-analysis of gene expression data. Nature Protocols. 2015;10(6):823-844. read 13 November 2026. View via DOI
  36. Zhou G, Soufan O, Ewald J, Hancock REW, Basu N, Xia J. NetworkAnalyst 3.0: a visual analytics platform for comprehensive gene expression profiling and meta-analysis. Nucleic Acids Research. 2019;47(W1):W234-W241. read 13 November 2026. View via DOI

Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.

How to cite this article

Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.

Pubrica Academy. (2026, October 9). NetworkAnalyst and the Problem of Reviewing a Web Tool. Pubrica. https://pubrica.com/academy/medical-writing/a-systematic-review-of-network-analyst-a-web-based-bioinformatics-tool-for-integrative-visualization-of-expression-data/

This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.

Tool review protocol, version record and application record

Three blocks. Fill the first before searching, build the second from the repository and documentation rather than the papers, and complete the third once per application paper.

WEB TOOL REVIEW — PROTOCOL AND RECORDS

=== BLOCK 1: SCOPE (complete before the search) ===
  Tool: ________________________________________________
  WHICH OBJECT is under review? (one box)
    [ ] the METHOD it implements
    [ ] the IMPLEMENTATION (code, defaults, behaviour)
    [ ] the INTERFACE and its defaults as used in the literature
    [ ] a DEPLOYED INSTANCE at a stated version and date
  Use case under review: _______________________________
  CLAIM being examined:
    [ ] what it does and how it has changed
    [ ] whether it has been independently evaluated
    [ ] how it has been used and whether that use was reported adequately
    [ ] whether published analyses can be reproduced
        -> if ticked, this is a REPRODUCIBILITY STUDY. Label it.
  Evidence type that could answer the claim: ____________
  Does that evidence exist?  [ ] yes  [ ] partly  [ ] no
    If no or partly, rescope now: ______________________
  Design label for the manuscript:
    [ ] systematic review  [ ] scoping review
    [ ] state-of-the-art review  [ ] living review, update cycle: ____
    [ ] reproducibility study with systematic sampling

=== SEARCH SOURCES (record date for every one) ===
  Databases, with platform and date: ___________________
  Tool name variants and misspellings searched: ________
  Citation tracking on the tool papers: index ______ date ______
  Software repository / release history: URL ______ accessed ______
  Documentation and tutorials: accessed ______
  Data repositories referenced by application papers: __________

=== BLOCK 2: TOOL VERSION RECORD (one row per release/change) ===
  Version or date | Modules available | Methods per module |
  Defaults changed | Input types | What changed | Source of this info
  ________________|__________________|___________________|
  ________________|__________________|___________________|
  Public changelog exists?  [ ] yes  [ ] no — record as a finding
  Version examined in this review: ______  Accessed: ______

=== BLOCK 3: APPLICATION RECORD (one row per paper) ===
Code every item: R = reported, A = ambiguous, N = not reported

  Paper ID: ____________________________________________
  Tool version as stated: ______________________  [R/A/N]
  Date of access stated: _______________________  [R/A/N]
  Module / workflow used: ______________________  [R/A/N]
  Input data type: _____________________________  [R/A/N]
  Data accession / source: _____________________  [R/A/N]
  Preprocessing: _______________________________  [R/A/N]
  Normalisation method: ________________________  [R/A/N]
  Statistical method selected: _________________  [R/A/N]
  Design / covariates: _________________________  [R/A/N]
  Effect-size threshold: _______________________  [R/A/N]
  Significance threshold: ______________________  [R/A/N]
  Multiple-testing correction: _________________  [R/A/N]
  Batch / confounding handling: ________________  [R/A/N]
  Annotation or pathway database + VERSION: ____  [R/A/N]
  Interaction database + VERSION: ______________  [R/A/N]
  Departures from defaults listed by the paper?  [R/A/N]
  Code or scripts available?  [ ] yes  [ ] no
  Extractor: ______  Second extractor: ______  Disagreements: ____

=== PER-ITEM REPORTING COUNTS (the headline finding) ===
  Item | Reported | Ambiguous | Not reported | % of sample reporting
  _____|__________|___________|______________|______________________
  (publish this table)

=== PERFORMANCE EVIDENCE — KEEP THESE SEPARATE ===
  Developer-reported performance (tool and update papers):
    ___________________________________________________
  INDEPENDENT benchmarking, if any exists:
    ___________________________________________________
  If none exists, state so explicitly in the results: [ ] done

=== REPRODUCIBILITY COMPONENT (only if Block 1 ticked it) ===
  Sampling frame for analyses attempted: _______________
  Definition of "reproduced" (pre-specified): __________
  Protocol for unreported parameters: _________________
  Version and date YOU ran: ____________________________
  Attempts ____  Reproduced ____  Failed ____
  For each failure, reason it could not be resolved: ______

=== EXPIRY STATEMENT FOR THE DISCUSSION ===
  "Findings describe version ______ as accessed on ______.
   The tool, its defaults and the annotation databases it queries
   may have changed since that date."  [ ] included

Pubrica AcademyWritten and maintained by the subject matter experts of Pubrica who do this work. Articles are revised when the standard underneath them moves, and each one carries the date it was last checked.

How Pubrica helps

Pubrica scopes tool-focused reviews so that the question matches the evidence that exists, builds the version record from release histories and documentation rather than from citing papers, and runs the application-paper sample with a fixed reporting item list extracted independently by two people. Where the honest finding is that independent evaluation of a tool does not exist, we write that as the finding rather than assembling developer-reported figures into a comparison.

Find something else

Search the rest of the Academy.

Open with no account. Type what you are working on, or open the full library to filter by category, sub-topic or resource type.

Send us the tool and the question and we will tell you which review design the available evidence can support.

A scoping read costs nothing and takes two business days. You get a written view of what the work needs and who would hold it.

Get a free quote

Regulations

Check your project against this rule

  1. 1Your work
  2. 2Your details
  3. 3Send
What do you need?

Material you send is held under signed confidentiality, with access limited to the assigned team and encrypted transfer. It is not used to train anything and is not passed to third-party tools. Ask for the agreement before you send anything you would rather cover first.