Contents
On this page
- Why this review was needed
- What the 52 included studies actually showed
- MARS and uMARS: the most widely used tool, in detail
- The cultural appropriateness and LMIC gap
- What evaluating against clinical outcomes, not just quality scores, looks like
- How regulators have tried to fill the same gap
- The market this evaluation gap is trying to govern
- Privacy is a separate, independently measured problem
- What the authors recommend
Academy · Research Data, Analytics & AI · Medical App Evaluation Review
Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review
A 2021 JMIR mHealth and uHealth rapid review of 52 mobile health app quality methodologies found that only 2 reported reliability data at all, and none assessed predictive validity — exposing how thin the evidence behind 'trusted' health apps really is.
App stores rate mobile health apps with five-star systems that tell a user almost nothing about clinical safety or data quality. Woulfe, Fadahunsi, Smith, Chirambo, Larsson, Henn, Mawkin and O'Donoghue set out to map what more rigorous evaluation actually looks like, publishing their rapid review in JMIR mHealth and uHealth in October 2021. Starting from 841 identified records and narrowing to 52 included studies, they catalogued the methodologies researchers actually use to judge app quality — and found the field's own tools are far less rigorous than the apps they are meant to be judging.
What you will learn
- Why the review treats five-star app store ratings as unreliable proxies for quality, and what motivated a closer look at formal evaluation methods.
- The concrete gap in validity and reliability testing across the 52 included studies — not a general impression, but counted findings.
- Where the widely used Mobile App Rating Scale (MARS) falls short, and why that matters for anyone selecting an evaluation tool.
What to unlearn
- “If a health app scores well on a recognised rating scale like MARS, its quality assessment is comprehensive.” The review found that MARS does not address security and privacy at all, and that only 2 of the 5 generic methodologies examined covered both of those dimensions — a high MARS score says nothing about whether the app protects user data.
- “Mobile health app evaluation tools are validated to the same standard as clinical outcome measures.” Across the 52 studies reviewed, only 2 reported any reliability data and none assessed predictive validity; few tools had undergone factor analysis, which the authors say limits confidence in what these scales are actually measuring.
Why this review was needed
Woulfe F, Fadahunsi KP, Smith S, Chirambo GB, Larsson E, Henn P, Mawkin M and O'Donoghue J published their rapid review in JMIR mHealth and uHealth, volume 9, issue 10, article e28384, on 12 October 20211. Their starting premise was simple: mobile health apps have grown rapidly in availability and use, but poor-quality apps expose both patients and clinicians to real risk, and the five-star ratings app stores display are subjective and often unreliable as a quality signal1.
From 841 records identified in the initial search, the team's rapid review process — a faster but methodologically narrower variant of a full systematic review — included 52 studies describing methodologies for assessing mHealth app quality1.
What the 52 included studies actually showed
The central finding is a methodological one: there is no comprehensive, universally available methodology to assess mHealth app quality, and definitions of "quality" and its sub-dimensions vary enough across studies to make assessments inherently subjective1. More concretely, only 2 of the 52 studies reported any reliability data, and none assessed predictive validity1 — meaning that for the great majority of evaluation tools in circulation, there is no evidence the tool measures the same thing consistently, let alone that it predicts a real-world outcome.
The authors also note that few of the tools examined had undergone factor analysis, a statistical step that checks whether a scale's items actually group together the way the scale assumes — without it, confidence in what these instruments are really capturing is limited1. No generic methodology included a scoring rubric for interpreting results, leaving even a completed assessment open to inconsistent interpretation1.
MARS and uMARS: the most widely used tool, in detail
The Mobile App Rating Scale (MARS) itself comes from a separate, earlier paper: Stoyanov, Hides, Kavanagh, Zelenko, Tjondronegoro and Mani published it in JMIR mHealth and uHealth in 2015 as a 23-item scale rated by trained reviewers across four objective quality subscales — engagement, functionality, aesthetics and information quality — plus a separate subjective quality section and an app-specific section2. A user-facing variant, uMARS, was developed and validated the following year by Stoyanov, Hides, Kavanagh and Wilson specifically so that end users, not just trained reviewers, could rate an app's quality using a simplified version of the same subscales3. Both versions deliberately omit a dedicated security or privacy subscale, which is exactly the gap Woulfe et al.'s 2021 review flags when it notes MARS does not address these dimensions at all1. That omission is not an oversight unique to MARS — it reflects what the broader field's most widely used instrument was actually built to measure: usability and engagement, not data governance.
The cultural appropriateness and LMIC gap
A specific and often overlooked gap the review surfaces is applicability outside high-income settings. The authors flag internet access, readability and cultural fit as live concerns for users in low- and middle-income countries (LMICs)1. No generic methodology they reviewed explicitly examined cultural appropriateness, and only one addressed offline functionality at all1 — a significant blind spot given how much of global mHealth deployment happens precisely where connectivity is least reliable.
The authors are also transparent about the limits of their own review: only English-language papers were included, a single reviewer carried out study selection and data extraction, and no formal risk-of-bias assessment was performed1 — caveats worth citing alongside the findings, not omitting.
What evaluating against clinical outcomes, not just quality scores, looks like
A smaller strand of the literature skips quality-scale methodology altogether and asks a more direct question: do specific app features actually move a clinical outcome. Tarricone, Petracca, Svae, Cucciniello and Ciani's 2024 systematic review and meta-analysis of randomised controlled trials in eBioMedicine asked which behaviour change techniques used in diabetes self-management apps were actually associated with improved glycaemic control, rather than asking whether the app was well-designed in the abstract7. That outcome-first approach is the kind of evidence Woulfe et al.'s review found almost none of the 52 generic quality methodologies attempt — none of them assessed predictive validity at all1. On the engagement side specifically, Six, Byrne, Tibbett and Pericot-Valverde's 2021 systematic review and meta-analysis in JMIR Mental Health examined whether gamification features in depression apps improve outcomes, finding genuine but modest effects that varied by which specific game mechanics were used9 — again, a finer-grained, outcome-linked kind of evidence than a single aggregate quality score can capture.
Separately, Larsen, Nicholas and Christensen's 2016 longitudinal tracking study in JMIR mHealth and uHealth followed mental health apps in major app stores over time and documented how volatile app-store rankings and ratings actually are — apps entered and dropped out of top-rated lists within months, independent of any underlying change in clinical quality10. That volatility is a direct, empirical illustration of why Woulfe et al. treat star ratings as an unreliable quality signal: a rating computed from recent user reviews can shift substantially with no change to the app's actual evidence base or safety profile.
How regulators have tried to fill the same gap
Woulfe et al.'s academic review is not the only attempt to impose structure on this space. In the United States, the FDA's guidance on device software functions and mobile medical applications narrows the agency's own oversight to apps that meet its definition of a medical device, explicitly stating that it does not intend to actively regulate most lower-risk wellness and general health apps even though many of them make health-related claims4. That leaves a large share of the apps Woulfe et al.'s review is concerned with entirely outside formal regulatory evaluation. In the United Kingdom, NICE took a different approach with its Evidence Standards Framework for Digital Health Technologies, which tiers digital health products by the level of functional risk and clinical claim they make and specifies the corresponding type and strength of evidence — from basic usability data at the lowest tier through to the equivalent of clinical-trial evidence for products making the strongest health claims5. Unlike MARS, the NICE framework is explicitly built around risk tiering and evidence strength rather than usability scoring, but it still depends on developers submitting evidence voluntarily or as a condition of NHS adoption — it does not reach the wider consumer app-store market Woulfe et al.'s review is describing.
The market this evaluation gap is trying to govern
The scale of the evaluation problem is worth putting in context. Industry tracking by the IQVIA Institute has reported the digital health app market passing 350,000 apps available across major app stores, with new apps launching at a rate on the order of hundreds per day in recent years6. Against that volume, a review finding that only 2 of 52 academic evaluation methodologies report any reliability data1 is not a minor methodological footnote — it means the tools available to bring any order to a market of that size are themselves largely unvalidated, and app-store star ratings remain, by default, the main signal most users and many clinicians actually see.
Privacy is a separate, independently measured problem
Woulfe et al. note that generic mHealth evaluation methodologies largely ignore security and privacy1; a separate empirical study shows why that gap matters in practice rather than only in principle. Grundy, Chiu, Held, Continella and Bero's 2019 analysis in The BMJ examined the data-sharing behaviour of medicines-related apps directly, using network traffic analysis, and found that the majority of the apps studied shared user data with third parties, frequently without adequately disclosing this to users in their stated privacy practices8. That finding sits directly behind the review's recommendation that evaluation tools formalise security and privacy assessment rather than leave it as an afterthought — without it, an app can score well on every engagement and functionality subscale a generic tool measures while still routinely sharing a user's health data without clear disclosure.
What the authors recommend
Woulfe et al. call for formalising reliability testing so assessments become more transparent and comparable across tools, for developing scoring mechanisms that summarise which quality dimensions an app actually meets, and for giving explicit, structured attention to cultural appropriateness and readability rather than treating them as afterthoughts1. They also point toward the need for more inclusive, internationally coordinated policy on app evaluation methodology, rather than leaving it to individual research groups to build one-off scales1.
Take-home message
Take-home message
The evaluation tools researchers use to judge mobile health app quality are themselves under-validated — only 2 of 52 reviewed studies reported reliability data, so a confident-looking app rating may rest on an unvalidated scale.
How to apply this to your work
How to apply this to your own work
- Check whether your chosen app-rating tool reports reliability data before relying on it. The review found this is true for only 2 of 52 methodologies; absence of reliability reporting is the norm, not the exception, so verify it rather than assume it.
- Don't treat a high MARS score as covering security and privacy. MARS does not address these dimensions at all according to the review; pair it with a tool that does if data protection is part of what you need to assess.
- Build cultural appropriateness and offline functionality into your own evaluation criteria. The review found almost no generic methodology covers these explicitly, so a researcher evaluating an app for LMIC deployment needs to add these criteria rather than expect an off-the-shelf scale to include them.
Common questions
No. The 2021 review by Woulfe et al. found MARS does not address security and privacy, and that only 2 of the 5 generic methodologies examined covered both dimensions — a strong MARS score tells you about engagement and functionality, not data protection.
A rapid review uses a faster, more streamlined process than a full systematic review — in this case, English-only inclusion, single-reviewer selection and extraction, and no formal risk-of-bias assessment — trade-offs the authors disclose explicitly as limitations of their own work.
References and sources
- Woulfe F, Fadahunsi KP, Smith S, Chirambo GB, Larsson E, Henn P, Mawkin M, O'Donoghue J. Identification and Evaluation of Methodologies to Assess the Quality of Mobile Health Apps: Rapid Review. JMIR mHealth and uHealth. 2021;9(10):e28384. read 10 October 2026. View source
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile App Rating Scale (MARS): A New Tool for Assessing the Quality of Health Mobile Apps. JMIR mHealth and uHealth. 2015;3(1):e27. read 10 October 2026. View source
- Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR mHealth and uHealth. 2016;4(2):e72. read 10 October 2026. View source
- U.S. Food and Drug Administration. Policy for Device Software Functions and Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff. read 10 October 2026. View source
- National Institute for Health and Care Excellence (NICE). Evidence Standards Framework for Digital Health Technologies. read 10 October 2026. View source
- IQVIA Institute for Human Data Science. Digital Health Trends: market growth in digital health applications. read 10 October 2026. View source
- Tarricone R, Petracca F, Svae L, Cucciniello M, Ciani O. Which behaviour change techniques work best for diabetes self-management mobile apps? Results from a systematic review and meta-analysis of randomised controlled trials. eBioMedicine. 2024;103:105091. read 10 October 2026. View on PubMed Central
- Grundy Q, Chiu K, Held F, Continella A, Bero L. Data sharing practices of medicines related apps and the mobile ecosystem: traffic, content, and network analysis. BMJ. 2019;364:l920. read 10 October 2026. View source
- Six SG, Byrne KA, Tibbett TP, Pericot-Valverde I. Examining the Effectiveness of Gamification in Mental Health Apps for Depression: Systematic Review and Meta-analysis. JMIR Mental Health. 2021;8(11):e32199. read 10 October 2026. View on PubMed Central
- Larsen ME, Nicholas J, Christensen H. Quantifying App Store Dynamics: Longitudinal Tracking of Mental Health Apps. JMIR mHealth and uHealth. 2016;4(3):e96. read 10 October 2026. View source
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 10). Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review. Pubrica. https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/
Pubrica Academy. “Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review.” Pubrica, 10 Oct. 2026, pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/.
Pubrica Academy (2026) Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review. Pubrica. Available at: https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/ (Accessed: 15 October 2026).
Pubrica Academy. “Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review.” Pubrica. October 10, 2026. https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/
Pubrica Academy, ‘Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review’ (Pubrica, 10 October 2026) <https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/> accessed 15 October 2026.
Pubrica Academy. Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review. [Internet]. Pubrica; 2026 Oct 10 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/
Pubrica Academy, “Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review,” Pubrica, Oct. 10, 2026. [Online]. Available: https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/
@misc{/la_medical_app_evaluation_motivations_challenges_methodology_review_2026,
author = {{Pubrica Academy}},
title = {Motivations, Challenges, Recommendations, and Methodological Aspects of Medical App Evaluation: A Systematic Review},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/systematic-review/motivations-challenges-recommendations-and-methodological-aspects-of-medical-app-evaluation-a-systematic-review/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Mobile health app evaluation sanity checklist
Use this before trusting any app quality score in a research or clinical context.
[ ] Confirmed the tool reports reliability data [ ] Checked whether security/privacy is covered separately from engagement scores (MARS/uMARS do not cover this) [ ] Checked whether the app falls inside FDA or NICE evaluation scope, or outside formal regulation entirely [ ] Added cultural appropriateness and readability criteria if evaluating for LMIC use [ ] Checked for offline functionality assessment if relevant [ ] Looked for outcome-linked evidence (e.g. a behaviour-change-technique trial) rather than relying on a quality score alone [ ] Noted the evaluation tool's own validation limitations in your write-up
Pubrica AcademyWritten and maintained by the subject matter experts of Pubrica who do this work. Articles are revised when the standard underneath them moves, and each one carries the date it was last checked.
How Pubrica helps
Pubrica's systematic review team maps and critically appraises the methodological quality of health technology evaluation tools, so your review doesn't inherit an unvalidated instrument's blind spots.