Contents
On this page
- Justification for developing a new instrument
- Primacy of content validity and its derivation from patients
- Distinction between cognitive interviewing and pilot testing
- The COSMIN structure and the requirements of its standards
- Reliability and the limitations of Cronbach’s alpha
- Construct validity and pre-specified hypotheses
- Responsiveness and its frequent omission
- Time requirements of instrument development
- Reporting for subsequent use
- Key points
- Application to your study
- Frequently asked questions
Academy · Research Writing · Developing a PRO instrument
How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards
The first question in patient-reported outcome (PRO) instrument development is not how to develop an instrument but whether a validated instrument for the construct already exists, and for most research questions one does. Where none exists, development is a multi-year programme that begins with qualitative work on content validity, because the FDA guidance states explicitly that later psychometric testing cannot remedy an instrument that measures the wrong construct.
Instrument development is among the most frequently underestimated tasks in clinical research. It is commonly included in protocols as one objective among several, with a sample size chosen for the main analysis and a few months allocated to “validation”. The result is typically a questionnaire with a reliability coefficient and no evidence of content validity, which does not constitute a validated instrument. This article describes the order in which the work must be undertaken, the requirements of each stage and the reporting needed for others to use the instrument.
Learning objectives
- The reasons the first step is a documented search for an existing instrument rather than a draft item pool
- What content validity requires, and the reasons the FDA guidance requires it to be established first
- How cognitive interviewing differs from a pilot test, and its purpose
- The COSMIN structure — 3 domains, 7 measurement properties, 4 aspects — and the 14 + 14 standards for development and content-validity studies
- The reasons Cronbach’s alpha constitutes the weakest evidence in a validation paper, and the measures that should be reported instead
- What responsiveness means, how it differs from reliability, and why it is the property most frequently omitted
- The preparation of a validation paper that reports limitations as well as findings
Common misconceptions
- “No existing instrument is suitable for the study, so a new one must be developed.” Churruca and colleagues, in Health Expectations, screened 6,453 citations and identified 315 PROMs, only 39 of them generic. The usual problem is an undocumented search rather than an absence of instruments. The search should be conducted systematically and documented.
- “Validation consists of testing reliability and validity after the questionnaire has been written.” The FDA guidance reverses this order: content validity is established first, because no subsequent testing can remedy an instrument that measures the wrong construct. Four groups of measurement properties are reviewed thereafter, in addition to content validity rather than in place of it.
- “An expert panel can establish content validity.” Experts can establish relevance to clinical practice. Content validity for a patient-reported outcome requires evidence from patients in the target population; Patrick and colleagues, in Value in Health, describe concept elicitation and assessment of respondent understanding as two separate studies, both conducted with patients.
- “A Cronbach’s alpha above 0.7 indicates that the scale is sound.” Sijtsma, in Psychometrika, argues that alpha has very limited usefulness even as an estimate of reliability; Cortina, in the Journal of Applied Psychology, had previously shown that it is not evidence of unidimensionality. A high alpha can also be obtained by phrasing the same item in multiple ways.
- “A validation study can be a secondary objective of a trial.” Development and validation have their own design requirements, including qualitative sampling to saturation, a separate test-retest interval and hypotheses specified before construct validity is tested. When appended to another study, they usually satisfy none of these requirements.
Justification for developing a new instrument
Most researchers who intend to develop a PRO instrument should not do so. This reflects not their ability but the size of the existing pool of instruments and the cost of the work. Churruca and colleagues, reviewing the PROM literature in Health Expectations, screened 6,453 citations and identified 315 instruments meeting their inclusion criteria, of which 39 were generic and the rest condition-specific 3. For most constructs in most well-studied conditions, a suitable instrument is likely to exist.
The decision must therefore be documented rather than asserted. The statement that no suitable instrument was available is a claim a reviewer can verify quickly, and if a well-known measure of the construct is subsequently identified, the remainder of the paper is undermined. A defensible justification defines the construct in one sentence, names the search strategy and databases, identifies any systematic review of measurement properties for that construct, lists the candidate instruments and states the reason for rejecting each with reference to a property rather than a general impression: an inappropriate population, absence of content-validity evidence, a recall period incompatible with the visit schedule or an unobtainable licence.
Before concluding that development is necessary, authors should consider an intermediate approach. Rothman and colleagues, in Value in Health, set out the ISPOR task force position on using and modifying existing PRO instruments. Modifying a validated instrument requires re-establishing the properties that the modification could have affected, but this is usually a considerably smaller programme than developing an instrument from the beginning 22. Our companion page on PRO instrument types and how to choose one describes the selection process in full.
Primacy of content validity and its derivation from patients
The FDA guidance on patient-reported outcome measures makes the sequencing explicit: content validity is established first, because later psychometric testing cannot repair an instrument that does not measure the intended concept 2. This is among the most frequently disregarded requirements in the field. A questionnaire developed from the literature and a clinical advisory board and then tested for reliability has no content-validity evidence; it has only evidence of agreement among clinicians.
Patrick and colleagues split the work into two papers in Value in Health, and this division provides an appropriate conceptual model 14. Part one is concept elicitation: open qualitative work with patients in the target population to establish, in their own words, what the experience comprises before any item is drafted. Part two is assessing respondent understanding: returning draft items to patients and establishing that they are interpreted as intended. Brod and colleagues, in Quality of Life Research, and Leidy and Vernon, in Pharmacoeconomics, both treat the qualitative stage as the methodological foundation rather than a preliminary 21,26.
The choice of method is important. Vogt and colleagues, in Psychological Assessment, recommend focus groups with members of the target population specifically as a means of strengthening content validity, and Magasi and colleagues, reporting a PROMIS meeting in Quality of Life Research, set out the expectations of the field 11,29. Whichever method is used, it must be reported to the standard of a qualitative study rather than summarised in a single sentence, including sampling, saturation, the identity of the coders and the resolution of disagreements. Kuper and colleagues’ introduction to appraising qualitative research in BMJ provides a reasonable guide to the criteria readers will apply.
The evidence indicates that this stage is frequently omitted. Chiarotto and colleagues, in the Journal of Clinical Epidemiology, reviewed patient-reported instruments for physical functioning in low back pain, a large and well-funded field, and concluded that their content validity required investigation because it had largely not been examined 5.
Distinction between cognitive interviewing and pilot testing
A pilot test assesses whether the questionnaire can be administered: its completion time, completion rates and the functioning of the electronic form. A cognitive interview addresses a different and more demanding question, namely whether the respondent interpreted the item as intended 14. It is conducted with one participant at a time, with the respondent thinking aloud or being probed after each item, and its output is a list of items that failed and the reason for each failure.
The failures identified are usually specific and practical, which is why the method justifies the effort. Examples include a recall period that respondents cannot retain, an item addressing two concepts at once so that a single response cannot be interpreted, a response scale whose middle category means “sometimes” to some respondents and “unsure” to others, a clinical rather than everyday term, and a negatively worded item that a substantial proportion of the sample answers in the wrong direction. None of these problems is detected by a reliability coefficient, and several of them may inflate one.
Patrick and colleagues’ second paper addresses this stage explicitly, and Olshansky and colleagues, in the International Journal of Education and Psychological Assessment, make the same point regarding rating-scale construction: construct and content work must be undertaken on the items themselves, with respondents, before the scale is treated as a measure 12. Several rounds should be anticipated, with items revised between them, and the number of rounds and the changes made should be reported.
The COSMIN structure and the requirements of its standards
COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) provides the shared terminology for this work, and its correct use makes a validation paper considerably easier to appraise. Mokkink and colleagues, in the Journal of Clinical Epidemiology, reported the international Delphi consensus that produced the taxonomy: four written rounds, 43 participating experts and a 67% consensus threshold, yielding a structure of 3 domains, 7 measurement properties and 4 aspects. The three domains are reliability, validity and responsiveness 18. The properties are located within these domains: internal consistency, reliability and measurement error under reliability; content validity, construct validity and criterion validity under validity; and responsiveness in its own domain. The four aspects are located beneath validity: face validity, structural validity, hypotheses testing and cross-cultural validity 18. Interpretability and feasibility are recorded but are not measurement properties.
The content-validity methodology was developed separately and in greater detail. Terwee and colleagues, in Quality of Life Research, reported a Delphi study with 159 panellists from 21 countries over four rounds at the same 67% threshold, producing 14 standards for PROM development, 14 standards for content-validity studies, and a final set of 10 rating criteria 6. The two sets address different questions: the development standards assess whether the instrument was developed appropriately, and the content-validity standards assess whether a study of its content validity was conducted appropriately 6. A new instrument requires both to be addressed.
Alongside these, Mokkink and colleagues’ risk-of-bias checklist, also in Quality of Life Research, is the instrument that reviewers and subsequent systematic reviewers will apply to the study 7. Consulting it before designing the study is considerably less costly than consulting it afterwards.
Sources: Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology (taxonomy, Delphi rounds, 43 participating experts, 67% threshold). Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Quality of Life Research (159 panellists from 21 countries; 14 development standards, 14 content-validity standards, 10 rating criteria).
Reliability and the limitations of Cronbach’s alpha
Internal consistency is the most easily reported property and the least informative. Cronbach’s alpha depends on the number of items and the average inter-item correlation, and therefore increases when near-duplicate items are added, which is contrary to the aims of good instrument design. Cortina set this out in the Journal of Applied Psychology, and Sijtsma extended the argument in Psychometrika, arguing that alpha is of very limited usefulness even as a lower bound on reliability 20,32. An alpha of 0.95 on a ten-item scale is more likely to indicate redundancy than quality.
Internal consistency also addresses a different question from reliability itself. Reliability in the COSMIN sense concerns stability, that is, whether the instrument returns the same score when nothing relevant has changed. This requires test-retest data with a stated interval, chosen so that the construct should not have changed but recall of previous answers has diminished, and it is reported as an intraclass correlation with the model stated rather than as a Pearson correlation.
The third property in this domain is the most frequently omitted and the most clinically useful: measurement error, expressed as the smallest change in score that can be distinguished from measurement noise. Terwee and colleagues’ quality criteria in the Journal of Clinical Epidemiology and de Vet and colleagues’ Measurement in medicine both regard this as essential, because without it no user of the instrument can determine whether an observed change is meaningful 15,27. If only one property is reported in addition to internal consistency, it should be this one.
Construct validity and pre-specified hypotheses
Construct validity is not a coefficient but an argument supported by predictions that could have failed. The procedure is to state hypotheses before the analysis (for example, that a subscale should correlate strongly with an established measure of the same construct, weakly with a measure of a different construct, and should differ between two clinically distinct groups in a specified direction and approximate magnitude), then to test them and report which were supported.
When hypotheses are specified in this way, a mixed result is informative. When results are presented instead as a correlation matrix with the significant cells discussed, the analysis is unfalsifiable and reviewers will regard it as such. Messick’s argument in American Psychologist provides the theoretical background: validity is a unitary judgement of whether the inferences drawn from scores are warranted for a given purpose, not a set of criteria that can be satisfied individually 33. Aaronson and colleagues, in Quality of Life Research, and Lohr and colleagues, in Clinical Therapeutics, both frame instrument review as an appraisal of that overall argument 30,31.
Structural validity, that is, whether the data support the proposed scoring structure, also belongs in this category and is the property that most frequently contradicts an instrument’s intended subscales. If subscale scores are to be reported, the factor structure must support them; if it does not, the appropriate response is to change the scoring rather than the interpretation.
Responsiveness and its frequent omission
Responsiveness is the ability of an instrument to detect change over time in the construct it measures 18. It constitutes a separate domain in the COSMIN taxonomy, distinct from reliability and validity, and it is the property most frequently absent from validation papers, because it requires longitudinal data with an intervention or natural change, which a cross-sectional validation study cannot provide.
This is important because responsiveness is precisely the property a trialist requires. An instrument with excellent discriminative properties may be unable to detect the change produced by an intervention, and a trial using it may report a null result attributable to the instrument rather than the treatment. If the instrument is intended as an outcome measure, responsiveness is essential, and a validation paper that omits it should state this in the limitations rather than leave the reader to identify the omission.
Interpretability is closely related. A score change of four points is uninformative to a clinician without an anchor, such as a minimal important change estimate or reference values from a relevant population. It is not a measurement property in the COSMIN structure, but it determines whether the instrument can be used to inform decisions. Reeve and colleagues’ ISOQOL (International Society for Quality of Life Research) minimum standards, in Quality of Life Research, treat interpretability as part of the minimum requirements for an instrument used in comparative effectiveness research 10.
The requirements of each property, and the claims that cannot be made by a study omitting it
| Domain | Property or aspect | Required elements of the study design | If omitted, you cannot claim |
|---|---|---|---|
| Validity | Content validity | Qualitative concept elicitation with patients in the target population, to saturation | That the instrument measures the intended construct |
| Face validity (aspect) | Patient and expert judgement that items appear relevant | Nothing on its own; it is the weakest form of evidence | |
| Structural validity (aspect) | A field-test sample large enough to examine the factor structure | That the proposed subscale scores are legitimate | |
| Hypotheses testing (aspect) | Comparator measures and predictions stated before analysis | That scores behave as the construct implies | |
| Cross-cultural validity (aspect) | Data from each language or cultural group, with a documented translation process | That scores are comparable across the groups studied | |
| Reliability | Internal consistency | A single administration, interpreted alongside structural validity | Very little; it is not evidence of unidimensionality |
| Reliability (test-retest) | Repeat administration at a stated, justified interval in stable patients | That a score difference reflects true change rather than noise | |
| Measurement error | The same test-retest data, analysed for smallest detectable change | That any particular observed change is distinguishable from error | |
| Responsiveness | Responsiveness | Longitudinal data spanning true change, with an anchor | That the instrument is usable as an outcome measure in a trial |
Property structure follows the COSMIN taxonomy (Mokkink and colleagues, Journal of Clinical Epidemiology); design requirements and the consequences of omission are drawn from the COSMIN risk-of-bias checklist and content-validity methodology (Quality of Life Research), Terwee and colleagues' quality criteria (Journal of Clinical Epidemiology), and de Vet and colleagues, Measurement in medicine.
Time requirements of instrument development
The scale of the work should be estimated realistically, because unrealistic plans lead validation studies to report only the most easily assessed properties. Cella and colleagues, in the Journal of Clinical Epidemiology, reported that PROMIS developed and tested its first wave of adult self-reported health outcome item banks over the period 2005 to 2008, in a multi-year, multi-site, funded programme for a single wave of instruments 19. A doctoral project cannot be expected to match this, which is precisely the argument against undertaking one.
A minimally defensible programme for a single instrument involves a documented search establishing that no suitable instrument exists; qualitative concept elicitation with patients to saturation; item drafting; two or more rounds of cognitive interviews with revision between them; a cross-sectional field test large enough to assess structural validity; a test-retest subsample at a defined interval; pre-specified construct-validity hypotheses tested against established measures; and a longitudinal sample for responsiveness. The qualitative stages alone are likely to take longer than most protocols allow.
If such a programme is not feasible, the appropriate options are to use an existing instrument, to modify one and re-establish the affected properties, or to publish the development work as such, in a concept-elicitation and item-generation paper whose conclusions extend only as far as the evidence. The last of these is a genuine contribution; a full validation claim based on an alpha coefficient is not.
Reporting for subsequent use
A validation paper has a distinctive purpose: the reader is deciding whether to adopt the instrument, so the gaps are as important as the findings. A paper that identifies what it did not establish is more likely to be used than one that implies completeness, because the reader can identify the risks rather than discover them later.
In practice, this involves reporting by property using COSMIN terminology, so that a subsequent systematic review can map the paper onto its appraisal; stating the sample and setting for each property separately, since they usually differ; making the instrument and its scoring algorithm available, including the handling of missing items; and writing a limitations paragraph that lists the properties not examined. Where the study is a systematic review of existing instruments rather than a primary validation, the reporting guideline requirements differ: PRISMA 2020, described by Page and colleagues in BMJ, applies to the review, and the COSMIN guideline for systematic reviews of outcome measurement instruments governs the appraisal within it 4,8. Where reporting is the principal difficulty, medical writing support can assist; it cannot, however, supply a property that was not measured.
The central claim of a validation paper, presented in two ways
Claims more than it shows
The questionnaire was developed by the research team based on a review of the literature and input from an expert panel of five clinicians. It was administered to 120 patients. Cronbach’s alpha was 0.91, indicating excellent internal consistency, and the instrument correlated significantly with an established quality-of-life measure (r = 0.62, P < 0.001). The questionnaire is therefore a valid and reliable measure of symptom burden in this population.
Content validity is absent. A literature review and five clinicians establish clinical relevance, not that the instrument measures what patients experience, which the FDA guidance requires to be established first.
Alpha is used for a purpose it cannot serve. A value of 0.91 on an unexamined structure is as likely to indicate redundant items as a sound scale, and it provides no information on stability over time.
A single correlation does not establish construct validity. Without a hypothesis stated in advance, r = 0.62 is compatible with the instrument measuring the comparator, a related construct or an entirely different construct.
“Valid and reliable” names no property. Measurement error, test-retest reliability, structural validity and responsiveness are all unaddressed, and the sentence encourages the reader to assume otherwise.
Nobody can use it. Without a smallest detectable change, a clinician cannot determine whether an observed score difference is meaningful.
Specified by property
Items were generated from concept-elicitation interviews with 24 patients in the target population, continued until no new concepts emerged, and coded independently by two researchers. Draft items were revised across two rounds of cognitive interviews (n = 9 and n = 8); four items were dropped for double-barrelled wording and the recall period was shortened from four weeks to one after respondents reported being unable to hold the longer window in mind. In a field test of 120 patients, confirmatory factor analysis supported the proposed two-factor structure. Internal consistency was 0.91, which we report alongside the structural analysis rather than as evidence of unidimensionality. Construct validity was assessed against three pre-specified hypotheses, of which two were supported; the hypothesised difference between treatment-naive and treated groups was not. Test-retest reliability over 14 days in 41 stable patients gave an intraclass correlation of 0.84 (two-way mixed, absolute agreement) with a smallest detectable change of 7.2 points. Responsiveness and cross-cultural validity have not been examined, so the instrument should not yet be used as a trial endpoint.
The qualitative stage is reported as a qualitative study, with sample, saturation and independent coding.
Cognitive interviewing is distinguished from piloting, with the rounds, sample sizes and specific changes reported.
Internal consistency is reported but explicitly not over-interpreted, and is linked to the structural analysis.
Construct validity rests on pre-specified hypotheses, and the unsupported hypothesis is reported, which gives meaning to the two that were supported.
Measurement error is quantified, so that subsequent users can interpret an observed change.
The properties not examined are named, and the recommended use of the instrument is restricted accordingly.
The second version reports a less complete instrument but is considerably more likely to be adopted, because the reader can see exactly which inferences it supports. It should also be noted that several elements could not have been added retrospectively: the concept-elicitation interviews, the cognitive-interview rounds and the test-retest interval were all design decisions. A paragraph of this kind must therefore be planned at the protocol stage.
Key points
Key points
Development should be a last resort rather than the default. Churruca and colleagues identified 315 PROMs from 6,453 screened citations, which indicates that the usual reason no instrument appears suitable is an undocumented search. Where development is necessary, the FDA guidance determines the order: content validity derived from patients first, because no later testing can remedy an instrument that measures the wrong construct, followed by the remaining properties. Authors should use COSMIN terminology (3 domains, 7 measurement properties and 4 aspects, supported by 14 development standards and 14 content-validity standards), report measurement error rather than relying on Cronbach’s alpha, and name the properties that were not examined. A validation paper that states its limitations is more likely to be adopted; one that claims completeness is more likely to be scrutinised.
Application to your study
Application to your study
- Document the search that justifies developing a new instrument. The documentation should define the construct in one sentence, name the databases, list the candidate instruments and relate each rejection to a measurement property rather than a general impression.
- Consider modification of an existing instrument before creating a new one. Adapting a validated instrument and re-establishing the affected properties usually requires a fraction of the work, and Rothman and colleagues set out the obligations this entails.
- Include the qualitative work in the protocol with its own sample and timeline. Concept elicitation to saturation constitutes a study in its own right rather than a preliminary step.
- Conduct cognitive interviews in rounds and record the changes made between them. Double-barrelled items, unworkable recall periods and ambiguous middle categories are identified only at this stage.
- Specify construct-validity hypotheses before any analysis, including direction and approximate magnitude, and report those that are not supported.
- Report measurement error in addition to alpha. The smallest detectable change is the value that subsequent users require.
- Plan a longitudinal sample if the instrument is intended as an outcome measure. Without evidence of responsiveness, a trial using the instrument may report a null result attributable to the instrument rather than the intervention.
- Structure the limitations paragraph as a list of properties. Each COSMIN property not examined should be named, and the recommended use of the instrument restricted accordingly.
- Publish partial work explicitly as partial work. A concept-elicitation and item-generation paper constitutes a contribution; a validation claim based on a single coefficient does not.
Frequently asked questions
Almost certainly not. Churruca and colleagues, in Health Expectations, screened 6,453 citations and identified 315 PROMs, so for most constructs in most studied conditions a suitable instrument already exists, and the usual reason a researcher concludes otherwise is an informal search. The search should be conducted formally: the construct should be defined in one sentence, a published systematic review of its measurement properties should be sought, and the candidate instruments should be listed with the reason each is unsuitable recorded against a named property. If a genuine gap remains, modification of an existing instrument with re-establishment of the affected properties should be considered before development begins. Full development is a multi-year programme; PROMIS took from 2005 to 2008 for its first wave of adult item banks.
Content validity is evidence that the instrument’s items cover the construct as experienced by the people reporting it, and nothing beyond it. The FDA guidance on patient-reported outcome measures requires it to be established before other testing, on the explicit grounds that later psychometric work cannot remedy an instrument measuring the wrong construct; a scale can be highly reliable and internally consistent while measuring a construct adjacent to the research question. For a patient-reported outcome, the evidence must come from patients in the target population rather than from a clinical advisory board. Patrick and colleagues, in Value in Health, divide it into concept elicitation and assessment of respondent understanding, and both stages involve patients.
No. An expert panel establishes that clinicians consider the items clinically relevant, which is useful but does not constitute content validity for a patient-reported outcome. The construct is the patient’s experience, so the evidence must come from patients: open qualitative interviews to establish what the experience comprises, followed by cognitive interviews to confirm that draft items are interpreted as intended. Vogt and colleagues, in Psychological Assessment, recommend focus groups with the target population specifically on these grounds, and Magasi and colleagues set out the expectations of the field in Quality of Life Research. An expert panel is a reasonable supplement to work with patients but not a substitute for it.
There is no single figure, because each property requires a different sample. Concept elicitation continues to saturation rather than to a target number, so the sample size is an outcome of the work rather than an input. Cognitive interviewing uses small rounds with revision between them. Structural validity requires a field-test sample large enough for the proposed factor model. Test-retest reliability requires a separate subsample of patients who are stable over the chosen interval. Responsiveness requires longitudinal data spanning true change. A single figure quoted for “the validation study” usually indicates that one of these was planned and the others were not.
Because it addresses a narrow question and is easily inflated. Alpha depends on the number of items and the average inter-item correlation, so adding near-duplicate items increases it, which is contrary to good design. Cortina showed in the Journal of Applied Psychology that it is not evidence of unidimensionality, and Sijtsma argued in Psychometrika that it is of very limited usefulness even as a lower bound on reliability. It also provides no information on stability over time, which is what reliability means in the COSMIN sense and which requires test-retest data. Alpha should be reported alongside structural validity, together with measurement error (the smallest detectable change), because that is the figure users of the instrument will require.
Reliability is stability when nothing has changed; responsiveness is sensitivity when change has occurred. They are separate domains in the COSMIN taxonomy, and an instrument may possess one without the other. A questionnaire may return nearly identical scores on repeat administration in stable patients, indicating good reliability, yet fail to change when a treatment produces genuine clinical improvement, indicating poor responsiveness. The practical consequence is serious: a trial using a non-responsive instrument may report a null result attributable to the measure rather than the intervention. Responsiveness requires longitudinal data with an anchor for true change, which is why cross-sectional validation studies cannot provide it.
Rarely with satisfactory results. Development and validation have design requirements that a trial does not meet: qualitative sampling to saturation before any item is drafted, cognitive-interview rounds with revision between them, a test-retest interval chosen for stability rather than to fit the visit schedule, and construct-validity hypotheses specified in advance. When development is appended to a trial, the usual result is an internal-consistency coefficient and a correlation with another measure, which does not constitute validation. Where a PRO endpoint is required for a trial, an existing instrument should be used. Where development is intended, it requires its own protocol, and the qualitative stages should be expected to take longer than the quantitative stages.
As a list of properties, using COSMIN terminology, in the limitations section. Authors should write “responsiveness and cross-cultural validity were not examined” rather than “further validation is needed”, and should restrict their recommendation accordingly, for example by stating that the instrument is suitable for cross-sectional description but not yet as a trial endpoint. This is in the authors’ interest. A reader deciding whether to adopt the instrument needs to see where the risks lie, and a subsequent systematic review applying the COSMIN risk-of-bias checklist will identify the gaps regardless. A paper that names them appears rigorous; one that implies completeness appears overstated and is likely to be rated less favourably.
References
- International Committee of Medical Journal Editors (ICMJE). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals. 2026. read 9 October 2026. icmje.org
- US Department of Health and Human Services, Food and Drug Administration (CDER, CBER, CDRH). Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. 2009. read 9 October 2026. hhs.gov
- Churruca K, Pomare C, Ellis LA, Long JC, Henderson SB, Murphy LED, Leahy CJ, Braithwaite J. Patient-reported outcome measures (PROMs): a review of generic and condition-specific measures and a discussion of trends and issues. Health Expectations. 2021;24(4):1015-1024. doi:10.1111/hex.13254 read 9 October 2026. doi.org/10.1111/hex.13254
- Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021:n71. doi:10.1136/bmj.n71 read 9 October 2026. doi.org/10.1136/bmj.n71
- Chiarotto A, Ostelo RW, Boers M, Terwee CB. A systematic review highlights the need to investigate the content validity of patient-reported instruments for physical functioning in low back pain. Journal of Clinical Epidemiology. 2018;95:73-93. read 9 October 2026. link.springer.com
- Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Quality of Life Research. 2018;27:1159-1170. doi:10.1007/s11136-018-1829-0 read 9 October 2026. doi.org/10.1007/s11136-018-1829-0
- Mokkink LB, de Vet HCW, Prinsen CAC, et al. COSMIN risk of bias checklist for assessing the methodological quality of studies on the measurement properties of patient-reported outcome measures. Quality of Life Research. 2018. read 9 October 2026. link.springer.com
- Prinsen CA, Mokkink LB, Bouter LM, et al. COSMIN guideline for systematic reviews of outcome measurement instruments. Quality of Life Research. 2017. read 9 October 2026. link.springer.com
- Prinsen CA, Vohra S, Rose MR, et al. How to select outcome measurement instruments for outcomes included in a 'Core Outcome Set': a practical guideline. Trials. 2016;17(1):449. read 9 October 2026. link.springer.com
- Reeve BB, Wyrwich KW, Wu AW, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research. Quality of Life Research. 2013;22(8):1889-1905. read 9 October 2026. link.springer.com
- Magasi S, Ryan G, Revicki D, et al. Content validity of patient-reported outcome measures: perspectives from a PROMIS meeting. Quality of Life Research. 2012;21(5):739-746. read 9 October 2026. link.springer.com
- Olshansky E, Lakes KD, Vaughan J, et al. Enhancing the construct and content validity of rating scales for clinical research. The International Journal of Education and Psychological Assessment. 2012;10(1):36-50. read 9 October 2026. link.springer.com
- Terwee CB, Mokkink LB, Knol DL, et al. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist. Quality of Life Research. 2012;21(4):651-657. read 9 October 2026. link.springer.com
- Patrick DL, Burke LB, Gwaltney CJ, et al. Content validity: establishing and reporting the evidence in newly developed PRO instruments for medical product evaluation. Part 1: eliciting concepts for a new PRO instrument. Value in Health. 2011;14(8):967-977. read 9 October 2026. link.springer.com
- de Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in medicine. Cambridge University Press. 2011. read 9 October 2026. link.springer.com
- Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality of Life Research. 2010;19(4):539-549. read 9 October 2026. link.springer.com
- Mokkink LB, Terwee CB, Knol DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: a clarification of its content. BMC Medical Research Methodology. 2010;10:22. read 9 October 2026. link.springer.com
- Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology. 2010;63(7):737-745. doi:10.1016/j.jclinepi.2010.02.006 read 9 October 2026. doi.org/10.1016/j.jclinepi.2010.02.006
- Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, et al. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. Journal of Clinical Epidemiology. 2010;63(11):1179-1194. doi:10.1016/j.jclinepi.2010.04.011 read 9 October 2026. doi.org/10.1016/j.jclinepi.2010.04.011
- Sijtsma K. On the use, the misuse, and the very limited usefulness of Cronbach's alpha. Psychometrika. 2009;74(1):107-120. read 9 October 2026. link.springer.com
- Brod M, Tesler LE, Christensen TL. Qualitative research and content validity: developing best practices based on science and experience. Quality of Life Research. 2009;18(9):1263-1278. read 9 October 2026. link.springer.com
- Rothman M, Burke L, Erickson P, et al. Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR good research practices for evaluating and documenting content validity for the use of existing instruments and their modification PRO task force report. Value in Health. 2009;12(8):1075-1083. read 9 October 2026. link.springer.com
- Kuper A, Reeves S, Levinson W. An introduction to reading and appraising qualitative research. BMJ. 2008;337:a288. read 9 October 2026. link.springer.com
- Valderas JM, Ferrer M, Mendivil J, et al. Development of EMPRO: a tool for the standardized assessment of patient-reported outcome measures. Value in Health. 2008;11(4):700-708. read 9 October 2026. link.springer.com
- Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. Oxford University Press. 2008. read 9 October 2026. link.springer.com
- Leidy NK, Vernon M. Perspectives on patient-reported outcomes: content validity and qualitative research in a changing clinical trial environment. Pharmacoeconomics. 2008;26(5):363-370. read 9 October 2026. link.springer.com
- Terwee CB, Bot SD, de Boer MR, et al. Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology. 2007;60(1):34-42. read 9 October 2026. link.springer.com
- von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806-808. doi:10.1136/bmj.39335.541782.AD read 9 October 2026. doi.org/10.1136/bmj.39335.541782.AD
- Vogt DS, King DW, King LA. Focus groups in psychological assessment: enhancing content validity by consulting members of the target population. Psychological Assessment. 2004;16(3):231-243. read 9 October 2026. link.springer.com
- Aaronson N, Alonso J, Burnam A, et al. Assessing health status and quality-of-life instruments: attributes and review criteria. Quality of Life Research. 2002;11(3):193-205. read 9 October 2026. link.springer.com
- Lohr KN, Aaronson NK, Alonso J, et al. Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clinical Therapeutics. 1996;18(5):979-992. read 9 October 2026. link.springer.com
- Cortina JM. What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology. 1993;78:98-104. read 9 October 2026. link.springer.com
- Messick S. Test validity and the ethics of assessment. American Psychologist. 1980;35:1012-1027. read 9 October 2026. link.springer.com
Every source above is a peer-reviewed article, a publisher’s record of one, or a standard issued by the body that owns it. Each carries the date it was last read. Where a standard moves, this article is revised rather than left up with a note.
How to cite this article
Pick your referencing style and copy it. Your university or target journal may vary the punctuation — check its guide if it has one.
Pubrica Academy. (2026, October 9). How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards. Pubrica. https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/
Pubrica Academy. “How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards.” Pubrica, 9 Oct. 2026, pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/.
Pubrica Academy (2026) How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards. Pubrica. Available at: https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/ (Accessed: 15 October 2026).
Pubrica Academy. “How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards.” Pubrica. October 9, 2026. https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/
Pubrica Academy, ‘How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards’ (Pubrica, 9 October 2026) <https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/> accessed 15 October 2026.
Pubrica Academy. How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards. [Internet]. Pubrica; 2026 Oct 9 [cited 2026 Oct 15]. Available from: https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/
Pubrica Academy, “How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards,” Pubrica, Oct. 9, 2026. [Online]. Available: https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/
@misc{how_to_develop_and_evaluate_a_pro_inst_2026,
author = {{Pubrica Academy}},
title = {How to Develop and Evaluate a Patient-Reported Outcome (PRO) Instrument: Justification, Content Validity and COSMIN Standards},
howpublished = {Pubrica},
year = {2026},
month = {oct},
url = {https://pubrica.com/academy/medical-writing/how-to-develop-and-evaluate-a-pro-instrument/},
note = {Accessed: <span data-cite-acc="iso">2026-10-15</span>}
}This article is written to be learned from. Cite it if you are referring to what it says; if you are using the evidence behind it, cite the sources listed above instead.
Development and validation plan, by property
This plan should be completed before the protocol is finalised. Any property that cannot be planned here will need to be reported as a limitation, which is acceptable provided this is recognised at the planning stage rather than at submission.
PRO INSTRUMENT DEVELOPMENT AND VALIDATION PLAN
Construct: _________________________ Population: _________________________
Intended use (describe / discriminate / measure change / trial endpoint): _______
STAGE 0 - JUSTIFY DEVELOPING ANYTHING
[ ] Construct defined in one sentence
[ ] Databases searched and search terms recorded: ____________________
[ ] Published measurement-properties systematic review found? Y / N
[ ] Candidate instruments listed, with the property-based reason for
rejecting each:
____________________ reason: ____________________
____________________ reason: ____________________
[ ] Modification of an existing instrument considered and ruled out, with
reason: ____________________________________________
STAGE 1 - CONTENT VALIDITY (must precede everything below)
Concept elicitation
Sampling approach: ____________________ Target: to saturation
Interview or focus group: ____________ Who codes, and how many: ______
[ ] Analysis method stated and reportable as a qualitative study
Item generation
[ ] Each item traced to an elicited concept (keep the mapping table)
[ ] Recall period chosen and justified: ____________
[ ] Response options chosen and justified: ____________
Cognitive interviewing
Round 1 n: ____ Round 2 n: ____ Round 3 n: ____
[ ] Items revised between rounds, with changes logged
[ ] Checked for: double-barrelled items / unworkable recall /
ambiguous middle category / clinical jargon / negative wording
STAGE 2 - FIELD TEST, BY PROPERTY
Structural validity
Sample: ______ Model to be tested: ______________________
[ ] Scoring structure will be changed if the data do not support it
Internal consistency
[ ] Reported alongside structural validity, not as evidence of it
Reliability (test-retest)
Interval: ______ days, justified by: ______________________
Subsample: ______ Statistic: ICC, model: ______________________
Measurement error
[ ] Smallest detectable change will be calculated and reported
Construct validity - HYPOTHESES, WRITTEN BEFORE ANALYSIS
H1: ______________________________________ expected direction/size: ____
H2: ______________________________________ expected direction/size: ____
H3: ______________________________________ expected direction/size: ____
[ ] All hypotheses will be reported, including those not supported
Cross-cultural validity
Languages / groups: ______________________
[ ] Translation process documented
STAGE 3 - RESPONSIVENESS AND INTERPRETABILITY
[ ] Longitudinal sample available, spanning real change
Anchor for change: ______________________
[ ] Minimal important change to be estimated? Y / N Method: __________
[ ] Reference or norm values available or planned? Y / N
STAGE 4 - WHAT I WILL NOT BE ABLE TO CLAIM
Properties NOT examined in this programme:
______________________________________________________________
Limitation sentence, drafted now:
______________________________________________________________
Recommended use, scoped to the evidence:
______________________________________________________________
STAGE 5 - MAKING IT USABLE BY OTHERS
[ ] Instrument reproduced in full, or access route stated
[ ] Scoring algorithm published, including missing-item handling
[ ] Each property reported with its own sample and setting
[ ] COSMIN terminology used throughout Pubrica AcademyWritten and maintained by Pubrica’s subject matter experts. Articles are revised when the underlying standards change, and each article states the date on which it was last reviewed.
How Pubrica can help
Pubrica’s medical writing and biostatistics teams support both aspects of this decision: documenting the search that establishes whether development is necessary and, where it is, preparing a plan for each measurement property so that the qualitative stages are funded and scheduled. For completed work, Pubrica prepares the validation paper in COSMIN terms with its limitations stated, which facilitates adoption of the instrument.