How to develop and evaluate a PRO instrument?

What is PRO?    

PRO or Patient-reported outcome is a relatively novel method to evaluate the outcome of any treatment by clinician or researcher (in clinical trial) purely based on the inputs got from the patient alone.

The United States Food and Drug Administration (US-FDA) defines PRO as ‘any report of the patient’s health condition that comes directly from the patient, without interpretation of the patient’s response by a clinician or anyone else.’

Ask the doctor, why ask the patient?

Objective data regarding outcome can be given by the treating physician (cure/survival/disability limitation), by investigations (return of deranged biochemistry to normal) and by the caregiver (return to functional normality). These are Observer-reported outcomes (OROs) and medicine has long relied on these for evidence-based management.

Any illness followed by its treatment weighs physically, mentally, emotionally, economically on the patient. Patients perceive the outcome of the treatment in comparison to their expectations based on prior knowledge. Patient response and compliance to treatment are intricately linked to the ultimate treatment outcome.

Since long, clinicians have recorded patient symptoms in the daily clinical notes and modified their treatment accordingly. PRO goes beyond just recording symptoms. Central to this is the concept of Health-related Quality of Life (HRQOL), a multi-dimensional approach that records patient’s perceived effect of the illness and treatment on quality of life in his/her physical, mental, emotional, economical, and social space. By standardizing the questions and quantitating them as far as possible, data can be analysed, interpreted, and compared. PRO data is used to inform and guide patient-centered care, to assess risk and benefit from treatment, to decide on treatment protocols and to guide health policy makers.

What information do I want from the patient? – The PRO instrument

The outcome of interest that we want to study is the PRO Concept, while the PRO instrument is the means to collect the data related to the PRO concept, in the form of questionnaire and the information and documentation that supports its use.

In other words, PRO instrument is a questionnaire that compiles data related to the PRO concept of interest which can be summarized as a score.

            Content validity is the extent to which an instrument measures the concept of interest that is desired from both the developer’s and user’s perspective.

Designing a Valid PRO instrument

Steps involved are: –

1. Defining the concept – concept should be clearly defined based on review and literature gap analysis of existing PRO instruments as well as taking inputs from scientific research services, domain specialists, developers and most importantly the patients or the users and should focus on the core characteristics.
2. Conceptual Framework – Breakdown the concept to sub-concepts and finally to items or the questions that will be included in the questionnaire. Each item should be crisp and clear with focus on only one aspect. It should not unnecessarily cause strain to the patient in recalling.
3. End-point model– Endpoint is the final measured outcome of the treatment that will be compared, and an endpoint model summarises all the endpoint based on OROs and PROs.
4. Review of preliminary instrument –

Literacy review is done to understand the proportion of people who may or may not have the literacy level to take the questionnaire.

Translation review is done to ensure conceptual equivalence when translated to another language.

Cognitive interview is the process where the user is made to take the questionnaire by thinking aloud to understand the approach of the user to the questions and that it follows the intention of the item. This is done until saturation is reached on the information received for reviewing the instrument.

• Reliability Testing – Measure to estimate reproducibility and consistent true estimation of outcomes under consideration. It is further described below.
• Validation –

It is an ongoing never-ending process of improving the instrument by repeated feedback mechanism.

Evaluating PRO instrument 

            As clinical validation data accumulates, a PRO instrument undergoes review and revision to make them robust. It includes the evaluating for the following: –

1. Validity– is the extent to which the scores and their interpretations, supported by theoretical evidence, correlate with the proposed use of the instrument. It needs to be tested for the entire range of the proposed application. 

1. Content Validity- It is a measure of how much the instrument measures the concept of interest. US-FDA proposes using qualitative and quantitative methods.
2. Qualitative estimation – include items involved, their recall period, response options, scoring pattern, method of conducting the questionnaire etc.
3. Quantitative estimation – e.g.
4. Item response theory (IRT) – A powerful tool to assess whether the items used measure the entire spectrum or continuum of the patient responses.
5. Rasch analysis – Only IRT tool to use the total of all the items in the PRO instrument making it a simple model.
6. Construct Validity- It is based on comparison with convergent and divergent groups with known validity and showing that the instrument under study conforms to similar interpretations.
7. Criterion Validity- Comparison with known criteria or known gold standard for this concept.

• Reliability-
• Reproducibility or test-retest reliability- It is assessed under same conditions in same user in a close span of time to assess any variations in recording results that can be attributed to the instrument. It is calculated by intra-class correlation coefficient (ICC).
• Internal consistency- The items should deliver same scores in similar situations consistently. It is determined by Cronbach’s alpha or coefficient of reliability.
• Floor and ceiling effect – i.e. the instrument’s ability to measure extremes of construct helps choose the proper instrument for the study.
• Inter-interviewer reliability- there should not be significant difference in scores when interviewed by two different interviewers.
• Variability – Instrument should have ability to detect changes when outcomes are different, else it is no good. 

 Adhering to good practices while designing and validating a PRO instrument is primary for it to be effective and receive regulatory approvals. Readiness to continuously revisit and revise items is needed to do better justice to the concept of interest in the long run.

References:

1. FDA Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. 2009. Available at: http://www.fda.gov/downloads/Drugs/Guidances/UCM193282.pdf.
2. Patrick DL, Burke LB, Gwaltney CJ, et al. Content validity–establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1–eliciting concepts for a new PRO instrument. Value Health. 2011;14(8):967‐977. doi:10.1016/j.jval.2011.06.014.
3. Rothrock, N & Kaiser, Karen & Cella, David. (2011). Developing a Valid Patient-Reported Outcome Measure. Clinical pharmacology and therapeutics. 90. 737-42. 10.1038/clpt.2011.195.
4. Deshpande, P. R., Rajan, S., Sudeepthi, B. L., & Abdul Nazir, C. P. (2011). Patient-reported outcomes: A new era in clinical research. Perspectives in clinical research, 2(4), 137–144. https://doi.org/10.4103/2229-3485.86879