Content validity of the EQ-HWB and EQ-HWB-S in a sample of Italian patients, informal caregivers and members of the general public

Background

Economic evaluations are commonly used in the health technology assessment of treatments and interventions. Their main outcome measure is quality-adjusted life years (QALYs), which encompasses in a single metric survival (i.e., length of life) and health-related quality of life (HRQoL) [1]. QALYs are anchored to a 1 to 0 scale, where 1 corresponds to full health and 0 to death. There are different methods to put the “Q” into QALYs i.e., measuring and valuing HRQoL, and these include the use of vignettes (i.e., scenarios describing the impact of a condition), eliciting values directly from patient populations and using generic preference-based measures (GPBM) of health [2, 3].

There are numerous GPBMs in the literature, including the most commonly used EQ-5D [4, 5], Short-Form 6 dimensions (SF-6D) [6], and Health Utilities Index Mark 3 (HUI3) [7], the Quality of Wellbeing Self-Administered Scale (QWB-SA) and the Assessment of Quality of Life 8 dimensions [8, 9]. The validity and responsiveness of these GPBMs have been assessed in multiple health conditions, disease areas, cultural contexts, and numerous populations, showing in many cases results in support of the instruments [10]. Nevertheless, GPBMs were developed for measuring differences in health, and in that they may include dimensions and items that are too narrow for detecting important differences in other contexts where broader QoL aspects may matter [11]. For example, GPBMs have been criticized for not capturing aspects relevant to informal carers, social care users, and those with long-term conditions [12].

Other measures have been developed specifically targeting aspects relevant for other users. For example, the Caregiver Burden Interview was developed for caregivers, while the Adult Social Care Outcomes Toolkit (ASCOT) includes aspects important to social care users [11, 13, 14]. Yet, when conducting economic evaluations, using different measures in combination, such as a GPBM of health and a social care measure, may be difficult for different reasons. Using different outcome measures makes it difficult to make comparisons across sectors. Combining outcomes from different measures may not be possible but even where they can be combined, this may result in double counting of similar aspects covered by different instruments. A solution that has been proposed is to develop a single instrument that covers aspects important to users and beneficiaries of health and social care services.

The EuroQol Group recently developed two new instruments aimed at cross-sector comparisons, the EQ Health and Wellbeing (EQ-HWB) and EQ-HWB short version (EQ-HWB-S). The EQ-HWB (25 items) and EQ-HWB-S (9 items) cover items related to seven dimensions: activity, relationships, cognition, self-identity, autonomy, feelings and physical sensations. In doing this, they capture a range of health and broader quality of life (QoL) aspects, which may be relevant for use among members of the general public, patients, their families, social care users and informal carers. The EQ-HWB-S classifier is amenable to valuation, and therefore is intended for use in economic evaluation across health, social care and public health sectors. The measures were developed based on domains identified in a large qualitative review on the QoL aspects that patients, social care users and informal carers identified as important followed by item generation and both qualitative and quantitative testing of the selected items in these populations [15, 16]. Development and testing of items was undertaken in English (source version) and three other languages: Simplified Chinese, German and Argentinian Spanish. Both the EQ-HWB and EQ-HWB-S are experimental instruments, meaning that their descriptive system needs further testing and may still be modified.

Available guidelines such as the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) advocate for testing measures in all languages and populations for which instrument usage is intended [17]. One aspect of testing that is important for instruments is assessing the validity of the measure, that is, whether the instrument measures what is says it measures. Content validity is one type of validity testing that focuses on “the degree to which the content of an instrument is an adequate reflection of the construct to be measured” [18]. It includes assessment of face validity in any new language version, which addresses the extent to which an instrument appears to measure what it claims to measure, usually with non-experts (e.g., not instrument developers or clinicians). COSMIN guidance highlights the importance of assessing content validity as it gauges how well an instrument encompasses all relevant aspects of the construct it aims to measure, and it aids researchers in determining the measurement efficacy of research instruments. While the EQ-HWB and EQ-HWB-S development studies investigated the content validity of individual items to inform their selection in different languages and in mixed populations representing the target populations [19], there is limited evidence on the performance of the two newly developed instruments as a whole. Moreover, no study has investigated the content validity of the Italian versions of the instruments. The current study assesses the content validity of the Italian EQ-HWB and EQ-HWB-S in a sample of the Italian population composed of patients, informal carers, and healthy individuals.

Methods

The focus of this study was on the face validity of the measures, i.e., how comprehensive, comprehensible and relevant the items of the Italian version of the EQ-HWB and EQ-HWB-S are in a target population of potential users to ensure that the measures and the translation were valid. Qualitative interviews using cognitive debriefing were carried out between July and September 2020. Interviews aimed to explore potential users’ viewpoint on the items of the EQ-HWB and EQ-HWB-S and the overall questionnaires, and understand how different groups (i.e., patients, informal carers, general population) interpreted them. A translation from English (Appendix Table 6) to Italian was performed, based on international translation guidelines [19]. The translation included two parallel forward and backward translations, reconciliation of translated versions, approval from external independent reviewer and approval of final translated version from the study team.

Results

Face validity

Face validity results are available in Table 3. Broadly, participants found the EQ-HWB easy or quite easy to understand, and they considered the items included relevant and comprehensive. No important aspects of QoL was identified as missing and just one participant found the EQ-HWB too long. Some participants suggested clarifications in the wording of some items, which are reported below.

Table 3.

Response issues identified for the EQ-HWB self-administered experimental version, preceding 1.0 version

Open in a new tab

^**Relevance issues identified for the EQ-HWB-S self-administered experimental version, preceding 1.0 version. With regards to comprehension, recall, interpretation, response option selection and acceptability, these are considered as overlapping for the two questionnaires

Comprehension and interpretation

All participants stated that the questionnaires were “easy” or “quite easy” to complete. Although participants identified no odd wording (i.e., with unusual terms or sentences), items related to control (item 6), hopelessness (item 13) and feeling accepted by others (item 19) needed further explanation or an example to aid understanding.

Participants could generally easily interpret the EQ-HWB and EQ-HWB-S and their items, although sometimes (i.e., 15 times) they faced difficulties in the interpretation of the items. Items 4 (ability to do day-to-day activities), 6 (control over day-to-day life), 13 (hopelessness) and 19 (acceptance) were difficult for more than one participant, while items 16, 17, 18, 21, 24/25 were only difficult for one participant each.

For four participants (GPM1, GPF2, PF1, PM2), difficulties emerged from the fact that items could be equally interpreted as related to physical, or emotional aspects. This was related to items 4 (ability to do day-to-day activities), 6 (control over day-to-day life), 13 (hopelessness), 17 (sleep issues) and 18 (feeling exhausted). For example, in relation to items 4 (ability to do day-to-day activities), one participant (GPF2) argued that “within the questionnaire, I will think about physical difficulties. If you asked me this question outside of the questionnaire, my answer would have been related to concentration or time management”. For item 6 (control over day-to-day life), the same participant could not discriminate whether the question was related to physical or psychological aspects, even though this item had an example providing further information. In relation to the item 18 on feeling exhausted, one participant (GPM1) argued that “exhausted can mean being unmotivated, […] sad, […] or on the other hand […] being physically tired”. The participant argued this ambiguity could result in inconsistencies between participants.

Two participants (GPF2, PM2) found the item 6 on control over day-to-day activities difficult to interpret, as the word “control” is not frequently used in the Italian language in this context and may be understood as quite a harsh term to indicate the ability to manage or influence the events and circumstances in one’s life.

Three participants (PM2, CF5, CF6) found the item 13 on hopelessness difficult to interpret, in the Italian version (Italian translation: “Ho sentito di non avere aspettative”). This was because, as one participant (CF6) argued, “[it is unclear if we] are talking about distrust, pessimism […] or expectations from other people. Or is it simply not having goals, not having a purpose?”. A proposed solution was to add an example so that the item could be better defined.

As for item 19 on feeling accepted by others, two participants (GPM3, GPF4) from the general public noted that the item could have some difficulties in the interpretation because of different reasons. The first participant (GPM3) argued that it is difficult to understand the item from the perspective of a participant with no/few social contacts. The second participant (GPF4) argued that because the example in the item referenced “feeling like you are able to be yourself”, the item could also be interpreted as self-acceptance.

For one participant (PF1), the examples given in specific questions may have added to the difficulty in interpreting the questions. For example, for item 4 on the ability to do day-to-day activities, she considered that this item could be answered in different ways because “there are different types of work, housework is [for example] more [physically] strenuous [than other works]”. The same participant (PF1) reported that item 16 on feeling unsafe could be answered in different ways as it covered different issues, as “fear of falling” could be frequent in older adults, whereas “abuse or other physical harm” should never happen. Both these questions have examples or explanations which are aimed at aiding comprehension.

Additional information in items can also come from combining aspects in a single question, for example, sad and depressed. Two participants (PF8, GPF5) reported problems in relation to this double-barrelled item “sad” and “depressed”, as they argued that these are very different feelings, and that while sadness is familiar and normal to most, depression is a disease.

Acceptability: inappropriate/judgemental questions

Generally, the instruments were considered acceptable and appropriate. Four participants (PF1, PM2, PF4, PF8), who were all patients, reported issues with the acceptability of some items. Issues of acceptability were related to what may be considered extreme ends of the scale i.e. very bad or too good. Two patients (PF1, PM2) found the item 13 on hopelessness associated with stigma, as having “nothing to look forward to” was perceived as an emotionally burdensome topic for them to think about. For example, one participant (PM2) was interpreting it as “a moment of resignation, apathy” and asked if there could be a better way to formulate it, likely due to the profound emotions and strong impact it might have on respondents. On the other hand, one participant (PF4) felt that the item 20 on feeling good about yourself is inappropriate for patients as, in case of sickness, nobody feels good. Moreover, one participant (PF8) felt that the term “unable” in item 7 on inability to cope with day-to-day life was correct but a bit heavy to read for patients.

Relevance

No participants suggested additional domains for the EQ-HWB. Six participants found that there were close similarities between some items, which meant one or the other could be removed. More specifically, two participants (GPF2, GPF4) identified overlaps between item 4 on the ability to do day-to-day activities or item 6 on having control and item 7 on coping with day-to-day life.

Similarities between item 20 on acceptance by others and item 19 on feeling good about themselves were also noticed by a patient (PF1), because “in my opinion, if you feel good about yourself, you can also be in balance with the others”.

Irrelevant items were identified three times only, regarding item 11 (feeling frustrated), 13 (hopelessness) and 23 (severity of physical pain). Participants identified other items that were more relevant, namely item 12 on feeling sad/depressed (instead of items 11 and 13) and 22 (frequency of physical pain). In no cases more than one participant agreed on the irrelevance of an item.

Response options

Most participants felt that the response options were appropriate and easy to select. Five participants (GPM1, GPF2, PM2, GPM3, PM7) found difficult to distinguish between “only occasionally” and “sometimes” for the item control over day-to-day life and for the item feeling good about themselves. One participant (GPM3) felt that feeling accepted by others had too many response options, albeit no other participant reported problems regarding this item.

One participant (PF8) found that the term “unable” (translated in original language with the word “inabile”) was not appropriate, as it could be potentially considered judgmental in Italian. In fact, “unable” is often used as a derogatory term in common language, and used as a synonym of “incompetent, inadequate, sloppy”.

Recall period

The use of a 7-day recall period was generally considered appropriate by the current study participants. One participant (GPF4) stated that use of this recall period is easier for day-to-day and other types of activities than for emotions (i.e. such as the feeling of loneliness). In fact, this participant noticed that when the item does not begin with “I felt”, it seems a “more real and objective fact […], against a situation where a person has to remember the perception of that feeling of danger or loneliness”.

Ranking

Table 4 reports the result of the ranking exercise. Overall, the items considered most relevant were ability to do day-to-day activities, receiving support by people, control over own life, performing basic care needs, autonomy, independence, ability to get around indoors/outdoors) and ability to do activities.

Table 4.

Ranking of the most relevant questions for the different groups of participants: General public, patients, carers

Open in a new tab

* where nothing is written, no participants included the item in their top-5

Legend for the % of participants including the item in their top-5:

Note The order of the items reflects an early version of the questionnaire. Items in bold: included in the EQ-HWB-S. Questions in bold are EQ-HWB-S. EQ-HWB-S indicates short version of EQ Health and Wellbeing.^aEuroQol Research Foundation. EQ-HWB^TM is a trade mark of the EuroQol Research Foundation

There were differences in the support given to the items by different groups of participants. More specifically, items related to support, assistance, control over day-to-day activities and mobility were more frequently identified as relevant by patients and carers than by members of the general public. In contrast, loneliness and being accepted by others were ranked more frequently as important by members of the general public.

For carers, the most frequently endorsed items were day-to-day activities, coping with life and feeling unsupported by people. For patients, the most frequently endorsed items were day-to-day life, mobility, control over one’s life and feeling unsupported.

Discussion

This study evaluated the content validity of the Italian version of EQ-HWB and EQ-HWB-S by assessing their face validity within an Italian cohort, encompassing patients, social care users, and informal carers. The findings offer valuable insights into the usability of the instrument and highlight significant areas of concern that need attention for optimal utilization—some which relate to the translation and some to the face validity of items. Moreover, this study builds upon existing evidence supporting the efficacy of the instruments, as reported from various perspectives in previous research [15, 16, 25–33].

The study found evidence in support of the face validity of the Italian EQ-HWB and EQ-HWB-S. The instrument was relevant for different participants in a cohort of Italian subjects, including patients, carers and members of the general public. Both the EQ-HWB and EQ-HWB-S were easy to understand and to respond to. No participant identified missing aspects of health or QoL, showing the instrument is comprehensive. Minor issues were also identified, for example in the interpretation of the items or because of the presence of similar response options and items (each of these minor issues was identified more than 5 times by participants, whereas all other response issues were identified less than 5 times). The EQ-HWB-S version had items that were less likely to be endorsed as being problematic.

The way participants thought about health and QoL was along themes of physical and mental health, and absence of disease. Other themes were also related to broader constructs, such as social relations. These themes are similar to those identified in other studies, for example they are consistent with the evidence collected by Penton et al. [24], that also found that patients/members of the public interpreted health and QoL in terms of physical, mental health and other constructs such as social relations or ability to perform daily activities. They also closely reflect the WHO definition of health, which encompasses “complete physical, mental and social wellbeing” [34]. The relevance of these themes supports the core domains of the EQ-HWB and EQ-HWB-S and provide further suggestions for development and adaptability to a local context.

Some participants mentioned in their definition of health and QoL, that part of it is the possibility of accessing services that are needed. Such interpretation may be influenced by the current state of play in Italy, where accessibility to healthcare services (especially specialized care) is increasingly a problem, due to accessibility barriers, such long waiting times [35].

The ranking exercise identified the importance of the items on being able to do daily activities, autonomy and independence, and support from others. These results are aligned with previous studies [24, 36, 37], that also found physical functioning, autonomy and relationship with others as a notable aspect in the conceptualization of health and QoL. In addition, the items that were considered the most important by the sample are included in the short instrument, the EQ-HWB-S, apart from item 15 on support from others. In fact, this item was considered relevant by 44% (n = 4) of patients and 50% (n = 3) of carers in the ranking exercise, but was not included in the short questionnaire. Moreover, item 18 on feeling exhausted, that was not selected in the ranking exercise by the participants, was included in the EQ-HWB-S. This outcome confirms the selection of the most relevant items for the EQ-HWB-S in an Italian context, but also suggests some potential differences. Among carers and patients, themes like support and/or assistance, control over day-to-day activities and mobility were identified as relevant in the ranking exercise more frequently than the general public although the socio-demographic background and the relative familiarity of the sample with health and/or social care questionnaires might have also played a role. A recent study assessing different measures in the context of cancer found that the EQ-HWB-S was considered to be relevant but no generic measure covered all the concepts identified as important in patients with cancer [38]. The difference across the groups highlights the challenges of developing generic measures that are applicable across different populations including cultural contexts.

A relevant result that emerged was that some participants (i.e., 5) found it difficult to distinguish the response options “sometimes” and “only occasionally”. In Australian face validity study of older people (n = 24) using an English version, one person considered these options to be the same [39]. Face validation and psychometric assessment of the items in the source version and the three other languages in the development study did not highlight problems with individuals completing the items [16, 26]. The English version of the EQ-HWB-S has been valued in a feasibility study with evidence that individuals can distinguish between these two levels although the difference in disutility is not large e.g. −0.031 and −0.034 for “only occasionally” and “sometimes” being sad/depressed [40]. Consideration of alternative distinct response levels in the Italian and other language versions may be warranted which is important for valuation.

Participants provided suggestions related to the wording and the translation of the items. The translatability of concepts in a different language, Italian in this case, may indirectly have affected interpretation of some items. For example, item 13 on hopelessness posed challenges to be interpreted in the Italian version (Italian translation: “Ho sentito di non avere aspettative”). To address this, a suggested solution was to include an example to provide a clearer definition. Participants highlighted the significance of incorporating examples to guide responses and offer a more comprehensive context for interpretation, a point that was brought up several times during the discussions. Some other interpretation issues emerged, mainly in relation to topics that could have touched upon both physical and psychological aspects, leaving participants unable to correctly interpret the item by themselves. There is a potential risk that these issues impact responses and the validity of comparisons across participants, leading to biased scores and decision making. These results are aligned with those provided by the study conducted in Argentina [28], Australia [39] and USA [38] in terms of issues of interpretation of some items for some individuals. Some issues with the acceptability were identified by participants in the group of patients. It was suggested to soften them by replacing the term “unable” or have “nothing to look forward to”, as they are associated with stigma or they might not be accepted by some categories of participants. Identification and testing of a more appropriate translations and/or alternative wording may improve the instruments although this can be challenging for generic measures.

Some key limitations of this study need to be mentioned. First, as data collection was carried out by one single interviewer, deep-dives and more detailed understanding of some specific topics might have been influenced by the attitude of the interviewer during the interviews. To tackle this limitation, some actions have been employed such as the interviewer’s self-reflection, peer debriefing, and maintaining an audit trail, that the research team employed to mitigate the potential impact of biases on the study results. We used an existing framework to minimise bias but this may have framed our approach to the analysis. However, cognitive debriefing for face validation with focus on specified aspects of concern in instruments is a standardised approach that is recommended by COSMIN and can therefore help minimise bias. The inclusion of Italian and English speakers in the study, including one researcher involved in the development, also helped to minimise researcher bias. Second, as data were collected during the first wave of the COVID-19 pandemics (i.e., summer of 2020), this might have had an influence on how people felt about their quality of life and overall wellbeing, and the relative importance of some themes. The nature of videoconferencing as method of data collection could have also affected how the participants and the interviewer interacted. These last two limitations have been reported in similar studies conducted in the same time period [22]. Finally, although the study included the relevant target groups who may use the instruments (patients, informal carers, members of the public), it was not possible to consider all the different types of patients or informal carers where these generic instruments could be used due to resource and practical constraints. This includes for example, older individuals in receipt of social care support which is an important group in the Italian context due to the increasing average age of the Italian population. Exploring validity in these groups could be a valuable area for further research, involving a detailed examination of all the items.

Despite these limitations, the current study has also important strengths such as being the first study reporting evidence on content validity of the Italian version of the EQ-HWB and EQ-HWB-S in a sample of Italian patients, informal caregivers and members of the general public, and providing a preliminary proof of the cultural acceptability of these measures.

Conclusions

The Italian versions of EQ-HWB and EQ-HWB-S showed acceptable face validity in measuring health and wellbeing in an Italian cohort of patients, social care users and informal carers. Some issues with response options were identified, and they might potentially entail the risk of biasing scores obtained. These findings provide useful evidence that can be used to adapt the EQ-HWB and EQ-HWB-S to improve their validity.

Author contributions

First author: S.M; Last author: A.P.F; Middle authors: C.F, F.D.P, C.J, C.M.

Funding

Funding for the research was provided by the EuroQol Research Foundation. Views expressed in the paper are the authors’ and do not necessarily represent those of the EuroQol Research Foundation.

Data availability

Additional data will be available from Authors upon reasonable request.

Declarations