Bridging the gap from medical to psychological safety assessment: consensus study in a digital mental health context

Study design

An online Delphi technique was used to reach an experts’ consensus on how the medical methods used for assessing safety can be adapted to better meet the objectives of DMHIs. The Delphi technique is a tool used to reach a consensus, through controlled feedback on topics with limited evidence or where a consensus does not exist.^20,21 The main components of a Delphi technique are anonymity, iteration, controlled feedback and statistical stability.^13,21,22

Recruitment and participants

Purposive sampling was used to invite participants to participate in this Delphi study. Individuals were eligible to participate if they were professionals (researchers, clinicians or regulatory consultants) with experience/knowledge in the field of digital mental health in the UK. There were no exclusion criteria. Experts who met the eligibility criteria were contacted via email or LinkedIn. A snowball technique was also used to recruit participants. Thirty-six professionals were invited to join the panel, of whom 25 consented (69%) and 20 participated (55%). Thus, the sample size was in the average range for Delphi studies.²⁰ Of the 20 participants, 11 (55%) were females. The majority of the participants (17/20, 85%) were based in the UK, with three living in Canada, Australia and the Netherlands. Participants had an average of 19 years’ (range 5–37, s.d. = 7.45) general working experience and 8 years (range 2–15, s.d. = 4.51) in the field of digital mental health. See Table 1 for further details on the participants’ occupations. As common in expert Delphi studies, participating panel members were offered authorship on any resulting publications.²³

Data collection and analysis

Three rounds were conducted to reach a consensus in this study. Participants were given 2 weeks to complete each round.

Ethical clearance

This study was registered as a minimal risk study at King's College London under the reference number MRSP-22/23-34777. Under the above approval, consent was implied by participants’ email agreement to take part and completion of the various requested survey activities (Round 1–3). Furthermore, at the end of each online questionnaire, the following message was presented to participants: ‘Please be aware that submitting this form implies consent to participate in this research’.

Enhancing the quality of adverse events data in DMHIs

Under this theme, experts echoed researchers’ findings on the poor quality of adverse events data in DMHI studies.¹ A total of 94% of experts agreed that ‘The digital mental health field lacks clear guidance on what adverse events data should be collected, when and how this should happen, and how these should be categorised, analysed and reported’. All experts agreed that ‘Generating publication guidelines for reporting adverse events data that are required by journals can help ensure that adverse events data are collected and reported consistently’. Furthermore, under this theme experts concluded that ‘Determining the relatedness of the adverse event to the therapy is paramount for assessing safety’ (94%) and that ‘When assessing adverse events in digital mental health interventions, it is very important to differentiate between self-guided and clinician-guided therapies’ (89%).

Redefining SAEs for DMHIs

One expert said in round 1: ‘The current definition of serious adverse events (hospitalisation, death, congenital abnormalities … ) is not fit for purpose for mental health interventions compared to pharmaceutical, surgical etc.’. When this item was later shared with the panel of experts in round 3, 94% concurred with it. In all, 83% of experts agreed that ‘Congenital abnormalities as a serious adverse event of digital mental health interventions is not plausible’. Experts also expressed that the traditional definition of an SAE is not comprehensive of what should be considered an SAE for a DMHI. All experts agreed that ‘In the context of mental health therapies, it is important for “serious” to be inclusive of serious effects on mood, behaviour, general well-being and functioning, not just physical harms’. In addition to loss of life, hospitalisation and disability, suicide attempts (94%), increase in harm towards others (100%), increase in intentional self-harm (100%) and seizures that are induced because of on-screen flashes (100%) should be considered SAEs even if they do not lead to hospitalisation. Experts added that in the case of a DMHI, any harm that is sustained for longer than 6 months should be considered an SAE (89%).

Reassessing short-term deterioration in face-to-face and DMHIs as a therapeutic risk

Experts reached a consensus that ‘the short-term distress generated by focusing on the emotional experience (perhaps discussing painful or traumatic experiences or going into new or anxiety-provoking situations) is a part of the therapy process and should not be considered an adverse event or safety concern’ (89%). However, they also acknowledged that such a deterioration could lead to negative outcome/harm; thus, such deterioration needs to be monitored and not ignored (83%). For that, experts encouraged long-term follow-ups to differentiate between short-term and long-term deterioration (89%). They also suggested several ways to support individuals experiencing deterioration; eight of these reached consensus (see Supplementary Appendix C).

Maximising the benefit of the Yellow Card Scheme in DMHIs

All experts agreed that ‘There is not enough awareness among users that the Yellow Card Scheme is used for reporting adverse reactions in digital mental health therapies’. Furthermore, 83% of experts agreed with the item ‘I was not aware that the Yellow Card Scheme is used to report the adverse reactions of digital mental health therapies’. Experts acknowledged that one drawback of the Yellow Card Scheme for collecting adverse reactions is that it relies on individuals to report risks, which they may not be willing or able to do; for that, they suggested that Patient Reported Outcomes could be automatically collected within the digital therapy itself to help identify risks (94%).

Acknowledging the benefit of the Yellow Card Scheme, experts determined a few improvements that would make it more useful for DMHIs. First, experts noted that grouping all mental health adverse reactions under ‘psychiatric disorders’ risks losing useful and important detail on the different mental health adverse reactions experienced by individuals and may discourage reporting milder mental health adverse reactions (89% and 94% respectively). Instead, they suggested having a broader and more inclusive category such as ‘mental health adverse reactions’, with subtypes that relate to the different adverse reactions that could occur under that category (94% and 83% respectively). Experts settled on four proposed subtypes: intentional self-harm (100%), threat to others (100%), suicidal ideation/behaviours (94%) and new mental health symptoms (89%). Additionally, experts agreed that having the following information about every mental health adverse reaction reported on the Yellow Card website under the DMHI's profile would be helpful: severity/impact on the individual (94%), duration of adverse reaction (89%), whether the event led to the individual terminating therapy (89%) and a description of the adverse reaction (free text) (89%).

Developing a harmonised approach for assessing the safety of psychological interventions generally

Experts agreed that there should be a single harmonised approach for assessing and reporting the safety of face-to-face and DMHIs. Experts assented that such an approach would allow professionals to compare the safety of face-to-face and DMHIs (89%), and thus better understand the risks of mental health therapies, and contribute to stronger safety standards and safer interventions (94%). Moreover, experts speculated that ‘If there was one structured way to report adverse reactions in face-to-face and digital mental health therapies, clinicians and mental health professionals will be more likely to report them’ (83%). At the very least, the safety issues related to clinical support that are common to face-to-face and digital therapy such as the deterioration effect need to be harmonised, and the safety issues that arise from the use of a digital tool such as technical issues could be added to that (83%).

Items with high disagreement rates

We also explored the items that had high rates of disagreement. See Supplementary Appendix B for the percentage of disagreement (i.e. rating disagree/strongly disagree) for all items. No items reached consensus disagreement (80% or higher). However, two items equalled or exceeded 50%. A total of 61% of experts disagreed that ‘triggering trauma-related flashbacks’ should be considered an SAE. Half of the experts disagreed that ‘triggering a panic attack’ should be considered an SAE. Although these items arose directly at the suggestion of some experts, more than half the group disagreed. These disagreement rates illustrate the difficulty defining what constitutes an SAE for DMHIs and reveal that some areas divide opinion.

Improving adverse events data in DMHI trials

The Delphi study showed a consensus on the lack of clear guidelines on how to collect, analyse and report adverse events data in DMHIs. Using guidelines that were originally developed for pharmacological trials is adversely affecting the quality of adverse events data in DMHIs.^1,16 Consequently, experts unanimously agreed that generating publication guidelines to support professionals on what adverse events data need to be collected, how often and how they need to be analysed, will help standardise adverse events reporting in the field and allow for more comprehensive and systematic analysis of adverse events data (recommendation 1).¹³

To effectively assess the safety of DMHIs, determining the relatedness of an adverse event to the treatment is key (recommendation 2), which can be less straightforward in a DMHI compared with a pharmacological intervention.¹⁰ Additional data collection to support this judgement could include onset, previous similar instances, and participant's and clinician's opinions on relatedness. This could improve the robustness of relatedness decision-making.¹

When assessing or comparing the safety of DMHIs, it is important to differentiate between self-guided and clinician-guided DMHIs (recommendation 3). The concern is that self-guided interventions can pose more risk in the absence of a professional guiding the individual, explaining content to them, assessing risk and providing emotional support or encouragement to people when they need it.¹ However, a review of the state of adverse events data in DMHIs found that studies of clinician-guided interventions were more likely to address adverse events in their publication compared with self-guided interventions.¹ Adverse events in self-guided interventions are more likely to be missed because of the lack of human contact, and thus might be underreported. Thus, it is crucial for self-guided interventions to integrate within the product a way of collecting and monitoring adverse event and safety data, such as collecting pre/post data to detect deterioration or using automated triggers to alert clinicians to a worsening condition.²⁶ In such cases, individuals would need to be informed and provide consent for such data to be collected and shared with the clinical team.

Broadening the definition of SAE for DMHIs

Experts in this study agreed that the pharmacological frame of reference for SAEs is not ‘fit for purpose’ for DMHIs. They argued that the definition of an SAE in the digital mental health field is too narrow and needs to be inclusive of any potentially life-threatening effects on mood, behaviour, general well-being and functioning. They concurred that adverse events such as increase in harm towards others, increase in intentional self-harm and any harm that is sustained for longer than 6 months should be considered ‘serious’, regardless of whether it led to hospitalisation or not. An MHRA-led consultation on the proposed changes to the regulatory framework for medical devices in 2021 (891 respondents) found that professionals requested mental health impacts to be included under ‘serious’ adverse events.²⁷ The UK government's response was that at that time ‘appropriate mechanisms were not in place to sufficiently regulate the inclusion of these impacts’.²⁷ We therefore conclude that further consultation is needed to identify and operationalise how the existing SAE definition might best be broadened to capture serious impacts on mental health (recommendation 4).

Short-term deterioration is not a safety concern in mental health therapy

Experts in this study reached a consensus that short-term deterioration of psychological symptoms occurring in the course of a psychological treatment can be an integral part of therapeutic change and is therefore not inevitably a safety concern. Nevertheless, experts stressed that these symptoms could amount to an adverse reaction, for example if the person drops out of therapy as a result. Thus, professionals need to monitor such deterioration and support individuals through it (recommendation 5). Similar to other negative effects, individuals need to be informed of the possibility of experiencing deterioration during therapy, and such deterioration needs to be normalised as part of the process.²⁸ Additionally, guidelines for professionals and patients are needed on the management of short-term deterioration.²⁹ Our experts provided a list of suggestions (see Supplementary Appendix C) for supporting people experiencing therapy-related deterioration.

Adapting the Yellow Card Scheme for DMHIs

The medical field has already established that individuals are more likely than healthcare professionals to report adverse reactions that fall under the category ‘psychiatric conditions’.¹⁸ Experts in our study reached consensus on several ways to adapt the Yellow Card data collection and reporting process for DMHIs that are classified as medical devices, to maximise their benefit. First, there is a need for a campaign to raise awareness around the use of the Yellow Card Scheme for reporting the adverse reactions to DMHIs among both patients and professionals (recommendation 7). One suggestion is to make it a requirement for all DMHIs (including pre-market) to inform and direct people to the Yellow Card reporting website. Second, there was a consensus to use more appropriate, less stigmatising language by replacing ‘psychiatric disorders’ with ‘mental health adverse reactions’ (recommendation 8). Third, given that these are very general umbrella terms, it would be helpful to include further subcategories. Suggestions from our experts included ‘intentional self-harm’, ‘threat to others’, ‘suicidal ideation/behaviours’ and ‘new mental health symptoms’ (recommendation 9a). Fourth, it would be helpful to record the following contextual information for every reported adverse reaction: severity/impact on the individual, duration of adverse event, whether therapy was terminated as a consequence and a free text description of the adverse reaction (recommendation 9b).

Using a harmonised approach to assess the safety of psychological therapies generally

Perhaps one of the reasons why it has been difficult to adapt the medical model for DMHIs is that this model was not developed to assess the safety of face-to-face psychological therapies. The current quality of adverse event reporting in trials of face-to-face interventions is poor and is not subject to the same regulation as medical devices.¹² Face-to-face interventions would also benefit from a standardised safety assessment method. Currently there is insufficient information about how the negative effects or adverse events data of face-to-face and DMHIs compare.¹³ An ideal solution would be to develop one harmonised approach to assess the safety of face-to-face and DMHIs (recommendation 10). One harmonised approach could enhance the quality of safety assessments in traditional face-to-face interventions and ensure that safety evaluations of DMHIs are appropriate, proportionate and effective. Whether it is plausible to meet the needs of statutory regulators, clinicians, researchers, manufacturers and other stakeholders within a single overarching framework remains to be seen. Nevertheless, possible benefits our experts highlighted included better quality safety data, a more accessible method for non-medical mental health practitioners and better safety standards for both face-to-face and DMHIs.

Limitations

There are several limitations of this Delphi study. First, patients’ and their carers’ perspectives were absent in the results. Second, 65% of experts who participated work in higher education, leading to a likely overrepresentation of views from within this sector. It is important to note that 9 out of the 16 invitees (56.25%) who were invited but did not participate in the study came from non-academic backgrounds, primarily clinicians and co-founders. Third, evidence reliability is not possible in Delphi studies as the results might have differed if a different group of experts had been selected.