Table 5.
Overview of evaluation frameworks used to assess the effectiveness and quality of language models (LLMs) in healthcare
| Specialized framework | Description | Application in LLM evaluation | Key evaluation criteria used by the specialized framework |
|---|---|---|---|
| SERVQUAL94 | A tool for evaluating service quality across different domains, including healthcare. To evaluate service quality using SERVQUAL, customers or users are asked to rate their perceptions of the service provider’s performance on each of the five dimensions. | Evaluates the service quality of the ChatGPT conversational agent in providing medical information to kidney cancer patients55. | Reliability, responsiveness, assurance, empathy, tangibles |
| Patient Education Materials Assessment Tool-Printable (PEMAT-P)95 | A tool to assess the understandability and actionability of patient education materials. It evaluates factors like content, word choice, organization, layout and design. | Evaluates the understandability and actionability of patient education materials generated by LLMs78,96. | Understandability, actionability |
| Structure of the Observed Learning Outcome (SOLO) taxonomy97 | A framework to assess the quality of learner responses based on the Structure of the Observed Learning Outcome (SOLO) taxonomy, which ranges from pre-structural to extended abstract levels. | Evaluates the complexity and depth of LLM responses to medical queries, ranging from basic factual information to more advanced, integrated understanding98. | Pre-structural, unistructural, multistructural, relational, extended abstract |
| Wang and strong99 | A framework to assess data quality from the perspective of data consumers. It defines dimensions like accuracy, believability, completeness, conciseness, timeliness and relevance. | Assesses the overall data quality of the medical information generated by LLMs, from the perspective of healthcare professionals and patients100. | Accuracy, believability, completeness, conciseness, timeliness, relevancy |
| METRICS101 | A checklist to standardize the design and reporting of AI-based studies in healthcare, covering aspects like study objectives, data, model development, evaluation, and transparency. | Ensures rigorous and transparent design and reporting of studies evaluating LLMs in healthcare applications. | Study objectives, data, model development, evaluation, transparency |
| CLEAR102 | A tool to assess the quality of health information generated by AI models, evaluating factors like accuracy, reliability, readability, and potential harms | Comprehensively assesses the quality and potential harms of medical information generated by LLMs. | Accuracy, reliability, readability, potential harms |
| DISCERN instrument103 | A questionnaire to judge the quality of written consumer health information on treatment choices, assessing factors like reliability, information on treatment options, and overall rating. Published in 1998. | Evaluates the quality of patient-facing medical information generated by LLMs, from the perspective of healthcare consumers78,104–107. | Reliability, information on treatment options, overall rating |
| AHRQ’s harm scale108 | A scale developed by the Agency for Healthcare Research and Quality to measure harm in healthcare settings. Applicable for evaluating potential harms from LLM-generated medical information. | Assesses the potential for patient harm arising from the use of LLM-generated medical information. | Severity of harm, preventability of harm |