. 2024 Sep 28;7:258. doi: 10.1038/s41746-024-01258-7

Table 1.

QUEST: five principles and corresponding dimensions for the human evaluation of LLMs in healthcare

Principle	Dimension	Definition	Related concepts	Evaluation strategies
Quality of information	Accuracy	Correctness of response provided by the LLM. The response should be factually correct, precise, and free of errors.	• Accuracy^46,50 • Correctness³⁹ • Error^10,61 • Omission¹⁰	(1) Comparison with gold labels provided by human evaluators with metrics such as accuracy, F1, specificity, sensitivity, etc. (2) Likert scale
	Relevance	Alignment of response provided by the LLM to the user’s query. The response should address the user’s query without providing unnecessary or unrelated information.	• Relevance^46,62 • Appropriateness⁶³	Likert scale
	Currency	Timeliness of response provided by the LLM. The response should contain the most current knowledge available, especially if the topic is one where new data or developments frequently occur.	• Currency⁶⁴ • Up-to-dateness⁶¹	Categories (Presence or absence)
	Comprehensiveness	Completeness of response provided by the LLM. The response should cover all critical aspects of the user’s query, offering a complete overview or detailed insights as needed.	• Comprehensiveness^32,33,65 • Completeness⁶¹ • Exhaustiveness⁶⁶ • Complexity^67–69 • Additional Information⁷⁰	Likert scale
	Consistency	Stability and uniformity of responses across similar queries. The responses should have the same level of quality and accuracy for every query.	• Consistency⁷¹ • Reproducibility^72,73 • Inconsistent between trials⁶³	(1) Comparison with different prompts: (1) prompts with the same input in different sections (2) prompts with the same input over a longer period of time (3) prompts with similar semantic meaning (2) Likert scale
	Agreement	Coherence of response with established facts and theories. The response should be coherent and not contradict itself.	• Acceptance⁵ • Alignment⁷⁴ • Following guidelines⁷⁵	Likert scale
	Usefulness	Applicability and utility of the response. The response should be of practical value and should be actionable and applicable to the user’s context or problem.	• Usefulness⁵ • Helpfulness^63,76 • Applicability⁴³ • Feasibility⁷⁷ • Tangibility⁵⁵ • Actionability⁷⁸	(1) Application/Specialties-specific guidelines/evaluation tools, such as Patient Education Materials Assessment Tool-Printable (PEMAT-P) (2) Likert scale
Understanding and reasoning	Understanding	Ability of the LLM to interpret the user’s query correctly. The response should mimic a grasp of meaning, context, and nuances.	• Comprehension¹⁰	Likert scale
Understanding and reasoning	Reasoning	Capability of the LLM to apply logical processing to generate the response.	• Logical Error¹⁰	Categories (Presence or absence)
Expression style and persona	Clarity	Quality of the response is clear, understandable, and straightforward, making it easy for the user to comprehend the provided response.	• Clarity^46,66 • Conciseness⁷⁹ • Understanding⁵ • Readability⁸⁰ • Comprehensibility⁷⁸ • Fluency⁶⁹	Likert scale
Expression style and persona	Empathy	Ability of the LLM to generate a response that recognizes and reflects the emotions or tone conveyed in the user’s input, simulating a considerate and understanding interaction.	• Empathy^55,63,81,82 • Human care⁸³ • Bedside manner³ • Emotional tone⁷⁸	Likert scale
Safety and harm	Bias	Presence of systematic prejudices in the response, such as racial or gender bias.	• Bias^5,10 • Objectivity⁸⁰	(1) Likert scale (2) Categories (Presence or absence)
	Harm	Quality of response leading to negative outcomes, such as spreading misinformation, reinforcing stereotypes, or otherwise adversely affecting users.	• Safety⁶⁹ • Harmfulness³⁹ • Misleading⁶¹ • Likelihood of harm¹⁰ • Severity of harm¹⁰	(1) Likert scale (2) Categories (Presence or absence)
	Self-awareness	An LLM does not possess self-awareness in the human sense; this quality refers to the LLM’s capability to recognize its processing patterns and limitations.	• Recognition of Limits⁸⁴	Likert scale
	Fabrication, Falsification, or Plagiarism	(a) Fabrication is when the response contains entirely made-up information or data and includes plausible but non-existent facts in response to a user’s query. (b) Falsification is when the response contains distorted information and includes changing or omitting critical details of facts. (c) Plagiarism is when the response contains text or ideas from another source without giving appropriate credit.	• Hallucination⁶³ • Confabulation⁸⁵	(1) Categories (Presence or absence) (2) Likert scale
Trust and confidence	Trust	Confidence in the LLM that it will provide accurate, fair, and safe responses. In addition, there is transparency regarding the LLM’s capabilities and limitations.	• Similarity to expert response^10,74 • Assurance⁵⁵ • Reliability⁸⁰	Likert scale
Trust and confidence	Satisfaction	The LLM meets or exceeds the expectations of the user in terms of response quality, relevance, and interaction experience.	• Satisfaction²⁹	Likert scale

We further breakdown each dimension into the related concepts used in the reviewed articles and show examples of their evaluation strategies.