A Lifecycle Governance and Learning Health System Framework for Trustworthy, Generalizable, and Sustainable Human-AI Partnership in Clinical Practice: Lessons from the Asthma-Guidance and Prediction System (A-GPS)

2.1. A-GPS Platform Architecture

GPS (Guidance and Prediction System) is a disease-agnostic clinical decision support platform, an API-first backbone of reusable services that let us build and operate multiple AI-enabled care solutions consistently. A-GPS is an asthma specialization built on top of that common layer, but the underlying architecture is designed to extend to other diseases (e.g., transplant, gastroenterology, neuro-oncology) by swapping in disease-specific ‘data cards’ and workflows while keeping the platform services (e.g., EHR integration, validation, monitoring, and governance) consistent.

In GPS, the disease context is expressed as a composition of versioned ‘cards’, especially Data Cards that define the data elements, constraints, and provenance needed for a given clinical scenario (Figure 3). Those cards become reusable modules: the platform services stay stable, while the disease implementation changes by selecting and composing the appropriate cards and workflow logic. GPS provides a secure, API-first integration layer that enables real-time, bi-directional communication between clinician workflows in Epic and patient-facing remote patient monitoring devices and wearables. We showcase our Epic workflow, but note that this approach is EHR-agnostic and can be adapted by any health record system given similar human and technical capabilities. Through standardized service interfaces, GPS allows patient-generated data from vendor RPM devices to flow into clinical decision support, while clinician updates, such as care plans or medication changes, are pushed back to patient apps in near real time.

Equally important are advances in AI and digital health infrastructure that enable CDS systems to move beyond standalone algorithms toward practice-integrated, scalable solutions. Interoperability standards such as SMART-on-FHIR (Substitutable Medical Applications and Reusable Technologies-on-Fast Healthcare Interoperability Resources) allow AI-enabled CDS tools to be embedded directly within EHR workflows, delivering context-aware insights at the point of care rather than as external reports.^57,58 This “substitutable app” paradigm, supported by standardized data access and scoped authorization, reduces integration friction and enables deployment at the locus of clinical decision-making, a prerequisite for trustworthy human-AI partnership. Related approaches (e.g., CDS Hooks) further enable context-triggered alerts, which can improve the timeliness and uptake of embedded decision support.^59,60 Cloud-based infrastructure supports scalability, maintainability, and continuous learning, while also enabling centralized governance and monitoring across deployments. In this framing, “advancement” is not defined by model sophistication alone, but by the ability to deliver workflow-concordant, auditable, and maintainable decision support that strengthens (rather than fragments) clinician-patient-AI collaboration.

At the analytic level, advances in natural language processing (NLP) allow clinically meaningful information embedded in unstructured free-text EHR notes, such as symptoms, triggers, and care gaps, to be systematically extracted and synthesized, addressing long-standing limitations of using structured data alone.^61–70 Machine learning methods can then integrate longitudinal structured and unstructured data to estimate future risk and support proactive, precision management.^56,71 In parallel, RPM technologies extend care beyond episodic visits by generating patient-generated digital health data that can inform longitudinal decision-making.^55,72–75

A-GPS was explicitly designed to leverage these advancements in combination. The system synthesizes fragmented clinical data, including free-text EHR notes, using validated NLP pipelines;^{63–67,69,76–81} estimates future asthma exacerbation risk using machine learning models applied to longitudinal EHR data;^56,82,83 integrates RPM data streams to support remote, patient-in-the-loop management;⁵⁵ and is deployed as a SMART-on-FHIR application operating seamlessly within EHR workflows.⁵⁵ By coupling these technological advances with workflow-concordant design, governance, and lifecycle monitoring, A-GPS exemplifies how contemporary AI and digital health capabilities can be operationalized to deliver auditable, maintainable, and clinically actionable decision support within complex health care systems.

Building on this foundational work, a broader Guidance and Prediction System (GPS) framework is now being extended to adult asthma, transplantation, gastroenterology, and neuro-oncology practices, positioning GPS as a scalable and disease-agnostic model for trustworthy AI translation across the clinical and translational science spectrum from discovery to implementation

2.2. Establishment and Proactive Engagement of a Vibrant Strategic Multi-Partner Ecosystem with Shared Value and Goals

Because clinical value depends on adoption, sustained use, and safe scale, not model performance in isolation, A-GPS was built as a socio-technical intervention whose trustworthiness is co-produced with patients, clinicians, and operational stakeholders. Implementation science frameworks and community-engagement theory explain why: workflow fit, role clarity, organizational capacity, and system constraints are primary determinants of whether decision support improves care or becomes ignored noise.⁸⁴

To ensure that each A-GPS output corresponds to an explicit clinical decision target (and to surface failure modes before scale), we operationalized co-design as a structured requirements-and-risk method rather than a qualitative add-on. Consistent with participatory design principles, we treated clinician partner engagement not as a consultation but as a structured method for eliciting requirements, identifying failure modes, and producing implementation-relevant evidence early in the lifecycle.^53,85 For example, we held meetings of an Asthma Advisory Group (AAG), a collection of primary care providers, subspecialists in allergy and pulmonology, and asthma care coordinators, to discuss the design of A-GPS based on National Asthma Education and Prevention Program (NAEPP) 2007 guidelines.⁸⁶ These deliberations served as a clinical content governance mechanism: they constrained intended use, prioritized guideline-concordant decision targets, and clarified what “actionable” means at the point of care for specific user roles. Subsequently, we assessed the feasibility of automatically extracting each prioritized variable from the EHR using structured data and NLP, ensuring that the final feature set balanced clinical importance with technical viability to finalize the list of variables that are displayed in A-GPS dashboard (Figure 1) and/or used in the AI model.⁵⁶ This “clinical salience × technical feasibility” step operationalized an explicit design tradeoff that is often implicit in AI development and is a common source of translational fragility when unmanaged.⁵⁴ The AAG since then has met regularly to continue to guide user interface design, model thresholding, and interpretation of AI outputs. In parallel, we conducted upstream formative research (shadowing, semi-structured interviews, and EHR walkthroughs) to characterize real workflow conditions, cognitive strategies, and capacity constraints across diverse practice settings, thereby grounding design decisions in observed practice rather than idealized pathways.⁵³

Community Advisory Boards (CABs), and a specific Pediatric CAB, provided patient- and family-centered perspectives that informed early design and integration of RPM within A-GPS. They highlighted practical, equity-relevant, and ethical considerations, including variability in home broadband and Wi-Fi access; the need for device-agnostic functionality (e.g., laptop access in addition to smartphones); privacy and data governance (including data management and use); and challenges faced by adolescents in accessing health data during the school day when mobile phone use is often restricted. These insights directly shaped design decisions, such as prioritizing cross-platform access, supporting asynchronous data review rather than real-time interaction requirements, and aligning RPM workflows with realistic patterns of technology access in home and school environments.

However, meaningful integration of stakeholder input required moving beyond episodic consultation toward a more vibrant, cohesive, and collaborative ecosystem that connects practice, research, education, and community partners around shared goals and values. Rather than functioning as a static advisory body, engagement must be structured as an iterative, bidirectional, and dynamic process in which insights from patients, clinicians, researchers, and educators continuously inform model refinement, workflow redesign, evaluation metrics, and governance decisions—creating a learning cycle consistent with learning health system principles. This integrated ecosystem—spanning clinical practice, translational research, workforce education, and community engagement—embodies the kind of broadly engaged team science that is essential for trustworthy, generalizable, and sustainable human-AI partnership in real-world settings. Without such a mechanism, AI tools risk being developed in isolation from the values, constraints, and workflows that ultimately determine their clinical benefit or harm. The A-GPS experience demonstrates that this ecosystem is not a supplementary feature of AI development but a foundational requirement.

2.3. Learning Health System (LHS) Framework for Trustworthy Human–AI Partnership in Practice and Research

Because calibration, workflow effects, and equity properties can degrade as documentation practices, populations, and care processes evolve, A-GPS is governed as a learning health system with pre-specified monitoring signals and corrective actions rather than as a one-time ‘build-and-deploy’ artifact. Operationally, A-GPS treats real-world use as an evidentiary stream that triggers predefined governance actions (e.g., refinement, workflow adjustment, constrained use) when model performance, burden, or fairness signals drift. In an ideal LHS, knowledge generation is embedded in routine care such that data from clinical operations are systematically converted into actionable insights and reintegrated to improve subsequent care delivery, creating a closed-loop cycle rather than episodic evaluation.⁸⁷ Literature suggests this orientation is a foundational for clinical AI, where model behavior and clinical impact can change over time as data, workflows, and clinical practice evolve.⁸⁸

The system spans the full translational science spectrum, from problem definition and data curation (T0) to model development (T1), workflow integration and clinical evaluation (T2–T3), implementation (T3), and dissemination with population-level impact (T4).^89,90 Consistent with contemporary translational science frameworks, these stages were intentionally non-linear, with evidence and operational learning from later phases feeding back to refine earlier assumptions, measures, and design decisions. For A-GPS, this included iterative refinement of the asthma exacerbation prediction model based on real-world model performance and socioeconomic fairness assessments, as well as evolution of data card and reporting formats, from quarterly printouts used in the initial randomized trial,⁵⁶ to real-time EHR-embedded interfaces supporting remote patient monitoring,⁵⁵ and toward broader primary care deployment based on end-users’ feedback. This iterative process operationalized an LHS approach in which model governance, workflow integration, and monitoring strategies continuously evolved in response to real-world use.

The AI Validation Stewardship Program (AVSP) is an institutional translational research and operational infrastructure designed to develop, standardize, and apply reproducible evaluation methods for AI-enabled digital health solutions like A-GPS described above. AVSP convenes multidisciplinary expertise (e.g., health care professionals, software engineering, data science, community engagement, biostatistics, bioinformatics, user experience research, ethics, and governance) to develop, standardize, and apply the foundational methods to generate decision-grade evidence on safety, effectiveness, usability, fairness, and workflow integration across the full AI lifecycle, thereby enabling accountable deployment and sustained stewardship in routine clinical practice.³² AVSP’s core contribution is the operationalization of lifecycle assurance for clinical AI, treating trustworthiness as a testable evidentiary standard with explicit decision points across the preparation, development, validation, deployment, and maintenance of AI-based digital health solutions (Table 2).^49,53,54

Within this framework, AI is not treated as a static artifact but as an evolving component of care delivery. Evidence generated through clinical use informs model refinement, workflow adjustments, and governance decisions. Operationally, this requires predefined mechanisms for surveillance and corrective action, for instance, risk-based monitoring, issue triage, root-cause analysis, or equity risks are detected and mitigated rather than discovered retrospectively.⁹¹

Complementary “passive deployment” methods, such as staging, simulation, and silent deployment, can further strengthen the learning cycle by enabling extended observation and verification prior to broad user exposure.⁴⁹ This approach enables ongoing assessment of safety, effectiveness, usability, fairness, and sustainability, supporting responsible AI deployment over time. Because health systems are socio-technical systems, this continuous learning posture is also essential for sustaining trustworthy human-AI partnership: it evaluates not only model performance, but also the evolving interaction among clinicians, patients, workflows, and organizational constraints that ultimately determines benefit and risk in practice.⁹²

2.4. Task Definition and Prediction Model Specification for A-GPS

For asthma management, the ideal model would identify what intervention will prevent an asthma exacerbation. However, answering that type of question requires strong causal evidence, typically from randomized clinical trials, which is not always available in early development. Instead, we focused on a more practical and clinically useful goal: identifying which patients are at higher risk of asthma exacerbation (AE) as an “outcome reasoning” problem[CIT]^93,94, for which machine learning classification models are well-suited for finding reliable, but not necessarily causal, correlations.

Given the available data, we developed a model to estimate whether a patient is likely to experience an exacerbation within the next year.⁵⁶ The first version of A-GPS tried a Naïve Bayes classifier with a practice-based cohort of 590 pediatric patients drawn from the primary care panel. Of these, 300 patients were allocated to the development (training) cohort and 290 to a completely held-out validation cohort.⁹⁵ Recognizing that correlation-based models can only provide exploratory insights and not reliable causal relationships, but that these exploratory insights can serve as a useful sanity check (or, face validity), five variables emerged as the most influential correlates: prior asthma exacerbation, asthma symptoms, hospital visits, rescue medication use, and controller medication use. The final Naïve Bayes model incorporated these features and generated an AE risk score that was updated quarterly and communicated to primary care providers as a binary risk category (high vs. low), using a threshold derived from receiver operating characteristic (ROC) curve analysis.⁵⁶ This Naïve Bayes classifier was selected to predict one-year asthma exacerbation risk in a pediatric primary care cohort. Model performance, assessed by area under the ROC curve (AUCOC), was 0.74 in the development cohort and 0.78 in the independent validation cohort, demonstrating good discrimination and generalizability. Although AUROC is an informative summary of discrimination, it does not evaluate calibration (whether bins of estimated risk empirically match with the fraction of positive cases in each bin) nor quantify the clinical consequences of errors at plausible decision thresholds. Because acceptable discrimination can coexist with miscalibration and clinically important failure modes, we treat calibration assessment, decision-analytic evaluation, and subgroup model performance heterogeneity as required evidence for trustworthy use.^96–100

For a second version, which is currently what is deployed, more experienced machine learning modelers tried a wider range of models. In practice, random forests are the best off-the-shelf classifier, able to automatically find nonlinear correlations and correlations with interactions of input variables and the binary response variable,¹⁰¹ and so they form a useful baseline; in addition to these, regularized logistic regression is often a good lightweight (and more interpretable) choice. In the modelers’ testing, they found a hyperparameter for which L₂-regularized (ridge) logistic regression had the best performance, over random forests as well as L₁-regularized (lasso) logistic regression and elastic net logistic regression over all tested hyperparameters.

With data that has been collected on an ongoing basis over the past few years, we plan to further refine A-GPS not only in terms of its performance within the given task (estimating the probability of an asthma exacerbation event within 1 year for a given pediatric patient), but refining the task definition and making new models to fit this new target. Specifically, we plan to try a survival (timeto-event) model to produce more timely insights, for which we hope at-home spirometry integration will prove useful; and we will attempt some causal modeling by which to say what input variables might be intervened upon to prevent asthma exacerbations (recognizing the fragility of such approaches, and the need for robust follow-up testing).

2.5. Data Card and Model Card as a Key for Human-Centered Co-Design of AI Model Development and Evaluation Framework

To support calibrated reliance at the point of care, A-GPS was designed so that every displayed element is (a) decision-relevant, (b) interpretable in clinical context, and (c) aligned with guideline-concordant actions under real time and workload constraints. This work aligns with and discretizes international efforts such as STANDING Together.¹⁰² As described above, clinicians in the AAG identified variables that were most relevant for asthma management based on national guidelines and practical experience. Then, to make data assumptions transparent, auditable and resilient to upstream drift, A-GPS uses standardized Data Cards (or Feature Cards) that specify each data element’s definition, provenance, known limitations, and governance expectations. (Figure 4). Data Cards provide an interpretable summary of both structured and unstructured data elements associated with a specific AI model or project.⁷⁷ For each data element, the Data Card documents: (1) a clinical description to contextualize relevance and intended use; (2) visual summaries illustrating data availability, completeness, and distribution; (3) reproducible descriptions of data extraction logic; and (4) data source and lineage information to support auditability and governance. At the project level, Data Cards include metadata such as project scope, clinical task, and responsible leads to ensure clarity and traceability. To support lifecycle stewardship, Data Cards are maintained under configuration management (version control) and reviewed at predefined checkpoints, enabling reproducible re-extraction, change-impact assessment, and longitudinal surveillance for drift as source systems, ETL (Extract, Transform, and Load) logic, or clinical workflows evolve. Where feasible, the documentation is designed to be partially computable (e.g., structured representations) to support automation, interoperability, and enterprise scale stewardship. This approach supports a plug-and-play templating strategy, allowing the A-GPS architecture to be efficiently extended to other clinical domains while maintaining transparency and trustworthiness.

In contrast, Model Cards are summaries of the purpose, provenance, and operation of AI/ML models. We have specifically adopted the “Model Facts” label version of Sendak et al.¹⁰³ (Figure 5). Model Cards complement Data Cards by linking data provenance to intended clinical claims and by specifying intended and non-intended use, operating conditions, uncertainty, and known limitations. For A-GPS, the Model Card documents data preprocessing; model class, specification, and any tuning parameter values; training data provenance; target population; and overall performance and tests for performance disparities by subgroup like for SES using HOUSES. Finally, we map these phase-aligned artifacts and evaluations to contemporary reporting expectations (e.g., DECIDE-AI, SPIRIT-AI, CONSORT-AI, TRIPOD-AI), to strengthen methodological rigor and reproducibility across translational stages while preserving local accountability for real-world deployment.^97,104–107

Precise version matching between Model Cards and Data Cards is essential to ensure reproducibility, enable meaningful drift detection, and prevent silent misalignment when data definitions, ETL pipelines, or source systems evolve. Without disciplined versioning and pairing, model updates and data changes can propagate undetected discrepancies into clinical use.

2.6. Fairness Evaluation of AI by Socioeconomic Status

Ensuring fairness in clinical AI requires evaluation beyond traditional demographic categories to include structural determinants of health, particularly socioeconomic status (SES). To assess the fairness of the A-GPS model, we leveraged the HOUSES index (HOUsing-based SocioEconomic Status), an individual-level SES measure derived from publicly available housing data and validated for health research. Unlike aggregate area-based measure, HOUSES enables patient-level assessment of socioeconomic context within EHR data.^108–110 Using the HOUSES index, we found that children in lower SES strata exhibited higher prediction error rates and greater EHR data incompleteness, highlighting structural contributors to AI bias rather than intrinsic model flaws.¹⁰⁸ Specifically, lower SES was associated with greater EHR data incompleteness, including higher rates of missing asthma severity documentation (41% vs. 24%) and undiagnosed asthma despite meeting established criteria (12% vs. 9.8%). These findings suggest that SES-related differences in EHR data quality contribute to differential model performance and calibration.

This approach is consistent with current guidance and reporting frameworks that emphasize transparent documentation of intended use, performance limitations, and monitoring plans for clinical AI.¹¹¹ Our “user-centered” approach outlining the critical information about model performance, limitations, and training data that is accessible to clinicians allows implementation of human centric design. More broadly, these findings informed ongoing efforts to address representation gaps and underscored the importance of SES-aware assessment when documentation quality and access to care vary systematically across populations.¹⁰⁸ Building on this work, we recently received NIH funding (HL171508) to systematically investigate and mitigate AI bias in pediatric asthma prediction models, with a focus on SES alongside race/ethnicity and rurality. Part of this work is developing new and applying existing biostatistics frameworks to carry out statistical testing for model disparities, as even descriptively extreme disparities may not be statistically significant if there is a small sample size. This work represents a critical step toward trustworthy, equitable, and responsible application of AI in asthma care.

2.7. Human-Centered Workflow Integration

Effective AI must be integrated into clinical workflows in ways that augment clinical cognition and coordination rather than introduce additional fragmentation or task burden. Recent literature underscores the amplification of human efforts through effective human-machine collaboration.¹¹² Accordingly, we conceptualize A-GPS as a workflow-concordant socio-technical intervention engineered to deliver context-relevant information at the locus of decision-making while preserving clinical agency. Consistent with contemporary workflow-integration requirements, we explicitly specify the baseline (i.e., “as is”) standard of care workflow and the envisioned (e.g., “to be”) workflow with A-GPS, including trigger points, decision owners, downstream action, and tasks intentionally reduced or eliminated; where workflows vary across sites or services, we treat this heterogeneity as a first-order design and evaluation constraint rather than a nuisance variable.^49,113,114 A-GPS defines explicit human-in-the-loop decision pathways that clarify when AI outputs are reviewed, by whom, and how they inform clinical action. To prevent unsafe automation bias and to preserve accountable clinical decision-making when A-GPS signals conflict with clinician judgment, we predefine escalation pathways, override mechanics (with rationale capture), and resilient downtime workflows. For clinically consequential CDS, we additionally define decision control and “override” mechanics (including rationale capture) and specify fallback/downtime workflows to preserve safe care when outputs are unavailable, unreliable, or intentionally disabled, thereby operationalizing resilience and auditability as part of routine use.^91,113,114 Validation metrics, their risk thresholds, and alerting strategies were determined collaboratively with clinicians to balance true positive rate, positive prediction rate, flag rate (workload), and patient safety.¹¹⁵(Figure 6) Threshold selection was treated as a clinical-utility problem, explicitly considering the asymmetric consequences of false reassurance versus false alarms and the operational risks of alert fatigue rather than as an optimization of discrimination alone. Accordingly, threshold decisions were paired with prospective measurement of alert burden, downstream actions, and override/discordance rates as primary safety and effectiveness signals, recognizing that “model performance” is mediated by workflow, attention, and organizational response capacity.^113,116

A-GPS evolved from offline summary reports to a fully EHR-embedded CDS tool using SMART-on-FHIR integration.^55,66 This deployment architecture enables real-time, context-aware delivery within the existing clinical workflow while improving maintainability and governance through substitutable, standards-based integration. To reduce translational risk prior to scale, we incorporate staged deployment assessments (staging, simulation, and shadow/silent deployment) and limited-release pilots, consistent with emerging guidance for early-stage evaluation of AI decision support in real clinical settings.¹⁰⁷

2.8. AI Evaluation via AI Simulation Lab and Pragmatic Clinical Trials

Simulation usability testing is being advanced to characterize use(r) errors, comprehension of risk displays, and decision behavior under realistic time and workload constraints—providing human factors evidence to refine the interface and interaction design prior to broader implementation.^54,117 AI evaluation in a simulation setting is a novel concept and emerging field, and our current work on evaluating A-GPS using an AI Simulation Lab is beyond the scope of this review paper. We are planning to report this in a separate paper in the future and focus on AI (A-GPS) evaluation by practice-integrated pragmatic RCTs.

To establish decision-grade evidence that A-GPS improves care delivery (utility, safety, workflow burden, and feasibility) under routine conditions—not merely retrospective discrimination—we prioritized pragmatic randomized evaluation of the deployed intervention. Accordingly, in 2021, an early version of A-GPS was evaluated through a practice-integrated pragmatic RCT conducted in partnership with primary care leadership.⁵⁶ This design choice was deliberate as pragmatic trials estimate effectiveness under routine clinical conditions and generate decision-relevant evidence about clinical utility, safety, workflow impact, and implementation feasibility—domains that are frequently under-characterized when AI tools are evaluated solely in retrospective datasets.^118,119 Importantly, the trial was embedded within existing practice operations rather than conducted as a parallel research initiative. Practice leaders were engaged from study design through implementation planning, enabling rapid translation of findings into workflow refinements and continued clinical and translational science (CTS) work following trial completion. This partnership reduced time from evaluation to operational scaling and ensured that the intervention aligned with real-world constraints.

In a one-year single-center pragmatic RCT involving pediatric primary care patients with asthma (with 90 patients under A-GPS intervention vs. 94 patients under usual care), A-GPS significantly reduced clinician EHR review time for asthma management by 67%.⁵⁶ Notably, this efficiency endpoint captured clinician-perceived chart synthesis burden, a workflow-sensitive outcome that is increasingly recognized as clinically consequential because health IT related stress is prevalent and associated with burnout.¹²⁰ While reductions in asthma exacerbations (AE) were observed in both groups, differences between groups were not statistically significant. Importantly, no adverse events were identified, clinician satisfaction was high, and mean health care costs during the trial period were lower in the intervention group compared with controls (–$1,036 vs. +$80), although this difference did not reach statistical significance (P=0.12). These findings are consistent with the broader argument that clinical AI should be evaluated as an intervention embedded in care delivery (including its human-AI interaction and workflow effects), and that evidence from RCTs remains the gold standard for credible claims of benefit.^105,118

To further advance A-GPS toward remote, practice-integrated care, real-time patient data from a home spirometry device and mobile application (AsthmaTuner) were integrated with EHR-based CDS. In a dual-site, parallel-group, pragmatic RCT involving 22 clinician-patient dyads across Division of Community Pediatric and Adolescent Medicine, we demonstrated the feasibility, usability, and satisfaction of this integrated remote asthma management approach.⁵⁵ Because AI-enabled digital health tools are inherently socio-technical and multi-component, we explicitly foreground feasibility and usability as primary endpoints in early-stage studies, consistent with guidance emphasizing rigorous, transparent reporting of real-world clinical evaluation for decision support.^107,121

Collectively, this work established a reproducible framework for evaluating AI-enabled digital health interventions in pragmatic clinical trials and routine practice, including rural and community-based settings. By demonstrating feasibility beyond tertiary care environments, this approach supports scalable dissemination and adaptation of AI-enabled CDS for asthma care across diverse healthcare systems nationwide. To strengthen interpretability and reproducibility of these evaluations, reporting expectations such as CONSORT-AI and DECIDE-AI provide a structured basis for making explicit the AI-enabled workflow context, user roles, and handling of errors and unintended consequences.^105,107

2.9. Clinical Utility Cards and Post-Deployment AI Evaluation Framework

Because documentation does not detect transportability failures, calibration drift, or equity regressions in live care, post-deployment trustworthiness requires prospective evaluation and surveillance beyond model-card reporting. Empirical evaluations of widely deployed clinical prediction tools illustrate why models may exhibit strong apparent model performance during development yet demonstrate materially degraded transportability, calibration, and clinical utility when implemented in new settings or under evolving care processes, limitations that are not resolved by documentation alone and can remain invisible without prospective evaluation and surveillance.^122,123

As a tool to help bridge this gap, we introduce the concept of a Clinical Utility Card (Figure 7), which documents how an AI model is operationalized as a clinical product and whether it improves outcomes, processes, or efficiency. Building on principles from SPIRIT-AI guidelines and Quality Improvement science, Clinical Utility Cards document the transition from model to intervention, and a structure for post-deployment monitoring. The Clinical Utility Card describes how the model is delivered (e.g., “generating a risk score from EHR data” or “reporting findings back to the medical record”), who uses it (specific clinician roles), in what clinical context decisions are made, and how outputs are communicated and supported with guidance or resources. Importantly, Clinical Utility Cards define the target patient population and explicitly assess alignment with the model development cohort, noting whether use represents perfect overlap, interpolation, or extrapolation, with rationale provided when populations differ. Evaluation extends beyond predictive accuracy to include outcome measures (e.g., health or safety outcomes), process measures (e.g., actions promoted by the intervention), and balancing measures to detect unintended consequences. Consistent with contemporary prediction-model methodology, we treat calibration, clinically consequential error characterization, and subgroup model performance as required evidence for trustworthy use, because discrimination (performance) metrics alone can obscure systematic over- or under-estimation of risk and heterogeneous harms across clinical contexts and populations.^96,122 In addition, post-deployment evaluation must contend with intervention-outcome feedback (i.e., clinical actions triggered by model outputs that alter downstream labels), which can confound naïve “performance drift” interpretations and reinforces the need for prespecified monitoring designs and decision rules.¹²⁴

The choice of study design to evaluate clinical utility should be based on pragmatic considerations and the severity of potential harms from an AI intervention. As such, the Clinical Utility Card is study design agnostic, with options ranging from explanatory randomized trials to robust observational studies when appropriate. Thus, Clinical Utility Cards provide a structured, transparent framework for demonstrating whether AI-enabled CDS tools achieve their intended clinical impact in real-world settings. Applying this to A-GPS, A-GPS was deployed to primary care providers managing pediatric patients with asthma, with full alignment between the deployment population and the model development cohort. Delivered quarterly through the EHR, A-GPS provided actionable summaries of asthma care quality, risk factors, and one-year exacerbation risk, along with management options. Clinical utility was evaluated in a stratified randomized controlled trial comparing A-GPS-supported care with usual care, using asthma exacerbations as the primary outcome and care efficiency and follow-up timeliness as key process measures. Critically, the Clinical Utility Card also functions as the “measurement contract” for stewardship: it prespecifies which real-world signals constitute acceptable operating model performance, and which deviations require recalibration, constrained use, or decommissioning. Because clinical environments, documentation practices, and patient populations evolve, we further specify post-deployment monitoring as an explicit evidentiary obligation: 1. Monitoring data quality and feature availability (to detect upstream shifts), 2. Monitoring calibration and clinically meaningful error patterns over time, 3. Monitoring subgroup model performance and equity-sensitive measures, and 4. Monitoring workflow/operational signals (e.g., alert acceptance, override patterns) as leading indicators of utility and safety (Figure 8).^122,123

To ensure that Clinical Utility Cards capture decision-grade evidence aligned with national and regulatory expectations, other established EQUATOR reporting guidelines across the AI translational lifecycle can be utilized. For example, early-stage feasibility and workflow studies of AI-assisted clinical decision support can be guided by DECIDE-AI, while protocol development for confirmatory trials follow SPIRIT-AI, and completed randomized clinical trials are reported using CONSORT-AI. Depending on study objectives, additional frameworks—including TRIPOD-AI and PROBAST-AI for prediction model evaluation and CHEERS-AI for economic impact—can be incorporated.^{97,105,107,125,126}

2.10. Implementation and Dissemination of Know-How for Trustworthy Human-AI Partnership

We will need optimized and integrated systems and processes enabling us to address sustainability of AI models and AI-powered CDS platforms. The required systems and processes are: 1) REAIM framework-based operational plans assessing implementation and sustainability, 2) priority of decision making metrics for sustainability by stakeholders, 3) centralized AI inventory listing all AI models in the pipeline and production, 4) value beyond traditional AI evaluation metrics (safety, effectiveness, utility, fairness, and sustainability), 5) realistic evaluation for institutional capacity for monitoring and maintenance of AI models, 6) longitudinal monitoring and threshold for decommissioning AI models and 7) automated system ad process for AI model monitoring.

Implementation of A-GPS followed a staged deployment strategy, beginning with engaged clinicians and expanding incrementally across practices to enable safe, scalable translation while addressing system-level barriers in real-world practice. The Asthma Advisory Group (AAG) clinicians and care coordinators were engaged early from development stage to implementation in a single pediatric primary care clinic, subsequent expansion across a regional practice network, and ultimately a scale-up across the entire pediatric and then adult asthma practice. At each stage, operational, technical, and workflow-related challenges have been and will be systematically identified and addressed, allowing iterative refinement prior to broader dissemination.

Institutional governance structures played a critical role in ensuring safe, effective, and ethical use. In 2024, A-GPS was reviewed by the institutional governance structures,⁴³ including the AI Translation Advisory Council and the Software as a Medical Device (SaMD) Review Board.⁴³ Together these bodies documented regulatory and translation risks and clarified responsibilities for post-deployment monitoring. The SaMD Review Board contributed by interpreting laws pertaining to CDS software technologies (i.e., the 21^st Century Cures Act and the Federal Food, Drug, and Cosmetic Act §520(o)(1)(E)); further, this body aided the A-GPS team in evaluating the Food & Drug Administration’s evolving interpretation of the laws and regulations through guidance documents published on CDS tools, including the 2022 and 2026 publications.^127,128 As part of this evaluation, a risk analysis was completed to identify appropriate mitigations to manage patient safety; the process was informed by ISO 14971:2019.¹²⁹ The AI Translation Advisory Council then reviewed evidence supporting Enterprise-established AI best practices for A-GPS and recommended progression to the planning phase for integration within the existing clinical workflow. Risks identified during review were explicitly enumerated, including (1) the need for continued clinical validation of model performance to support next-step guideline recommendations and (2) the importance of clearly defined responsibilities for post-deployment auditing, monitoring, and maintenance. They recommended that these risks do not preclude implementation but are incorporated into shared documentation, accountability structures, and monitoring plans. This risk-aware, governance-driven approach ensures responsible scale-up of A-GPS while maintaining clinical trust, safety, and sustainability.

Following deployment into the clinical practice, A-GPS will be subject to ongoing monitoring to ensure safe and appropriate use per Enterprise policy. Required activities include assignment of accountable clinical and technical owners, tracking of usage and fidelity to intended workflows, and identification of safety or model performance issues through routine feedback and incident reporting. The scope and rigor of post-deployment monitoring increase in proportion to the clinical risk and potential impact of the technology, with higher-risk tools subject to more intensive oversight and review These practices align with the AI Translation Advisory Board’s articulated AI best practices and with prevailing academic perspectives on risk-based post-deployment monitoring of clinical AI systems.

By leveraging SMART-on-FHIR architecture and standardized documentation, key components of A-GPS have been evaluated and adapted across institutions, including collaborations with external pediatric centers. While structured data elements (e.g., laboratory results, ICD/CPT/SNOMED codes) can be relatively straightforward to standardize and extract across institutions, dissemination of NLP-based ascertainment algorithms (a core component of A-GPS) poses substantially greater challenges. For example, two most important NLP algorithms integrated in A-GPS are NLP-PAC and NLP-API which are known to help with; 1) early diagnosis of asthma and reducing delayed diagnosis or undiagnosed asthma, 2) serving as digital biomarkers identifying and predicting a subgroup of patients with asthma at high-risk of ARI and AE, and 3) identifying subgroups of asthma with higher risk of multimorbidities.^{63,64,67,69,70,76} Given the scientific and clinical value of these NLP algorithms, we assessed their generalizability used for A-GPS in different settings. We adapted the NLP-PAC algorithm at Sanford Children’s Hospital to evaluate its external validity.⁶⁹ Both criterion validity (i.e., accuracy of asthma ascertainment) and construct validity (i.e., expected associations between established asthma-related risk factors and NLP-PAC derived asthma status) were assessed, demonstrating the generalizability of the automated asthma ascertainment algorithm across institutions. In addition, we systematically examined the challenges, solutions, and best practices associated with cross-institutional deployment of NLP asthma ascertainment algorithms during implementation of NLP-PAC and NLP-API at Texas Children’s Hospital. We followed a structured adaptation framework that included (1) legal and security review processes, (2) identification of cross-institutional differences in EHR, (3) generation of institution-specific annotations for gold-standard labeling, (4) preprocessing and data-format harmonization, and (5) iterative algorithm refinement and validation. Our findings demonstrate that cross-institutional deployment of NLP asthma ascertainment algorithms is feasible and reproducible when guided by a structured adaptation framework that explicitly addresses technical, institutional, and governance-level barriers to implementation. These efforts provide a reproducible pathway for scaling AI-enabled CDS tools while maintaining trust and equity. Together, SMART-on-FHIR–enabled deployment and these dissemination strategies provide a reproducible pathway for scaling NLP-enabled AI tools across diverse institutions, supporting collaborative asthma research, equitable implementation, and broader adoption.