Abstract
Artificial intelligence/machine learning models are being rapidly developed and used in clinical practice. However, many models are deployed without a clear understanding of clinical or operational impact and frequently lack monitoring plans that can detect potential safety signals. There is a lack of consensus in establishing governance to deploy, pilot, and monitor algorithms within operational healthcare delivery workflows. Here, we describe a governance framework that combines current regulatory best practices and lifecycle management of predictive models being used for clinical care. Since January 2021, we have successfully added models to our governance portfolio and are currently managing 52 models.
Keywords: artificial intelligence, machine learning, decision support systems—clinical, Governance Models—organizational, Health Information Management/Organization & Administration
INTRODUCTION
Tools for clinical decision-making based on artificial intelligence/machine learning (AI/ML) typically use data from clinical systems. Those data are then processed to generate individual-level estimates of risk. Although use of such models has seen recent rapid growth, interviews with clinicians, data scientists, and executive leadership from hospitals and health systems underscore the need for governance principles and best practices to guide model development, implementation, evaluation, and maintenance.1 Importantly, this groundswell of interest parallels the US Food and Drug Administration’s (FDA’s) focus on incorporating the regulation of AI/ML tools within its “Software as a Medical Device (SaMD)” regulatory framework, which is based upon risk stratification and quality management.2
Despite regulatory guidance and publications addressing specific components of predictive model lifecycles, a framework encompassing the full lifecycle of an algorithm or model—development, validation, implementation, and continuous quality assurance—does not exist in the healthcare domain.3 Recently, clinical models have been scrutinized for lacking accuracy and appropriate impact when deployed in “real-world” healthcare environments.4–6 This issue has been further compounded by siloes that characterize AI/ML development, with imperfect hand-offs from commercial model developers and academic researchers on one side and clinical teams and patients on the other. Differences in clinical processes, available resources, and technological infrastructure all can adversely affect performance of new AI/ML models, even those approved by the FDA. They may also waste valuable resources or result in deployment of biased models that can harm patients. A multidisciplinary system of governance that includes the entire model lifecycle, allowing AI/ML models to be used safely and effectively, is essential for avoiding such outcomes.
Below, we describe a governance framework for AI/ML tools created and deployed at the Duke University Health System (Duke Health)—one that combines regulatory best practices and an integrated view of the model lifecycle to engage development teams and locally deploy AI/ML models of varying conditions or risk.
A FRAMEWORK FOR ALGORITHM-BASED CLINICAL DECISION SUPPORT
ABCDS lifecycle phases
The development process for medical devices includes preclinical, pilot, pivotal, marketing application/approval, and postmarketing surveillance stages.7 We superimposed this framework onto the software development lifecycle8 to arrive at a simplified algorithm-based clinical decision support (ABCDS) lifecycle process with 4 phases: (1) model development, (2) silent evaluation, (3) effectiveness evaluation, and (4) general deployment. Combining the lifecycles of both medical devices and software integrates medical device development’s rigorous requirements for development teams to demonstrate preclinical and pilot effectiveness with software development’s emphasis on quality. Quality assurance and quality control are expected in the development and maintenance phases, respectively, to ensure that tools meet their intended purposes and function properly. At an institutional level, we overlaid a rigorous governance structure to provide support at 3 checkpoints between these phases, each featuring review by an expert multidisciplinary subcommittee. Figure 1 illustrates the phases using as an example the assessment of a third-party readmission model, which is described in greater detail in the Supplementary Materials. Checkpoints were inspired by quality gates, which are predefined quality measures that software development projects must meet to proceed to the next stage.9,10
Figure 1.
“Just-in-time” checkpoint gates (Gx) along the ABCDS lifecycle that help development teams prepare for approval to proceed to the next ABCDS lifecycle phase. Top: Examples of key information requested from development teams for Gx approval. Bottom: Example information provided by development teams to proceed for a third-party readmission model (see ACBDS examples in Supplementary Materials for further details). Algorithm-based clinical decision support (ABCDS); model development checkpoint (G0); silent evaluation checkpoint (G1); effectiveness evaluation checkpoint (G2); general deployment checkpoint (Gm).
Below, we describe each phase.
Model development
As a precursor to AI/ML model development, the clinical and model performance requirements are first defined and documented by a development team; the model is then developed and the data infrastructure built in an iterative process for retrospective evaluation. This process typically includes “traditional” model validation, such as internal or external validation in the population for which the tool is intended. Performance metrics are selected and measured and potential clinical impact is studied.
Silent evaluation
Prospective silent evaluation ensures that the workflow is appropriate and the model’s predictions are similar to those generated during development. The tool, now defined as the model and its user interface (UI), is integrated with clinical information technology systems. To test the tool, model performance is studied in silent mode (without results being shown to clinical users) to evaluate how the model performs with real-life clinical data. The development team also partners with information technologists and implementation science experts to refine the UI and workflow. Finally, end-user acceptance testing is performed.
Effectiveness evaluation
Following silent evaluation, a prospective effectiveness evaluation is performed to determine the tool’s impact and value in a limited clinical setting. Ideally, this step consists of a structured comparative study focused on prespecified clinical or patient-facing outcomes. The goal is to compare the new tool’s performance to an existing standard, thereby determining net improvement in clinical care. The evaluation also assesses how well the tool is adopted by its intended users and their perceptions of it.
General deployment
During general deployment, the model is included in workflows for routine clinical use. Performance and impact are monitored regularly to detect any performance drift. When deviations occur, the model is subjected to an additional development cycle, predominantly focused on retraining. Any issues and enhancement requests noted by clinical end-users are also addressed in this phase.
Model registration, triage, and checkpoints
Model registration
The first step in creating awareness of a new model and facilitating transparency is model registration, which must occur before a model proceeds through checkpoints. During registration, information on model characteristics, development data, and intended uses are collected, following best practices.11–13 At Duke Health, the Machine Intelligence Tracking Platform for Algorithms (MITRA) captures facts related to an algorithm, including its clinical and operational stakeholders, indication, methodology, and past/current versions, among other metadata.
Model triage
As we implemented our governance framework, we recognized that it is also needed for algorithms beyond just those that are AI/ML-driven, in order to ensure high-quality care for our patients. All algorithms used to deliver patient care are registered and triaged. Upon registration, we categorize models based on knowledge transparency: (1) Standard of Care Models, defined by supporting literature, guidelines, and/or professional organization; (2) Knowledge-Based Models, derived by local clinical consensus; and (3) Data-Driven Models, including AI/ML models trained on patient data (Figure 2). We triage each model using a risk-based approach. Rigor of review is based on (1) level of knowledge transparency, (2) whether a model constitutes SaMD under the FDA definition according to an internal review, and (3) whether the model has received FDA approval (Figure 3).
Figure 2.
Categorization of knowledge transparency among models. ABCDS: algorithm-based clinical decision support.
Figure 3.
Algorithm-based clinical decision support (triage process). *Fast track versus full evaluation is subject to ABCDS discretion. SaMD: Software as a Medical Device; FDA, US Food & Drug Administration.
Based on these risk-based considerations and other information provided in the registration form (see Supplementary Material), a model could be triaged to require full committee review, a fast-track review, or no further review (Figure 3). For example, ML models that have not received FDA approval but could be considered SaMD require full committee review. In contrast, standard-of-care models do not require further review unless a particular model is being used for a purpose or patient population unsupported by its underlying evidence base.
Between each development phase are checkpoint “gates” as illustrated in Figure 1 (G0, G1, G2, and Gm). These checkpoints were established so that the development team could engage with the ABCDS governance to: (1) review current progress; (2) review model performance characteristics and/or application to the outcome of interest; (3) assess adherence to clinical decision support and technological best practices; and (4) assess readiness for progression to the next lifecycle phase. Each checkpoint is overseen by a subcommittee comprising specialists with expertise relevant to each checkpoint’s focus. After the model has undergone registration and triage, it proceeds to one of the following checkpoints described below to ensure its readiness for the next phase of its lifecycle (more detailed descriptions of checkpoints G0, G1, and G2 are provided in the Supplementary Material).
Checkpoint G0
The G0 checkpoint ensures that the model is ready to proceed to Silent Evaluation. At this checkpoint, retrospective model results and training data are presented and reviewed, as are design and performance criteria for the Silent Evaluation Phase. The evaluation subcommittee reviews the implementation plan, focusing on workflow impact and UI design. Clinical and regulatory experts may offer guidance on how the final UI and workflow design could affect whether the ABCDS tool is characterized as SaMD. Any gaps in clinical, development, or technological resources relevant to the project’s ultimate success are also identified.
Checkpoint G1
This checkpoint ensures that the ABCDS tool is ready to be implemented and evaluated in a limited capacity during the Effectiveness Evaluation phase. At this checkpoint, the results of the silent evaluation would be presented and reviewed and model performance and effectiveness metrics established. Details of the effectiveness study design are also reviewed before the model is approved to proceed further. The regulatory framework for the model and model classification (eg, as SaMD) are assessed by specialists with expertise in FDA medical device regulatory affairs within our School of Medicine. Tools classified as SaMD but not approved as such by the FDA may only proceed to effectiveness evaluation upon consultation with the FDA or approval by the Duke Health institutional review board.
Checkpoint G2
This checkpoint determines whether the ABCDS tool can be deployed beyond the study settings involved in the Effectiveness Evaluation phase. Justification for general deployment is based on expected improvements in outcomes and processes seen during the effectiveness study. These improvements are then evaluated to assess whether they justify the widespread adoption of, and resources needed to maintain, this ABCDS-based intervention. Further, monitoring plans assessing implementation, operational, and resource readiness are reviewed, and monitoring criteria of the model inputs and outputs to detect model deviation are created.
Checkpoint Gm
Throughout the General Deployment phase, the ABCDS tool is monitored at regular intervals with various checkpoints (Gm). Monitoring criteria established during G2, including model performance, stability, adoption, clinical outcome, and fairness and equity metrics are reviewed. Justification by clinical and business owners are provided for ongoing deployment and use.
PRINCIPLES AND VALUES
The ABCDS Oversight process was designed around 2 key criteria: quality and clinical impact. Regarding the former, we focus on quality by design, quality by validation, and quality control through monitoring. The earliest stages of our process focus on the critical step of identifying and developing a fit-for-purpose AI/ML model.14 Specifically, we scrutinize the model’s intended use and weigh the capacity for seamless integration of the ABCDS tool with existing clinical workflow. We ensure central archiving of documentation and testing evidence for each tool, and require end-user testing to demonstrate usability. Because AI/ML models are expected to drift following deployment due either to inherent changes in underlying data or different end-user behaviors, we also require that monitoring plans are established.15
Population-level monitoring of clinical outcomes lies at the heart of the second criterion of clinical impact. This requires us to examine the effects of each ABCDS tool on patient-facing clinical outcomes, in contrast to other examples of clinical decision-support governance that emphasize model-level statistical measures more exclusively.16 Our multi-stage framework restores the focus to the patient, yet still ensures appropriate methodological rigor.
Considerations for health equity and algorithmic fairness
Health equity—ensuring high-quality healthcare for all patients—is a fundamental principle for the ABCDS committee. However, we also recognize that AI/ML tools not only have the potential to promote health equity but also are capable of worsening health inequities. For this reason, the committee considers the potential for inequities across 4 key domains: sex, race/ethnicity, age, and insurance status. Although we do not exclude the use of race/ethnicity in CDS tools, we require developers to justify their inclusion and the mechanism(s) through which race/ethnicity may operate on risk.17 During the G0 stage, we assess a tool’s potential for algorithmic bias. While there is no single best metric for evaluating the presence of algorithmic bias, we scrutinize the algorithm for differential performance across subgroups as well as whether the resulting decision rule shows differential capture (ie, sensitivity). However, we also recognize that health equity may not be assessable with observed data.18 Finally, while we continue to assess differential performance during G2 and GM stages, we focus on the resulting actions of the CDS tool to ascertain whether more services and interventions are provided to particular groups (algorithmic fairness).
RESPONSIBILITIES
Successful model development, implementation, evaluation, and monitoring require clear definitions of roles and responsibilities. Multiple areas of expertise are represented, including clinical, biostatistics, data science, informatics, policy/legal, and implementation/population health science. Below, we describe individual teams and committees with direct input into the ABCDS process.
Development team
The development team includes the data scientists responsible for developing the model; the clinical owner responsible for the clinical workflow; and the business owner who articulates the business need and defines the model requirements.
ABCDS oversight committee
Given the complexity and resource requirements of AI/ML deployments, we have established an executive-level committee that provides institution-wide oversight and governance. At our institution, this committee is cochaired by the Duke Health System’s chief health information officer and the School of Medicine’s vice dean for data science. It also includes the chairs of the 3 subcommittees that scrutinize models at their respective checkpoints (Figure 1), as well as representatives of health system nursing and innovation. This committee defines the ABCDS lifecycle and checkpoints as well as cataloguing, review, and approval of all model deployments as they progress through their lifecycles. The work of this committee and the respective subcommittees is organized and managed by an experienced program leader with expertise in the health IT product lifecycle.
As noted earlier, 3 subcommittees (Evaluation, Implementation and Monitoring, and Regulatory) report to the ABCDS Executive Committee. These subcommittees are convened on an ad hoc basis to assist development teams and/or provide guidance to the ABCDS Oversight Committee (Table 1).
Table 1.
ABCDS oversight structure
| Subcommittee | Expertise | Responsibilities |
|---|---|---|
| Evaluation | Data scientists & physician-scientists with deep expertise on model development & evaluation |
|
| Implementation & Monitoring | Individuals from IT with expertise in workflow design & implementation; clinical operational leads with experience in project deployment & change management |
|
| Regulatory | Regulatory & legal backgrounds plus expertise from IRB |
|
CURRENT STATE
Our ABCDS governance framework has been in effect since January 2021; as of October 2021, we have 52 models in our portfolio (Table 2). Date stamps are recorded at each step of the process to assess operational metrics. The average time from registration to triage is 1 week, while checkpoint presentation before the ABCDS Oversight committee occurs on average in 1 month. The checkpoint decision with stipulations and recommendations from the committee is rendered less than 1 week after checkpoint presentation.
Table 2.
Current portfolio distribution across ABCDS phases and clinical service lines
| ABCDS lifecycle | N | % |
|---|---|---|
| Model development | 21 | 40% |
| Medicine | 15 | |
| Obstetrics & Gynecology | 1 | |
| Pediatrics | 1 | |
| Surgery | 4 | |
| Silent evaluation | 5 | 10% |
| Medicine | 1 | |
| Obstetrics & Gynecology | 2 | |
| Surgery | 2 | |
| Effectiveness evaluation | 7 | 13% |
| Medicine | 5 | |
| Pediatrics | 1 | |
| Surgery | 1 | |
| Monitoring | 19 | 37% |
| Medicine | 11 | |
| Obstetrics & Gynecology | 2 | |
| Radiology | 1 | |
| Surgery | 5 | |
| Total | 52 |
FUTURE DIRECTIONS & CONCLUSIONS
As our governance process matures, we expect to address numerous emerging challenges. For example, bias throughout the AI/ML lifecycle has become an important topic, as high-profile failures19,20 illustrate pitfalls in the field. We are currently in the early stages of embedding researchers specializing in equity issues within the ABCDS governance to develop and operationalize guiding principles of algorithmic fairness.
Developing, validating, deploying, and maintaining models require considerable human and technological resources. Although it is tempting to focus the majority of resources on model development, a positive impact requires close attention to technical and workflow integration, as well as maintenance and continuous monitoring. We are currently determining the necessary skills and technological resources to create a standardized approach and infrastructure to support integration, maintenance, and monitoring of models in production. We also plan to quantify the return on investment and return on health for models that demonstrate clinical impact.
Striking the right balance between promoting innovation and ensuring quality and safety is a challenging task. We believe our framework allows models to be deployed, tested, and validated using patient-focused clinical outcomes.
The methodological and regulatory landscape of AI/ML in healthcare has evolved quickly. By presenting our approach to ensure high-quality, safe, and effective algorithms are deployed at the bedside, we hope to accelerate the translation of data science into clinical impact.
AUTHOR CONTRIBUTIONS
AB, NE, BG, EH, EP, and MP designed the framework, process, and wrote the manuscript. BG provided the example material. AY, EJ, CO, AP, SE, KY, and SB provided critical feedback and helped shape manuscript.
SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.
Supplementary Material
ACKNOWLEDGMENTS
The authors wish to acknowledge the work of David Gallagher, MD and Jonathan McCall, MS.
CONFLICT OF INTEREST STATEMENT
None declared.
Contributor Information
Armando D Bedoya, Department of Medicine, Duke University, Durham, North Carolina, USA; Duke University Health System, Durham, North Carolina, USA.
Nicoleta J Economou-Zavlanos, Duke University School of Medicine, Durham, North Carolina, USA.
Benjamin A Goldstein, Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA.
Allison Young, Duke University School of Medicine, Durham, North Carolina, USA.
J Eric Jelovsek, Department of Obstetrics and Gynecology, Duke University, Durham, North Carolina, USA.
Cara O’Brien, Department of Medicine, Duke University, Durham, North Carolina, USA; Duke University Health System, Durham, North Carolina, USA.
Amanda B Parrish, Duke University School of Medicine, Durham, North Carolina, USA.
Scott Elengold, Office of Counsel, Duke University, Durham, North Carolina, USA.
Kay Lytle, Duke University Health System, Durham, North Carolina, USA.
Suresh Balu, Duke Institute for Health Innovation, Durham, North Carolina, USA.
Erich Huang, Department of Medicine, Duke University, Durham, North Carolina, USA; Duke University Health System, Durham, North Carolina, USA.
Eric G Poon, Department of Medicine, Duke University, Durham, North Carolina, USA; Duke University Health System, Durham, North Carolina, USA; Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA.
Michael J Pencina, Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA; Duke AI Health, Duke University School of Medicine, Durham, North Carolina, USA.
Data Availability
The data underlying this article will be shared on reasonable request to the corresponding author.
REFERENCES
- 1. Watson J, Hutyra CA, Clancy SM, et al. Overcoming barriers to the adoption and implementation of predictive modeling and machine learning in clinical care: what can we learn from US academic medical centers? JAMIA Open 2020; 3 (2): 167–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.U.S Food and Drug Administration. Digital Health Center of Excellence. Artificial Intelligence and Machine Learning in Software as a Medical Device. 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device Accessed December 30, 2021.
- 3. Chen JH, Asch SM.. Machine learning and prediction in medicine—beyond the peak of inflated expectations. N Engl J Med 2017; 376 (26): 2507–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med 2021; 181 (8): 1065–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Bedoya AD, Clement ME, Phelan M, Steorts RC, O'Brien C, Goldstein BA.. Minimal impact of implemented early warning score and best practice alert for patient deterioration. Crit Care Med 2019; 47 (1): 49–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Xiao C, Choi E, Sun J.. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J Am Med Inform Assoc 2018; 25 (10): 1419–28. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.U.S. Food and Drug Administration website. The Device Development Process. 2018. https://www.fda.gov/patients/learn-about-drug-and-device-approvals/device-development-process Accessed December 30, 2021.
- 8.24748-3-2020—ISO/IEC/IEEE International Standard—Systems and software engineering—Life cycle management—Part 3: Guidelines for the application of ISO/IEC/IEEE 12207 (software life cycle processes); 2020.
- 9. Cannon D. ITIL Service Strategy. 2nd ed. London: The Stationery Office; 2011. [Google Scholar]
- 10. Pyzdek T. The Six Sigma Handbook. 3rd ed. New York: McGraw Hill; 2014. [Google Scholar]
- 11. Sendak MP, Gao M, Brajer N, Balu S.. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit Med 2020; 3: 41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Stoyanovich J, Howe B.. Nutritional labels for data and models. In: Quarterly Bulletin of the Computer Society of the IEEE Technical Committee on Data Engineering. Vol. 42.Washington, DC: The Committee; 2019. https://par.nsf.gov/servlets/purl/10176629 Accessed December 30, 2021. [Google Scholar]
- 13. Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency; New York, NY: Association for Computing Machinery; January 2019: 220–9.
- 14. Seneviratne MG, Shah NH, Chu L.. Bridging the implementation gap of machine learning in healthcare. BMJ Innov 2020; 6 (2): 45–7. [Google Scholar]
- 15. Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D.. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019; 17 (1): 195. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Kawamanto K, Flynn MC, Kukhareva P, et al. A pragmatic guide to establishing clinical decision support governance and addressing decision support fatigue: a case study. AMIA Annu Symp Proc 2018; 2018: 624–33. [PMC free article] [PubMed] [Google Scholar]
- 17. Vyas DA, Eisenstein LG, Jones DS.. Hidden in plain sight—reconsidering the use of race correction in clinical algorithms. N Engl J Med 2020; 383 (9): 874–82. [DOI] [PubMed] [Google Scholar]
- 18. Yan M, Pencina MJ, Boulware LE, Goldstein BA.. Observability and its impact on differential bias for clinical prediction models. J Am Med Inform Assoc 2022; 29 (5): 937–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Obermeyer Z, Powers B, Vogeli C, Mullainathan S.. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366 (6464): 447–53. [DOI] [PubMed] [Google Scholar]
- 20. Pencina MJ, Goldstein BA, D'Agostino RB.. Prediction models—development, evaluation, and clinical application. N Engl J Med 2020; 382 (17): 1583–6. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data underlying this article will be shared on reasonable request to the corresponding author.