Abstract
Artificial intelligence (AI) and machine learning (ML) for healthcare (ML4H) must be reproducible for reliable clinical use. We evaluate over 200 ML4H research papers and find that health compares poorly to other application areas for AI and ML, particularly concerning data and code accessibility, and propose recommendations for reproducible research.
Introduction
Science requires reproducibility, but many sub-fields of science have recently experienced a reproducibility crisis, eroding trust in processes and results and potentially influencing the rising rates of scientific retractions (1, 2). Reproducibility is also critical for machine learning research (3), whose goal is to develop algorithms to reliably solve complex tasks at scale, both with limited or no human supervision and in the context of human-in-the-loop systems. Failure of a machine learning system to consistently replicate an intended behavior in a context different from which that behavior was defined may result in dramatic consequences. These risks are particularly high in artificial intelligence (AI) and machine learning (ML) for healthcare (ML4H), where algorithmic findings are meant to directly affect human health (4). As we continue to approve more AI tools using ML4H in clinical practice, ensuring that manufacturer reported performance metrics are reproducible is of significant public interest. This is further underscored by the significant lack of randomized control trials for deep learning based systems in ML4H, and the high risk of bias that has been found in deep learning non-randomized clinical trials (5).
Unfortunately, several factors relating to the availability, quality, and consistency of clinical or biomedical data make reproducibility especially challenging in ML4H applications. In this work, we analyze the state of reproducibility in ML4H, contrasting the specific difficulties of ML4H to AI and ML more broadly, ultimately making several contributions: First, we develop a set of criteria for reproducibility tailored to ML4H applications, and designed to capture reproducibility goals more broadly. Second, we use these criteria to define several metrics geared towards quantifying the particular challenges in reproducibility faced within ML4H, and conduct a comprehensive review of the published literature to support our claims and compare ML4H to machine learning more generally. Finally, we build on this analysis by exploring promising areas of further research for reproducibility in ML4H.
Reproducibility Criteria
The common understanding of reproducibility in machine learning can be summed up as follows: A machine learning study is reproducible if readers can fully replicate the exact results reported in the paper. We will call this concept technical reproducibility, as it is centrally concerned with whether or not one can exactly reproduce the precise, technical results of a paper under identical conditions. Though intuitive, we argue technical reproducibility is actually only a small part of the goal of “reproducibility” more generally. This discrepancy has been noted historically in other domains in various ways (6,7) and is made apparent by the use of the term in the natural and social sciences, where attempted reproductions will often occur at different labs, using different equipment/staff, etc. However, to our knowledge, this discrepancy has not been explored in the context of ML4H, and discussions of reproducibility in ML4H, a subfield of machine learning where reproducibility is especially critical, have hence been limited to technical reproducibility. We argue that in order for a study to be fully reproducible, it must meet three reproducibility criteria:
Technical Reproducibility
Can results be reproduced under technically identical conditions?
Statistical Reproducibility
Can results be reproduced under statistically identical conditions?
Conceptual Reproducibility (Replicability)
Can results be reproduced under conceptually identical conditions?
Technical reproducibility refers to the ability of a result to be fully technically replicated, yielding the precise results reported in the paper. For example, this entails aspects of reproducibility related to code and dataset release. Statistical reproducibility refers to the ability of a result to hold under re-sampled conditions that may yield mildly different numerical results, but should not statistically affect the claimed result (e.g., a different set of random seeds, or train/test splits). Note this is related, but not identical, to the notion of internal validity (8) commonly used in social science research. Lastly, conceptual reproducibility, or replicability, describes how well the desired results reproduce under conditions that match the conceptual, high-level description of the purported effect. Note that, much as statistical reproducibility is much like internal validity, replicability is heavily related to external validity (8), as it describes the notion of how well the desired results reproduces under conditions that match the conceptual description of the purported effect. Note that replicability is task-definition dependent; claiming a task has a greater conceptual horizon of generalizability makes it harder to satisfy this requirement.
All three of these reproducibility criteria are central for full reproducibility: without technical reproducibility, one’s result cannot be demonstrated. Without statistical reproducibility, one’s result will not reproduce under increased sampling and the presence of real-world variance. And lastly, without replicability, one’s result does not depend on the desired properties of the data, but instead depends on potentially unobserved aspects of the data generation mechanism that, critically, may not reproduce when deployed in practice.
Under each of these lenses, ML4H differs from general machine learning domains in critical ways and presents unique challenges.
Core Reproducibility Challenges in ML4H
In this section, we illustrate both through qualitative arguments and a quantitative literature review that ML4H lags behind other subfields of machine learning on various reproducibility metrics. Our literature review procedure entailed manually extracting and annotating over 400 papers from various venues from 2017–2019, spanning ML4H, Natural Language Processing (NLP), Computer Vision (CV), and general machine learning (general ML). The full procedure used for this statistical review is detailed in the Supplementary Material, and final quantitative results on several key metrics can be seen in Figure 1, though they are also detailed in the text where appropriate. The full set of results is available at https://github.com/evidationhealth/ICLR2019.
Figure 1:
Fraction of papers satisfying certain conditions by ML field. See the Supplementary Material for detailed descriptions of the underlying data collection procedure. Note that ML4H consistently lags other subfields of machine learning on all measures of reproducibility save inclusion of proper statistical variance.
Technical Reproducibility
ML4H faces several key challenges with regards to technical reproducibility. Firstly, health data is privacy sensitive, making it difficult to release openly without either incurring risks of re-identification, or diminishing utility by applying aggressive de-identification techniques. As a result, few public datasets are available, and those that are available are used extremely frequently, leading to a risk of dataset-specific over-fitting. To this point, approximately only 55% of the ML4H papers we examined used public datasets, as compared to over 90% of both CV and NLP papers, and approximately 85% of general ML papers. In addition, we also analyze the kinds of datasets used by ML4H papers more thoroughly in the Supplementary Material.
ML4H scores even more poorly when it comes to code release, preprocessing specification, and cohort description; only approximately 20% of the papers we analyzed released their code publicly, compared to approximately 38% in CV and 48% in NLP. A previous study that reviewed the prevalence of code release in AI reported that only 6% of papers released code (3), which is lower than all of our estimates. This discrepancy may be due to sampling papers from different conferences and different time periods (2013–2016 vs. 2017–2019 for the current study). Specific prior works have also already examined the prevalence of code release and dataset cohort sub-selection within ML4H, finding that even when restricting focus to public datasets, studies often do not release code or include sufficiently informative text to enable a full technical reproduction (9). Note that code release is itself not necessarily sufficient for full technical reproducibility; even when code is released, it may not run correctly, it may exclude critical details, or it may fail to generate the results reported in the paper.
Statistical Reproducibility
To assess the state of statistical reproducibility in ML4H, we quantified how often papers describe the variance around their results (e.g., by listing both the mean and the standard deviation of a performance metric over several random splits). Interestingly, while ML4H’s rate of this is still relatively low (approximately 41%), it is higher than that of CV, NLP, or the general domain (21%, 29%, and 36%, respectively).
Though this is an encouraging sign, we feel there is still significant room for improvement. Even in other fields of machine learning, with arguably less complex data types, repeated studies have shown that published models fail to statistically reproduce when appropriate statistical procedures are implemented and fair hyperparameter search/preprocessing methods are used: in (10), researchers find that published models evaluated using the public ImageNet—a large public dataset & ML competition for CV models—test set show consistent performance drops when trained/tested on other random splits within ImageNet. Issues with statistical reproducibility may also arise due to a Continental Breakfast Included (CBI) effect, in which researchers routinely spend more effort optimizing their method rather than comparison points (e.g., baselines) (11).
Due to our inability to technically reproduce many ML4H works given dataset and code release issues, we cannot assess the extent of these issues to the same degree in ML4H. However, it seems doubtless that similar problems affect our field, perhaps to an even greater degree given that our datasets are frequently relatively small, high dimensional, sparse/irregularly sampled, and suffer from high rates of noise.
Conceptual Reproducibility (Replicability)
The critical issue in ML4H preventing replicability is the lack of multi-institution datasets in healthcare and the lack of usage of those that do exist. Whereas approximately 80% of CV studies and approximately 58% of NLP studies used multiple datasets to establish their results, only approximately 23% of ML4H studies did. Using only a single dataset is potentially dangerous, as it is known that developing ML4H models that generalize over changing care practices or data formats is challenging. In (12), researchers establish that without using manually engineered representations, ML4H models exhibit significant degradation in performance over time as care patterns evolve. These results are not surprising; health data is rife with hidden confounders, differs significantly between data collection and deployment environments, drifts over time, and further differs in structure and concept between different healthcare institutions (13).
Opportunities for Improvement
In this section, we present several practical suggestions for enhancing reproducibiltiy for ML4H from the perspective of three main ML4H stakeholders: the ML4H research community, data providers, and related journals and conferences. Figure 2 summarizes these suggestions.
Figure 2:
Summary of recommendations for different stakeholders.
Create Shared Research Resources
Data providers such as hospitals, clinical research centers, and government agencies produce vast amounts of valuable data. Unfortunately, as suggested by our literature review, few datasets are available for a wide range of researchers to explore. While this is understandable, given the difficulty of ensuring safe, appropriate release of clinical data, the high perceived value of proprietary health data, and the difficulty in designing datasets across non-inter-operable health platforms, more shared resources would be extremely helpful in enabling reproducible ML4H research. We call for more instances of large data trusts where medical institutions can anonymously pool data for researchers to use and create algorithms. Several prominent examples already exist of such resources, demonstrating these challenges are possible to navigate, including MIMIC (14), the U.K and Japan Biobanks (15,16), and eICU (17).
Integrate Multi-Institute Datasets
Multi-institute datasets enable studies to assess their ability to translate to new contexts, a critically understudied facet of ML4H research. Recent strides have been taken in this domain with the release of the eICU dataset (17), one of ML4H’s first large scale, multi-institution EHR datasets, and researchers are already analyzing how to form generalizable models using this resource (18). Lastly, Observational Health Data Sciences and Informatics (OHDSI) (19) provides a mechanism to run observational health studies across multiple institutions and countries. We encourage more collaborative efforts in this area from data providers to improve replicability.
Prospectively Collected or Directly Consented Data
Data collected as a by-product of care, then later released for research purposes (e.g., MIMIC (14)), presents serious privacy risks and contains many confounding variables. The landscape of these privacy risks and the nature of the confounding variables change (though they do not necessarily lessen) if data is instead prospectively collected directly from new, consented participants. These types of data collection regimes are logistically challenging, but are possible. The NIH’s All of Us Research Program (20), Evidation’s DiSCover Project (21), and Google’s Project Baseline (22) are examples of these approaches. Additionally, the growing use of directly consented data should be considered, enabled via systems such as the CMS Blue Button (23).
Adopt Rigorous Statistical Methods
ML4H researchers should be more rigorous in the development, refinement, and dissemination of statistical best practices (e.g., the proper procedures for model comparisons, etc.). Holding ourselves to high standards of statistical rigor, potentially including periodic statistical audits of our own statistical replicability, will help ensure we mitigate the problems other fields have found with “over-fitting via publication,” e.g. (10).
Develop New Privacy-Preserving Analysis Techniques
Technological solutions can also be employed to help mitigate privacy concerns. ML4H researchers can explore noised, fully- or partially- simulated, or encrypted datasets. In cases where data cannot be released, techniques to train distributed models without sharing data have been proposed (24). Synthetic data can be an excellent tool to help enable researchers to still meaningfully release their code with full end-to-end realizations of their pipeline. Technology for producing synthetic patient-level data also already exists (25).
Propose Pre-Registration Alternatives
In the biomedical sciences, “observational studies,” under which definition nearly all ML4H research falls, undergo intense scrutiny to ensure they are not susceptible to statistical artifacts—in particular, increasingly these studies are required to be pre-registered, meaning they are required to report their goal and planned analyses before they run any experiments in order to avoid intentional or unintentional statistical fraud, a move with both significant proponents and detractors among scientists and publishers (26, 27). Such prospective checks make unintentional statistical fraud more difficult, but are utterly absent in ML4H. A verbatim application of the pre-registration practice in use in the epidemiology community is likely an impractical solution due to the intrinsic exploratory nature of model development, and the fact that datasets used in ML4H are commonly highly multi-purpose, but other techniques, such as systematic re-release of new data and/or rotation of official train/test splits have been found to help reduce the presence of statistical errors in other fields. ML4H researchers and academic publishers should engage in serious conversation on what the best vehicle for these checks and balances are for model development.
Establish Reporting Requirements and Require Code Release
If conferences and journals required greater rates of data and runnable (perhaps through synthetic and/or sample data and the use of software containers for easy environment management) code release, or the use of additional data/code availability statements at publication, this would instill a significant pressure on the field to address some of the foundational barriers to reproducibility. Conferences and journals have the additional ability to insist on high standards for the reporting of statistical variance around results, hyperparameter search procedures, and evaluation mechanisms, each of which would help ensure that we maintain high standards of technical and statistical reproducibility. While code release may be difficult, especially in cases of intellectual property (IP) restrictions, navigating these issues, which may require a cleaner separation between research and commercial applications, is essential for improving reproducibility in this field. Note that another important issue in code release is the choice of license used for the released software; for our purposes, however, any license that enables reproduction of the stated results (which most standard licenses will) is sufficient for reproducibility, though other, also important principles may motivate specific licensing choices (e.g., the use of an open source initiative approved license).
Develop Data Standards and Reporting Standards
Collaborative efforts in developing data standards and reporting standards is another avenue for improving reproducibility. Healthcare analytics organizations have developed data standards like the OMOP standard (28) or the FHIR standard (29), but they are not commonly adopted in ML4H research. Increased use of standards would make it easier to technically and conceptually replicate ML4H studies. Similarly, when datasets are created with the intent to be used in ML4H, we need greater descriptions of their contents, potential confounders and biases, missing data prevalence and distribution, and how they were created. Increasing the use of “specs” or “datasheets” describing datasets would help allay these concerns and ensure that datasets are statistically adequate for the kinds of analyses ML4H researchers pursue—efforts in the broader machine learning community are also approaching this goal (30) and we should vigorously adopt these practices in ML4H.
Conclusion
In this work, we have framed the question of reproducibility in ML4H around three foundational lenses: technical reproducibility, statistical reproducibility, and conceptual reproducibility (replicability). In each of these areas, we argue both qualitatively and quantitatively, through a manual, extensive review of the literature, that ML4H performs worse than other machine learning fields in several reproducibility metrics we have identified. While keeping in mind the intrinsic challenges of data acquisition and use that plague the field, we highlight several areas of opportunities for the future, focused around improving access to data, expanding our trajectory of statistical rigor, and increasing the use of multi-source data to better enable replicability.
Supplementary Material
Acknowledgments
This paper benefited substantially from the help of many people. Most notably, Bret Nestor, Amy Lu, Denny Wu, Elena Sergevea, and Di Jin all helped annotate papers for our analysis.
Additionally, M.B.A.M. was funded in part by National Institutes of Health: National Institutes of Mental Health grant P50-MH106933 and a Mitacs Globalink Research Award. M.G. was funded in part by a CIFAR AI Chair at the Vector Institute, a Canada Research Council chair, Microsoft Research, and an NSERC Discovery Grant.
References
- 1.Baker M, 1,500 scientists lift the lid on reproducibility, Nature News 533, 452 (2016). [DOI] [PubMed] [Google Scholar]
- 2.Cokol M, Ozbay F, RodriguezEsteban R, Retraction rates are on the rise, EMBO reports 9, 2 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Gundersen OE, Kjensmo S, State of the art: Reproducibility in artificial intelligence, Thirty-Second AAAI Conference on Artificial Intelligence (2018). [Google Scholar]
- 4.Beam AL, Manrai AK, Ghassemi M, Challenges to the reproducibility of machine learning models in health care, JAMA (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nagendran M, et al. , Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies, bmj 368 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Drummond C, Replicability is not Reproducibility : Nor is it Good Science, Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML (Montreal, Canada, 2009). [Google Scholar]
- 7.Plesser HE, Reproducibility vs. Replicability: A Brief History of a Confused Terminology, Frontiers in Neuroinformatics 11 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Campbell DT, Relabeling internal and external validity for applied social scientists, New Directions for Program Evaluation 1986, 67 (1986). [Google Scholar]
- 9.Johnson AE, Pollard TJ, Mark RG, Reproducibility in critical care: a mortality prediction case study, Machine Learning for Healthcare Conference (2017), pp. 361–376. [Google Scholar]
- 10.Recht B, Roelofs R, Schmidt L, Shankar V, Do ImageNet Classifiers Generalize to ImageNet?, arXiv:1806.00451 [cs, stat] (2018). ArXiv: 1806.00451. [Google Scholar]
- 11.Hu Q, Greene CS, Parameter tuning is a key part of dimensionality reduction via deep variational autoencoders for single cell rna transcriptomics., PSB (World Scientific, 2019), pp. 362–373. [PMC free article] [PubMed] [Google Scholar]
- 12.Nestor B, et al. , Feature robustness in non-stationary health records: Caveats to deployable model performance in common clinical machine learning task, Proceedings of MAchine Learning Research 106: Machine Learning for Healthcare (Ann Arbor, MI, USA, 2019). [Google Scholar]
- 13.Caruana R, et al. , Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission, Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, 2015), pp. 1721–1730. [Google Scholar]
- 14.Johnson AEW, et al. , MIMIC-III, a freely accessible critical care database, Scientific Data 3, 160035 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Nagai A, et al. , Overview of the biobank japan project: study design and profile, Journal of epidemiology 27, S2 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Sudlow C, et al. , Uk biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age, PLoS medicine 12, e1001779 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Pollard TJ, et al. , The eICU Collaborative Research Database, a freely available multicenter database for critical care research, Scientific Data 5, 180178 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Johnson AEW, Pollard TJ, Naumann T, Generalizability of predictive models for intensive care unit patients, arXiv:1812.02275 [cs, stat] (Montreal, Canada, 2018). ArXiv: 1812.02275. [Google Scholar]
- 19.Hripcsak G, et al. , Observational health data sciences and informatics (ohdsi): opportunities for observational researchers, Studies in health technology and informatics 216, 574 (2015). [PMC free article] [PubMed] [Google Scholar]
- 20.National Institutes of Health, All of Us: About. [Google Scholar]
- 21.Evidation Health, Digital Signals in Chronic Pain (DiSCover Project), Clinical Trial NCT03421223, U.S. National Library of Medicine; (2018). [Google Scholar]
- 22.Verily, Why Baseline (2017). [Google Scholar]
- 23.Mohsen MO, Aziz HA, The blue button project: Engaging patients in healthcare by a click of a button, Perspectives in health information management 12 (2015). [PMC free article] [PubMed] [Google Scholar]
- 24.Vepakomma P, Gupta O, Swedish T, Raskar R, Split learning for health: Distributed deep learning without sharing raw patient data, arXiv preprint arXiv:1812.00564 (2018). [Google Scholar]
- 25.Walonoski J, et al. , Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record, Journal of the American Medical Informatics Association 25, 230 (2018). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Loder E, Groves T, MacAuley D, Registration of observational studies, BMJ 340, c950 (2010). [DOI] [PubMed] [Google Scholar]
- 27.Lash TL, Vandenbroucke JP, Commentary: Should Preregistration of Epidemiologic Study Protocols Become Compulsory? Reflections and a Counterproposal, Epidemiology 23, 184 (2012). [DOI] [PubMed] [Google Scholar]
- 28.Overhage JM, Ryan PB, Reich CG, Hartzema AG, Stang PE, Validation of a common data model for active safety surveillance research, Journal of the American Medical Informatics Association : JAMIA 19, 54 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.HL7, Introducing HL7 FHIR, Tech. rep., HL7 (2018). [Google Scholar]
- 30.Gebru T, et al. , Datasheets for Datasets, arXiv:1803.09010 [cs] (2018). ArXiv:1803.09010. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.