Automating chart review for identification of infected patients using natural language processing

Introduction

Sepsis, a lethal syndrome characterized by a dysregulated host response to infection causing organ impairment, is a leading cause of mortality among hospitalized patients.^1,2 Timely and accurate recognition of infection is essential to reduce mortality from sepsis, as delayed antibiotics worsen outcomes.^3,4 It is also vital to avoid inappropriate antibiotic administration to patients without infection to prevent adverse effects and antimicrobial resistance.^5,6 Despite its importance, accurate identification of infected patients is challenging for clinicians and algorithms alike.^7–10 Methods commonly used in research have demonstrated varying levels of sensitivity, missing up to half of admissions with infection.¹¹ In contrast, manual chart review with expert annotation is the gold standard but is onerous and costly. Thus, models that accurately replicate manual chart reviews for identifying infected patients can facilitate epidemiological research and the development of data-driven clinical decision support for early sepsis detection to prevent poor health outcomes.

One promising methodology to automate chart review for identifying infection is natural language processing (NLP) of clinical documentation. During routine clinical care, providers aggregate patient symptoms, histories, vital signs, and test results to identify infection and document their assessments in their notes. Clinician experts conducting retrospective chart reviews similarly use the same electronic health record (EHR) data and notes to assess the likelihood of infection. Thus, an NLP algorithm based on these components may be able to perform similarly to chart review. While some studies have implemented NLP to identify patients with sepsis or infection, they have frequently been limited by imperfect gold standards and lack of external validation.^12–19 In contexts outside of sepsis and infection, studies attempting to replicate manual chart review have demonstrated impressive performance using clinical NLP.^20–23 However, the performance of NLP in classifying patients with infection as determined by manual chart review is unknown.

Objective

In this study, we develop and externally validate NLP machine-learning models to identify suspected infection on admission among hospitalized patients, using manual chart review by clinical experts as the reference gold standard. We additionally investigate whether models using both text and structured data outperform models relying on any single data modality. Using clinical documentation and structured data with machine-learning and NLP, our goal was to develop a computable phenotype that replicates manual chart review in determining the likelihood of infection on admission.

Methods

Results

Discussion

In this study, we developed and externally validated prediction models that accurately discriminated between patients with and without suspected infection on admission when compared against expert manual chart review as the gold standard. We found that the best-performing model was a multimodal model using features from unstructured clinical note text data and structured data, which more accurately discriminated between infected and non-infected patients than a model using only structured data. The majority of important features in the multimodal model were derived from unstructured data, highlighting its utility for replicating chart reviews in identifying patients with infection. To our knowledge, this is the first study to test NLP approaches for identifying infected hospitalized patients as determined by expert chart review, supporting future development of clinical decision support tools to detect infection early and estimates of sepsis prevalence.

Prior research has sought to develop NLP and multimodal predictive models to identify patients with sepsis as defined by several criteria. First, Sepsis-3 defines suspected infection as having a body fluid culture within 24 hours after an antibiotic or antibiotic within 72 hours after a body fluid culture, and defines sepsis as suspected infection and a 2-point increase in the Sequential Organ Failure Assessment (SOFA) score.³⁶ The Sepsis-3 definition has been used in several multimodal machine-learning models. For example, Liu et al¹³ developed a model combining GloVe embeddings and a recurrent neural network with structured features to predict Sepsis-3 definition onset early in Medical Information Mart for Intensive Care-III (MIMIC-III) data. In a similar approach, Amrollahi et al¹⁴ developed a model combining ClinicalBERT embeddings with structured features in MIMIC-III data. Second, the Rhee definition uses sustained antibiotic administration over 4 days (with the first dose being intravenous) and a blood culture order to identify serious infection, deemed sepsis if complicated by acute organ dysfunction.³⁷ Delahanty et al¹⁵ trained a gradient boosting supervised machine learning classifier to predict the Rhee target, including structured data and chief complaint keywords as features. Third, ICD codes have frequently been used, including infection, sepsis, and organ failure billing codes identified by Angus.^24,25 For example, Goh et al¹⁶ used ICD-10 codes for sepsis to train a multimodal model incorporating structured features and notes represented by topic modeling in Singapore ICU data. Consistent with many of these studies on sepsis^13,14,16 as well as research in other clinical applications,²⁹ we found that a multimodal model combining both structured and unstructured data performed better than that using any feature set alone.

Our study differs from past NLP research by using suspected infection as determined by expert chart review as the endpoint of interest, without requiring organ dysfunction. Infection is a desirable target outcome for clinical decision support to ensure early antibiotic initiation prior to patient deterioration. Additionally, chart review is a preferable gold standard because commonly used EHR-based infection definitions have varying levels of sensitivity and specificity. In our prior work from this cohort, the Sepsis-3 infection definition had the highest sensitivity (81%), followed by Angus (77%) and Rhee (52%), while Rhee was the most specific and Angus billing codes most accurately reflected infection prevalence.¹¹ In the current study, we similarly found that the Sepsis-3 criteria were more sensitive than the Rhee criteria for detecting infection on admission (70% vs 54%), whereas the Rhee criteria were more specific (97% vs 93%). Our approach has the advantage of producing a range of probabilities with different performance characteristics, so that the balance of sensitivity vs specificity can be matched to the study’s scientific question. Furthermore, our models only used data from the first 24 hours of admission, whereas the Sepsis-3 and Rhee criteria include information from several days after admission. It is likely that including additional data could improve the performance of our models and will be the focus of future work. To our knowledge, few studies have used clinical free text to develop models that predict infection, and those that did used endpoints such as ICD-9 codes,¹⁷ antibiotic mentions in documentation,¹⁹ or blood cultures with IV antibiotics¹⁸ rather than chart review. Additionally, one study from our group has used this chart review cohort to predict infection on admission using structured data.⁴⁸ The present study builds on this work by incorporating clinical free text.

While expert chart review is the gold standard, it can be expensive, time-consuming, and difficult to adapt for large-scale applications. A high-performing NLP model could expedite annotation by leveraging free text written by clinicians, potentially permitting better targeting of texts to expert annotators or, at times, automating chart review entirely. Our study demonstrates the feasibility of such an approach. Examining the most important features in the combined model shows that CUIs containing words pertaining to antibiotics, cultures, or infections exerted large influence on the model predictions. When providers believe their patients to be infected, they likely use language in their documentation which map to such medical concepts, allowing these patterns to be internalized during model training.

As other studies have done, our study maps the clinical note free text to medical concepts or CUIs and applies term-frequency inverse-document frequency weighting, which reduces the influence of overly frequent terms while highlighting those which are more informative.^23,29 However, we found that model performance in the external validation site is improved by further mapping these concepts to their preferred text unigrams and bigrams to use as features for our models. This may be beneficial because variability in commonly used terminology across sites and clinicians could result in different CUIs being assigned to similar clinical concepts. By instead representing CUIs through their standardized preferred text, our model may capture overlapping textual elements across such related concepts, improving model generalizability. Subsequent research should investigate whether using preferred text as model inputs is preferable to using CUIs in other contexts.

Our paper has several additional strengths. We included patients outside of the ICU in our sample, in contrast with many other studies, such as those solely using MIMIC-III data. While sepsis patients often are admitted to the ICU, signs of infection and sepsis frequently occur in the emergency room or on the wards.^8,49 Sepsis often presents differently in the ward than in ICU, such as increased prevalence of Gram-negative pathogens, differences in the infection site, and organ dysfunction originating in the kidneys and heart as opposed to lungs.⁵⁰ Correctly and swiftly recognizing these cases is necessary for quality patient care.

Additionally, we externally validated our algorithm in a different health system than where it was developed. This finding is reassuring given significant differences between the 2 health systems in patient demographics, comorbidities, and hospitalization characteristics, as well as possible differences in hospital protocols which could impact structured data collection and clinical documentation styles.

Our work contributes to a larger literature about computable phenotyping, in which biomedical informaticists generate criteria that can be used to query large datasets to identify members of a patient cohort of interest.⁵¹ Researchers interested in measuring sepsis prevalence, developing models for early infection detection, retroactively assessing healthcare quality, and more have reason to desire extracting a cohort of patients with infection from the EHR. As we have described, identifying patients with infection is challenging, and simple algorithms may not be sensitive and specific enough to suffice. Our multimodal algorithm leveraging clinical free text and structured EHR data serves as an alternative method of achieving this purpose.

Our study had several limitations. First, some hospitals may not be equipped to run cTAKES on all clinical documentation, due to lack of computational resources or expertise, which would prevent use of this algorithm. Second, the feature sets were limited to summary measures of structured vitals, labs, and nursing documentation values as well as CUIs and their preferred text n-grams; notably, time-varying trends in structured values and the raw text of clinical notes were not used. There are also several additional settings in cTAKES, such the sectionizer and negation that were not used in this study. Incorporating these features may improve model performance. Furthermore, while models exhibited high discrimination, they were not well-calibrated in the external validation site, indicating that predicted probabilities did not align well with actual probabilities of infection. Methods to improve calibration would be needed before these models could be used prospectively. Finally, identifying infection is a difficult task, and it is possible that chart reviews included some mistakes. However, the annotators achieved high levels of inter-annotator agreement for identifying infection.¹¹

Conclusion

We developed and externally validated prediction models leveraging clinical free text and structured EHR data to predict suspected infection on admission, using the reference standard of expert clinician review. We found that modeling CUIs using their preferred text, as opposed to the CUIs directly, improved model performance in the external validation site. We also found that a multimodal approach using CUIs and structured data together performed better than structured data alone. Our efforts have the potential to expedite manual annotation of infection cases, facilitating epidemiological sepsis research and expanding datasets for developing machine-learning models for early infection detection.

Supplementary Material

Author contributions

Afia Z. Khan (Formal analysis, Investigation, Visualization, Writing—original draft, Writing—review & editing), Kevin G. Buell (Writing—review & editing), George Karway (Writing—review & editing), Kyle Carey (Data curation, Investigation, Writing—review & editing), Nicole Dussault (Writing—review & editing), Jay Dumanian (Writing—review & editing), Sivasubramanium Bhavani (Writing—review & editing), Emily R. Gilbert (Investigation, Writing—review & editing), Christopher Winslow (Writing—review & editing), Nirav Shah (Writing—review & editing), Majid Afshar (Conceptualization, Investigation, Writing—review & editing), Dana P. Edelson (Investigation, Writing—review & editing), and Matthew M. Churpek (Conceptualization, Funding acquisition, Investigation, Supervision, Writing—review & editing)

Supplementary material

Supplementary material is available at [JAMIA Open] online.

Funding

This work was supported by the National Institutes of Health (NIH) National Institute of General Medical Sciences grant number R35 GM145330 (M.M.C.) and the National Institute on Aging grant number T32 AG 74937 (A.K.).

Conflicts of interest

Drs M.M.C. and D.P.E. are inventors on a patent for patient risk evaluation (US11410777) and receive royalties from this intellectual property from the University of Chicago. Dr D.P.E. is employed by and has an equity stake in AgileMD, which markets and distributes eCART. eCART was not used in this study.

Data availability

The data used in this study were obtained from 2 hospital systems with IRB approval and cannot be shared publicly due to regulatory and legal restrictions outlined in the data use agreements.