Derivation and Validation of Predictive Models for Early Pediatric Sepsis

Introduction

Sepsis, in which infection causes life-threatening organ dysfunction, is a leading global cause of pediatric death.¹ In the US, where more than 75 000 children are hospitalized with sepsis yearly, the in-hospital mortality is 5% to 20%.^2,3 Outcomes are worsened with delayed diagnosis and treatment, yet predictive models have not been broadly deployed successfully to improve early diagnosis of sepsis in children.^4,5,6,7

While some models have been developed to predict mortality in pediatric sepsis, there are few that address the earlier phase of prognostic identification and treatment, before organ dysfunction is present, when a model may be most helpful for early diagnosis and treatment of high-risk patients.^8,9 Existing early, pre–intensive care unit models have suboptimal test characteristics for clinical use.^10,11,12,13 Models based on multicenter datasets have not been reported that predict organ dysfunction among children in the emergency setting, in whom organ dysfunction is not already present. New diagnostic criteria for pediatric sepsis, the Phoenix Sepsis Criteria (PSC), were recently published with the authors explicitly stating that the criteria identify infection-associated life-threating organ dysfunction but do not predict sepsis in children.^14,15 Prediction is the important next step to reduce preventable harm from sepsis.

To address the need to improve early prognostic diagnosis of sepsis in children in the ED setting, this study used multicenter electronic health record (EHR) data to derive and validate predictive models to identify patients at risk for pediatric sepsis before organ dysfunction had already occurred. Specifically, the goal of this study was to develop machine learning models using EHR data from the first 4 hours after ED arrival to estimate the probability of developing sepsis with organ dysfunction in the subsequent 48 hours.

Methods

Results

There were 3 060 199 eligible ED visits with 2 323 720 meeting inclusion criteria, with 1 604 422 visits in the training cohort and 719 298 in the test cohort (Figure 1). In the training cohort, 0.35% (95% CI, 0.34-0.36) of visits met PSC sepsis criteria and 0.15% (95% CI, 0.15-0.16) met PSC shock criteria in the outcome window with similar proportions in the test set (Table 1). PSC sepsis and shock rates varied across sites (Table 1). Demographic description of the cohort by site is in eTable 2A in Supplement 1 and by training or temporal validation datasets in eTable 2B in Supplement 1 with an overall median (IQR) age of 4.7 years (1.7-10.1) and 48.6% of visits among female individuals. The median (IQR) hospital length of stay for those visits in the overall cohort resulting in admission was 2 (1-4) days, for visits meeting PSC sepsis criteria was 6 (3-11) days, and for those visits meeting PSC shock criteria was 5 (3-10) days. The overall cohort had a mortality rate of 0.015%, with visits meeting PSC sepsis criteria having mortality of 2.164%, and visits meeting PSC shock criteria with mortality of 3.307%.

Performance characteristics for each of the models predicting PSC sepsis in the temporal validation dataset were excellent, with AUROC of 0.923 (95% CI, 0.918-0.928) in the logistic regression models and 0.936 (95% CI, 0.931-0.940) in the gradient tree boosting models (Table 2). The specificity for the sepsis models with target sensitivity of 90% ranged from 0.779 to 0.807. These correspond to positive likelihood ratios (LR+) of 4.067 to 4.674. The Youden cutpoint models for sepsis resulted in sensitivities of at least 0.84 and specificities ranging from 0.854 to 0.861 with LR+ ranging from 5.788 to 6.179. At a sensitivity of 90%, the LR+ of the PSC sepsis gradient tree boosting model was 4.674 with the PPV of 1.7% compared to the observed overall prevalence of 0.37%. Over all models, the number needed to evaluate ranged from 45 to 68 (Table 2).

Model results for PSC shock are presented in Table 2. AUROCs of 0.923 (95% CI, 0.915-0.931) were observed for the logistic regression and 0.926 (95% CI, 0.918-0.934) for the gradient tree boosting models.

The receiver operating characteristic and precision-recall curves for the gradient tree boosting models for PSC sepsis and shock applied to the temporal validation dataset are depicted in Figure 2. There was minor variability in model performance by site (PSC sepsis AUROC range, 0.922 to 0.945) (Figure 2). The receiver operating characteristic and precision-recall curves for the logistic regression models applied to the temporal validation set are depicted in eFigure 2 in Supplement 1. Calibration of the gradient tree boosting and logistic regression models for the temporal validation set are shown in eFigure 3 in Supplement 1. To allow for comparison with the temporal validation model performance, model characteristics for all models in the training dataset are presented in eTable 3 in Supplement 1 with the ROC and precision-recall curves for PSC sepsis and shock in the training dataset depicted, respectively, in eFigures 4 and 5 in Supplement 1.

The top 20 features for the gradient tree boosting PSC sepsis models with and without Emergency Severity Index are presented in Figure 3 with many similar predictive features, including age-adjusted vital signs, age, and markers of medical complexity. Models excluding Emergency Severity Index triage data as features had similar performance characteristics, AUROCs, and area under the precision-recall curves (eTable 4 in Supplement 1). Proportions of missingness of features in the models, training and validation sets, and by site, are presented in eTable 5A and 5B in Supplement 1, respectively.

Assessment of fairness in PSC sepsis gradient tree boosting model performance by age, gender, race, ethnicity, language, insurance, and site demonstrated AUROC for patients with Medicaid insurance of 0.940 (95% CI:,0.934-0.946) compared to commercial payer mean of 0.920 (95% CI, 0.910-0.929). The model otherwise performed similarly across patient demographic groups (eFigure 6 in Supplement 1). In our stability assessment comparing models derived on incomplete training data, performance was generally a few percentage points inferior compared to the primary analysis of models derived on the complete training set (eTable 6 in Supplement 1). The lowest performance observed in this stability analysis was for predicting PSC shock using logistic regression (AUROC of 0.906 [95% CI, 0.895 to 0.915] for the lowest performing model in our stability analysis compared to 0.923 [95% CI, 0.915-0.931] observed in the primary analysis).

Discussion

In this cohort study, using a large dataset, including a diverse population, from 5 health systems, we developed machine learning models predicting the future development of sepsis based on EHR data collected early in the ED course, prior to the development of organ system dysfunction. Our models achieved robust AUROC measurements and meaningful likelihood ratios. The strengths of these results include the successful development of models capable of early prediction of a rare outcome with excellent performance characteristics. Notably, we meticulously excluded any children who already exhibited the outcome of interest at the time of presentation, and we deliberately avoided including markers of clinical judgment, which can introduce bias. To our knowledge, this study represents the first attempt to use robust machine learning models to predict the Phoenix Sepsis Criteria outcomes in a pediatric ED population that did not already have sepsis.^14,15

Although there have been screening tools for identification of patients with sepsis or those who may have increased sepsis risk in the ED,^29,30,31,32 there are no previously published multicenter predictive models of pediatric sepsis in the ED that exclude patients who already have met sepsis criteria within their first hours of emergency care. Most work in pediatric sepsis predicts mortality in patients with identified sepsis or provides screens for risk of sepsis without excluding patients who have already developed sepsis.^25,33,34,35 There is a distinct and important role for both prediction of sepsis and for triggers that help identify patients who already have sepsis to allow for early treatment. Our work, however, allows for rigorous prediction, using routine EHR data, in those patients who have not yet developed sepsis. This work indicates that identification of relevant data for prediction is possible using regularly collected data from the EHR. The strengths of machine learning modeling and its implementation within the EHR is gaining acceptance in medicine, especially in instances of difficult diagnosis and prognosis.^14,36 Important predictive features in our study included Emergency Severity Index category; vital signs, including oxygen saturation; age-adjusted shock index; and markers of medical complexity. This work is a step forward to achieve the goal of up-stream identification that may allow for preemptive treatment to improve outcomes.

Prior work by Scott et al^11,12 derived and validated models in 1 health system to predict septic shock in children already identified by clinicians as having risk of sepsis, using data known at the time of ED arrival and with data at 2 hours after arrival. That work included patients identified by suspected sepsis order sets and intended to risk stratify patients who did not yet have shock but in whom clinicians were concerned for sepsis. Our work, compared to these prior studies, included an undifferentiated cohort of ED patients. With our multicenter approach, the current study had strong AUROCs for prediction of sepsis and septic shock compared to prior work.

Use of machine learning for prediction in clinical care must be balanced with understanding of fairness of the models and evaluation of biases that may be introduced in the derivation of models.¹⁶ Fortunately, when assessing for model equity we found similar model performance, except for slightly improved performance in children with Medicaid insurance compared to those with private insurance. This may be because most of the cohort had public insurance and therefore the data used for derivation had majority representation of this community.

Conclusions

In conclusion, our study leveraged a large multicenter EHR database to identify cases of sepsis and septic shock occurring within 48 hours in children without sepsis in the initial 4 hours after ED presentation. We developed and validated models with high areas under the receiver operating characteristic curve and meaningful positive likelihood ratios. Limited positive predictive values underscore the difficulty in predicting the rare outcome of pediatric sepsis in the ED. Our research highlights the need for future studies that combine EHR-based models with clinical judgment to improve prediction.