Artificial Intelligence in Critical Care

Quick Recap

ICU Leadership, Communication & Systems, Protocol 3. Unlike most protocols in this library, this topic concerns a rapidly evolving technology landscape rather than a stable disease process — treat the specific tools named here as illustrative case studies of principles that will outlast any single product, not as an endorsement or current-as-of-reading product review.

1. Definition

Artificial intelligence (AI) in critical care: machine learning (ML), deep learning, and natural language processing systems applied to ICU tasks — predominantly early warning/deterioration prediction (sepsis, cardiac arrest, respiratory failure), clinical decision support, and increasingly, ambient documentation and triage. The central problem AI is marketed to solve: conventional scoring systems (SOFA, SIRS, qSOFA) are structured but often detect organ dysfunction only after significant damage has already occurred — AI models process large volumes of real-time data (vitals, labs, trends) to detect subtler patterns earlier than a clinician manually tracking the same variables would.

2. Conceptual Framework — Internal Validation Is Not External Validation

The single most important methodological concept for evaluating any AI tool before trusting it clinically: a model that performs well on the data it was developed and tested on (internal validation) can perform dramatically worse when deployed in a different hospital, patient population, or EHR system (external validation) — this is not a rare edge case but a well-documented, recurring pattern, and the Epic Sepsis Model case study below is the canonical illustration of exactly this failure mode at large scale.

3. The Epic Sepsis Model — A Canonical, Real-World Deployment Failure

Background: the Epic Sepsis Model (ESM) is a proprietary sepsis-prediction algorithm embedded in the Epic EHR, deployed at hundreds of US hospitals based substantially on the vendor's own internal validation claims, without independent external validation prior to widespread clinical use.

Wong et al., JAMA Internal Medicine 2021 — external validation at University of Michigan, 38,455 hospitalizations: at the vendor-recommended alert threshold, the ESM had sensitivity 33%, specificity 83%, positive predictive value 12%, negative predictive value 95%, and an AUC of just 0.63 — meaningfully worse than the performance characteristics implied by the vendor's own documentation, and only modestly better than chance at discriminating patients who would develop sepsis from those who would not.

A later, independent 2023 validation at two county emergency departments found the ESM performed even worse in that specific setting: sensitivity 14.7%, specificity 95.3%, PPV 7.6% within a 6-hour prediction window — confirming this was not a one-institution anomaly, and that performance can vary substantially even between external validation sites, not just between the original developer's data and any single external site.

The alert fatigue consequence: because the model generated a high volume of alerts relative to how often those alerts corresponded to true impending sepsis, it contributed directly to alert fatigue — clinicians becoming desensitized to warnings, a well-recognized patient-safety hazard in its own right (see Section 5) that can paradoxically make clinicians slower to respond to genuinely important alerts, including from other systems, once fatigue sets in.

A genuine, worth-acknowledging methodological counterpoint: some commentators have noted the Wong et al. analysis, while methodologically rigorous on discrimination/calibration, did not fully model the clinical protocol comparison (i.e., what happens to outcomes when the model IS versus ISN'T used to guide a specific clinical response) — a fair point that pure discrimination statistics (AUC, sensitivity) don't by themselves prove or disprove clinical utility, though the poor discrimination numbers alone were concerning enough to prompt many institutions to reassess their reliance on the tool.

4. Counter-Examples — What Rigorously Validated AI Deployment Looks Like

Not every AI early-warning system has failed this way — the contrast is instructive:

• Sepsis Watch (Duke Health): a real-time deep learning alert system integrated into ED/ICU workflow since 2018, associated with a 27% reduction in sepsis deaths at the deploying institution; a 2025 multisite validation confirmed strong discrimination (AUROC 0.906-0.960) and reasonable portability across different hospital systems — a genuinely different outcome profile from the ESM, achieved through prospective validation and workflow integration rather than internal-validation-only deployment
• TREWS: a real-time alert system with a prospective multisite study showing decreased mortality, decreased organ failure, shorter length of stay, and improved antibiotic timeliness — notably, this is prospective, multisite, patient-outcome-level evidence, a substantially higher evidentiary bar than the retrospective discrimination-statistic validations more common in this space
• Sepsis ImmunoScore: received FDA marketing authorization (de novo pathway) in April 2024 as the first AI diagnostic tool specifically authorized for sepsis — a genuine regulatory landmark, reflecting a formal evidentiary review process rather than vendor self-report

The practical lesson: the difference between the ESM and Sepsis Watch/TREWS is not "AI doesn't work for sepsis prediction" — it is the difference between internal-validation-only deployment and prospective, externally-validated, workflow-integrated deployment. Both are AI; the evidentiary rigor behind each is what differed.

5. Alert Fatigue — A Real, Quantifiable Harm From Poorly-Calibrated AI

High false-positive rates from a poorly calibrated model directly translate to increased clinician cognitive load and desensitization to alerts generally — this is not a hypothetical concern but a directly observed consequence in real deployments (explicitly documented as a driver behind institutions moving away from tools like the early ESM toward alternatives). A false-positive AI alert prompting unnecessary antibiotics or invasive monitoring in a misclassified non-infectious SIRS patient is a genuine, avoidable harm, not merely an inconvenience — evaluate any AI early-warning tool's false-positive burden with the same seriousness as its sensitivity.

6. Explainability and the "Black Box" Problem

Many high-performing ML/deep learning models (particularly neural networks) do not offer a transparent, human-interpretable explanation for why a given patient was flagged — this creates genuine tension between raw predictive performance and clinician trust/safe integration into high-stakes decision-making. Explainable AI (XAI) methods are an active, growing research area specifically trying to close this gap, aiming to preserve predictive accuracy while giving clinicians an interpretable rationale for any given alert (e.g., which specific variables/trends drove the score) — clinically, a model that can say "this alert is driven by rising lactate and falling MAP over the past 2 hours" is more actionable and more trustworthy than one that outputs a bare risk score with no rationale.

7. Algorithmic Bias — A Genuine, Underaddressed Risk

Models trained on non-representative data can systematically underperform for underrepresented patient populations — this is a recognized, active barrier across the AI-in-healthcare literature generally, not specific to sepsis prediction. This directly parallels the equity finding already documented in the Breaking Bad News protocol (the ICUconnect communication-intervention trial showing significantly less benefit for Black families than White families) — the pattern of technology/intervention benefit not being uniform across populations recurs across genuinely different domains within critical care, and should be actively screened for (not assumed absent) whenever a new AI tool is being evaluated for local deployment.

8. Practical Framework — Questions to Ask Before Trusting Any AI Tool Clinically

Checklist for evaluating a new AI/ML clinical tool before institutional adoption:

9. Consultation Matrix

10. Documentation & Medicolegal Checklist

• Institutional validation process (internal vs. external, retrospective vs. prospective) documented before adopting any AI clinical tool
• Alert response/override documented per standard clinical decision-support practices — an AI alert does not replace clinical judgment or documentation of the reasoning for acting or not acting on it

11. Key Guidelines

• No single regulatory body currently mandates a universal AI clinical-tool validation standard, though FDA's de novo authorization pathway (as used for the Sepsis ImmunoScore) represents a formal review process available for tools seeking it; professional guidance (e.g., AMA's Policy on Augmented Intelligence in Health Care, and the EU's Ethics Guidelines for Trustworthy AI) emphasizes transparency, accountability, and shared decision-making as governing principles

12. Landmark Evidence

13. Controversies

• The Epic Sepsis Model episode itself remains genuinely instructive rather than fully closed: even the methodological critique of the Wong et al. paper (that it evaluated discrimination statistics rather than a full clinical-protocol comparison) is worth taking seriously — poor AUC/sensitivity numbers are concerning but don't by themselves definitively prove a tool provides zero clinical value in every possible use pattern; this protocol treats the ESM as a clear cautionary tale about validation rigor, while acknowledging the full picture of "how much clinical harm did widespread ESM deployment actually cause" is more complex than the discrimination statistics alone convey.
• The gap between AI research publication volume and rigorously validated clinical deployment remains large: a substantial majority of published sepsis-prediction AI models are evaluated only via retrospective discrimination statistics (AUC, sensitivity, specificity) rather than prospective, patient-outcome-level trials — this protocol treats the TREWS/Sepsis Watch evidentiary standard as the appropriate bar for genuine confidence, not the more common retrospective-validation-only standard still dominant in the literature.
• Algorithmic bias across patient subgroups is likely underdetected, not merely underaddressed — most published validation studies are not explicitly designed or powered to detect differential performance across race/ethnicity/sex, meaning the true extent of this problem across deployed AI tools is probably larger than currently documented, mirroring the same concern raised regarding communication-intervention equity in the Breaking Bad News protocol.

14. References

1. Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. 2021;181(8):1065-1070.
2. Habib AR, Lin AL, Grant RW. The Epic Sepsis Model falls short—the importance of external validation. JAMA Intern Med. 2021;181(8):1040-1041.
3. External validation of the Epic sepsis predictive model in 2 county emergency departments. 2024.
4. Kamran F, Tjandra D, Heiler A, et al. Evaluation of Sepsis Prediction Models Before Onset of Treatment. NEJM AI. 2024.
5. Sepsis ImmunoScore: FDA-Authorized AI/ML Tool for Sepsis Prediction: Development and Validation. NEJM AI. 2024.
6. AI in Sepsis Management: Predicting ICU Outcomes. European Society of Medicine, 2025 (Sepsis Watch, TREWS, InSight, COMPOSER summary).
7. Explainable AI Advances Sepsis Prediction in ICUs — systematic review, 2020-2025 literature.
8. Exploring the Potentials of Artificial Intelligence in Sepsis Management in the Intensive Care Unit. Crit Care Res Pract. 2025.
9. The Washington Manual of Critical Care, 4th ed, 2025 — clinical informatics and decision support chapter.

See also: Sepsis/Septic Shock (Infectious Diseases System) for the clinical management framework any sepsis-prediction alert ultimately feeds into; Breaking Bad News & Difficult Conversations (this section) for the parallel equity-of-benefit finding in a different domain; Diagnostic Error & Cognitive Bias in Critical Care (this section) for how clinicians' own alert-fatigue-driven behavior compounds AI-related risk.