Abstract Significance: We propose to evaluate the performance of HindSight in a randomized controlled trial (RCT). HindSight is a novel encoding software designed to optimize alerts for sepsis prediction and detection. HindSight identifies clinicians’ sepsis-related decisions in the electronic health records of former patients and then uses these events to supply InSight with labeled examples of true positive sepsis cases. In our retrospective work, we have shown that HindSight enables InSight to adapt to the idiosyncrasies of real-world clinical deployment by successfully reducing false and irrelevant alarms, without human supervision. The goal of this project is to demonstrate that the retrospective success of HindSight can be successfully translated to live clinical environments. Research Question: To what extent can a machine-learning-based labeler, which has retrospectively learned to autonomously label sepsis cases according to a clinician-labeled sepsis gold standard, successfully reduce false alerts in a prospective RCT? Will this tool perform more successfully than a sepsis CDS tool that is not designed to autonomously reproduce clinician identification of sepsis? Prior Work: In our Phase I work, HindSight achieved an AUROC of 0.899, 0.831 and 0.877 for clinician sepsis evaluation, treatment, and onset, respectively. By using an online learning algorithm to incorporate HindSight-labeled data into the InSight predictor, we showed that the online-trained InSight can adapt to the HindSight-labeled data and outperform both baseline and periodically re-trained versions of InSight (p < 0.05). Specific Aims: To prospectively validate HindSight’s performance on real-time patient data streams in four diverse hospitals non-interventionally (Aim 1); and to evaluate the effect of the tool in a prospective, interventional RCT (Aim 2). Methods: HindSight will be evaluated in the background at four academic and community hospitals. Following any necessary algorithm optimization arising from live hospital validation, we will perform an RCT to evaluate reductions in false alerts from InSight trained on HindSight sepsis labels (experimental arm), compared to InSight trained on gold standard Sepsis-3 labels (control arm). The primary outcome measure of interest will be false alert reduction. Successful completion of Aim 1 will be demonstrated by a positive predictive value (PPV) in a live clinical setting for which the lower bound of the 95% confidence interval meets or exceeds the benchmark from prior retrospective studies. Meeting the retrospective PPV benchmark indicates that prospective CDS quality reflects retrospective CDS quality, and is sufficiently high to reduce alarm fatigue and improve clinical utility. Success of Aim 2 is contingent upon achieving a 15% relative reduction in false alerts when comparing between the two treatment arms (p < 0.05; Fisher’s Exact Test). Future Directions: The clinical validation of HindSight’s impact on reducing fa...