PROJECT SUMMARY/ABSTRACT The proliferation of “black box” Machine Learning (ML) models for Clinical Decision Support (CDS) has raised concerns regarding CDS interpretability, actionability and overall usability, rendering a critical need for a clear process that engages various stakeholders including both developers and users in implementation planning. Our long-term goal is to formalize a process to guide health systems in planning, monitoring and evaluating CDS implementation. The overall objective for this R21 is to develop and evaluate a generalizable strategy to bring multidisciplinary stakeholders together during the CDS exploration phase to identify facilitators and barriers to implementation in their contexts. In doing so, we will use Participatory System Dynamics (PSD) modeling as a multi-component strategy to evaluate and plan implementation with stakeholders during the exploration phase of implementation, when decision-making occurs, in a way where ML-enabled CDS can be sustained over time. As such, we will focus on the upstream implementation outcomes of acceptability, appropriateness, and feasibility of ML-enabled CDS. The rationale for this project is that a process that engages diverse stakeholders in implementation planning early on will clarify commitment to implementation and potential for adoption by revealing acceptability, feasibility, and appropriateness. For this project we will focus on one particular set of ML-enabled CDS: Early Warning Scores (EWSs), used to identify decompensating patients. We plan to accomplish our overall objective by pursuing two specific aims: 1. Engage multidisciplinary stakeholders involved in EWS implementation (users, developers, implementers, owners) from two systematically varying adoption contexts to co-define common barriers and facilitators to key implementation outcomes of CDS acceptability, appropriateness, and feasibility using group model building scripts from the field of system dynamics and 2. Evaluate the PSD process by measuring change in commitment to adopt CDS (using measures of acceptability, appropriateness, and feasibility), eliciting feedback, and estimating intervention effort. We will obtain data via a series of group modeling sessions from stakeholders who have used CDS in different contexts, where alerts vary by target user, time, and frequency among other factors. We will employ well-defined scripts from the field of System Dynamics modeling to facilitate group discussion toward developing a shared theory about the problem of ML-enabled CDS response (Aim 1). Because implementation of any strategy requires adaptation, we will evaluate the PSD process (Aim 2) to refine and prepare for use elsewhere. This contribution is significant because EWSs are widely used across both academic and community hospitals. This contribution is innovative by using group modeling techniques for the problem of ML-enabled CDS implementation, creating both methodological and substantive findings....