# Computational and brain predictors of emotion cue integration

> **NIH NIH R01** · STANFORD UNIVERSITY · 2020 · $451,783

## Abstract

The purpose of this project is to develop computational and brain-based models of emotion cue integration:
people’s inferences about others’ emotions based on dynamic, multimodal cues. Observers often decide how
targets feel based on cues such as facial expressions, prosody, and language. Such inferences scaffold
healthy social interaction, and abnormal inference both marks and exacerbates social deficits in numerous
psychiatric disorders. Psychologists and neuroscientists have studied emotion inference for decades, but the
vast majority of this work employs simplified social cues, such as vignettes or static images of faces. By
contrast, “real world” emotion cues are complex, dynamic, and multimodal. Cue integration—inference based
on naturalistic emotion information—likely differs from simpler inference at cognitive and neural levels, but this
phenomenon remains poorly understood. This means that scientists lack a clear model of how observers
adaptively process complex emotion cues, and how that processing goes awry in mental illness. Especially
lacking are mechanistic models that can describe the computations and brain processes involved in cue
integration with sufficient precision to predict inference in new cases, observers, and samples. This project will
merge tools from social psychology, computer science, and neuroscience to generate a novel and
rigorous model of emotion cue integration. We have demonstrated that in the face of complex emotion
cues, observers dynamically “weight” cues from each modality (e.g., visual, linguistic) over time, a process that
(i) tracks shifts in brain activity and connectivity; and (ii) can be captured using Bayesian models. Here, we will
expand this work in several ways. First, we will develop precise computational tools to isolate features of
emotion cues—such as facial movements, prosody, and linguistic sentiment—that track observers’ use of each
cue modality during integration. Second, we will develop multi-region “signatures” of brain activity and
connectivity that track emotion inference in each modality. We will use these signatures in conjunction with
machine learning to predict unimodal emotion inference and cue integration in new observers and samples,
based on brain data alone. Third, we will explore the context-dependence of naturalistic emotion inference by
testing whether reinforcement learning can bias observers’ cue integration and accompanying brain signatures.
Finally, we will model computational and neural abnormalities associated with cue integration in patients with
Major Depressive Disorder and Bipolar Disorder. At the level of basic science, these data will generate a
fundamentally new—and more naturalistic—approach to the neuroscience of emotion inference. The
computational and brain metrics we produce will also be made publically available to facilitate the open and
cumulative study of emotion inference across labs. At a translational level, we will provide a mechanistic, rich
accoun...

## Key facts

- **NIH application ID:** 9923725
- **Project number:** 5R01MH112560-04
- **Recipient organization:** STANFORD UNIVERSITY
- **Principal Investigator:** Jamil Zaki
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2020
- **Award amount:** $451,783
- **Award type:** 5
- **Project period:** 2017-05-19 → 2022-02-28

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/9923725

## Citation

> US National Institutes of Health, RePORTER application 9923725, Computational and brain predictors of emotion cue integration (5R01MH112560-04). Retrieved via AI Analytics 2026-05-23 from https://api.ai-analytics.org/grant/nih/9923725. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*