# Automatic Multimodal Affect Detection for Research and Clinical Use

> **NIH NIH R01** · UNIVERSITY OF PITTSBURGH AT PITTSBURGH · 2021 · $581,743

## Abstract

Project Summary
A reliable and valid automated system for quantifying human affective behavior in ecologically important
naturalistic environments would be a transformational tool for research and clinical practice. With NIMH
support (MH R01-096951), we have made fundamental progress toward this goal. In the proposed project, we
extend current capabilities in automated multimodal measurement of affective behavior (visual, acoustic, and
verbal) to develop and validate an automated system for detecting the constructs of Positive, Aggressive, and
Dysphoric behavior and component lower-level affective behaviors and verbal content. The system is based on
the manual Living in Family Environments Coding System that has yielded critical findings related to
developmental psychopathology and interpersonal processes in depression and other disorders. Two models
will be developed. One will use theoretically-derived features informed by previous research in behavioral
science and affective computing; the other empirically derived features informed by Deep Learning. The
models will be trained in three separate databases of dyadic and triadic interaction tasks from over 1300
adolescent and adult participants from the US and Australia.
Intersystem reliability with manual coding will be evaluated using k-fold cross-validation for both momentary
and session level summary scores. Differences between models and in relation to participant factors will be
tested using the general linear model. To ensure generalizability, we further will train and test between
independent databases as well. To evaluate construct validity of automated coding, we will use the ample
validity data available in the three databases to determine whether automated coding achieves the same or
better pattern of findings with respect to depression risk and development. Following procedures already in
place for sharing databases and software tools, we will design the automated systems for use by non-specialists
and make them available for research and clinical use. Achieving these goals will provide behavioral science
with powerful tools to examine basic questions in emotion, psychopathology, and interpersonal processes; and
clinicians to improve assessment and ability to track change in clinical and interpersonal functioning over time.
Relevance
For behavioral science, automated coding of affective behavior from multimodal (visual, acoustic, and verbal)
input will provide researchers with powerful tools to examine basic questions in emotion, psychopathology,
and interpersonal processes. For clinical use, automated measurement will help clinicians to assess
vulnerability and protective factors and response to treatment for a wide range of disorders. More generally,
automated measurement would contribute to advances in intelligent tutors in education, training in social
skills and persuasion in counseling, and affective computing more broadly.

## Key facts

- **NIH application ID:** 10162316
- **Project number:** 5R01MH096951-10
- **Recipient organization:** UNIVERSITY OF PITTSBURGH AT PITTSBURGH
- **Principal Investigator:** JEFFREY F COHN
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2021
- **Award amount:** $581,743
- **Award type:** 5
- **Project period:** 2012-05-01 → 2024-04-30

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10162316

## Citation

> US National Institutes of Health, RePORTER application 10162316, Automatic Multimodal Affect Detection for Research and Clinical Use (5R01MH096951-10). Retrieved via AI Analytics 2026-05-24 from https://api.ai-analytics.org/grant/nih/10162316. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*