CAREER: Video Understanding Like Humans Do: Empathy, Reasoning, and Privacy-Aware Learning

NSF Award Search · 01003031DB NSF RESEARCH & RELATED ACTIVIT · $518,708 · view on nsf.gov ↗

Abstract

Understanding human behavior in videos goes beyond recognizing objects or actions. It involves interpreting emotion, reasoning about context, and respecting privacy. Current artificial intelligence systems often rely on simple visual patterns, struggle to explain their decisions, and are trained on data that may include sensitive personal information. These limitations make it difficult to use them in real-world settings such as accessibility. This project addresses these challenges by developing methods for human-aligned video understanding that can interpret emotional and social context, provide clear explanations, and operate on privacy-preserving data. The outcomes will support socially responsible artificial intelligence systems, improve accessibility tools such as audio descriptions for blind and low-vision audiences, and provide open resources that advance research, education, and public understanding of artificial intelligence. This project introduces a unified, closed-loop learning framework in which tasks, evaluation, and data are jointly optimized to improve model performance. The approach integrates three components. Empathy-driven modeling captures social and emotional context. Reasoning-aware evaluation diagnoses and improves intermediate model behavior. Privacy-preserving data generation enables the creation of semantically meaningful data without exposing sensitive information. Within this framework, benchmark datasets and evaluation protocols are used to i

Key facts

NSF award ID
2541968
Awardee
Dartmouth College (NH)
SAM.gov UEI
EB8ASJBCFER9
PI
SouYoung Jin
Primary program
01003031DB NSF RESEARCH & RELATED ACTIVIT
All programs
Artificial Intelligence (AI), CAREER-Faculty Erly Career Dev, ROBUST INTELLIGENCE, EXP PROG TO STIM COMP RES
Estimated total
$518,708
Funds obligated
$297,131
Transaction type
Continuing Grant
Period
07/01/2026 → 06/30/2031