# Data-driven multidimensional modeling of nonverbal communication in typical and atypical development

> **NIH NIH R01** · GEORGIA INSTITUTE OF TECHNOLOGY · 2020 · $341,605

## Abstract

ABSTRACT
This proposal will develop innovative technology for data-driven, multimodal characterization of nonverbal
communication (NVC) in typical and atypical development. Prior research has provided qualitative descriptions
of the development of children's use of gaze and gesture to regulate social interactions, but there are no
objective, automated tools for measuring NVC behaviors, nor computational models to explain their
coordination and timing in social interactions. This proposal will apply advanced probabilistic modeling
techniques from machine learning and data mining to a rich corpus of children's behavior, including automated
measures of children's posture, head pose, gaze direction, arm movements, and hand configurations derived
from color and depth cameras and accelerometers. By automatically learning probabilistic latent variable
models from movement data, we will obtain compact, data-driven descriptions of NVC and its coordination in
children with autism, children with developmental delays without autism, and typically developing children (Aim
1). We will validate our models by demonstrating their ability to predict children's behavior, including diagnostic
group and one-year language outcomes (Aim 2). We will test whether novel NVC patterns can be uncovered
with bottom-up clustering of motor movement data (Aim 3). We predict our models will have greater
explanatory and predictive power compared to current measures of NVC, which are typically human-coded
behaviors that are descriptive, but rely on a-priori definitions of higher level behaviors.
The models we develop will capture the fine-grained structure, coordination, and timing of NVC behaviors
during social interactions, and thus allow us to characterize these behaviors with an unprecedented level of
detail. Because interventions for young children with ASD target NVC skills, our automated measurement tools
will provide clinicians with powerful new tools to assess the extent to which these treatments are efficacious. In
addition, automated tools for dense measurement of fine-grained changes in NVC would enable clinicians to
assess profiles of strengths and weaknesses for purposes of treatment planning, to dynamically tailor
treatment to clients' changing abilities, and ultimately to accurately capture whether treatment is working.
Finally, the measurement capabilities will provide researchers with a novel, cost-effective approach to analyze
video recordings, at a scale that is not currently feasible due to a reliance on human coding.

## Key facts

- **NIH application ID:** 9929027
- **Project number:** 5R01MH114999-03
- **Recipient organization:** GEORGIA INSTITUTE OF TECHNOLOGY
- **Principal Investigator:** James M. Rehg
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2020
- **Award amount:** $341,605
- **Award type:** 5
- **Project period:** 2018-08-01 → 2023-05-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/9929027

## Citation

> US National Institutes of Health, RePORTER application 9929027, Data-driven multidimensional modeling of nonverbal communication in typical and atypical development (5R01MH114999-03). Retrieved via AI Analytics 2026-05-24 from https://api.ai-analytics.org/grant/nih/9929027. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*