# Predicting Autism and Social Functioning from Computer Vision Analyses of Motor Synchrony During Dyadic Interactions

> **NIH NIH R01** · CHILDREN'S HOSP OF PHILADELPHIA · 2020 · $774,965

## Abstract

PROJECT SUMMARY
Our innovative and fully automated approach to the analysis of social behavior addresses the pressing
need for precise and scalable measurements of the autism spectrum disorder (ASD) phenotype. Using
computer vision and machine learning methods, we have created a novel, quantitative method for fine-grained
analysis of social interactions. Our approach directly measures interpersonal motor synchrony, a construct which
we use as a lens for understanding the social interaction differences that are at the core of ASD. Significance:
Genomics and neuroimaging methods continuously evolve, providing deeper insights into the biology of ASD.
However, methods for measuring the outward manifestations of ASD have not changed substantially in decades.
ASD is fundamentally a disorder of social interaction, but current clinical tools do not directly measure observable
social interactions. Instead, they summarize global impressions of these interactions via informant report
questionnaires or observational coding schemes that typically lack the behavioral granularity needed to robustly
measure individual differences and changes across time (e.g., treatment related change). Inter-rater agreement
on questionnaires is typically modest, while the alternative “deep phenotyping” by expert clinicians is a time-
consuming and often cost-prohibitive burden to studies, especially when large samples are required (e.g., in
genomics research). Approach: To resolve these problems, our team created a novel computational framework
that leverages advances in markerless video motion capture, computer vision, and machine learning to directly
capture dyadic social interactions. This allows us to capture all behaviors observable by expert clinicians but with
exquisite digital precision and objectivity. Preliminary Data: We developed a fully automatic quantitative
assessment of interpersonal social behavior focused on features of dyadic facial motor synchrony. When applied
to videos of brief conversations between confederates and young adults with or without ASD, our assessment
predicted diagnostic status with 91% accuracy – significantly better than highly trained clinical experts assessing
the same video recordings. The set of predictive social motor synchrony features that we identified also
correlated significantly with symptom severity in the ASD group, suggesting that it can be used for both diagnostic
classification and evaluating individual differences (vital for advancing precision medicine goals). Importantly,
our findings were reproducible across samples: the same features identified in our adult analysis also predicted
diagnosis in a child sample with high accuracy. Aims. In Aim 1, we will test the specificity of our computer vision
approach by expanding comparisons to include a mixed psychiatric disorder group; Aim 2 will test dyadic
synchrony in other body movements, and Aim 3 will define associations between interpersonal motor synchrony
and dimensional a...

## Key facts

- **NIH application ID:** 9842013
- **Project number:** 5R01MH118327-02
- **Recipient organization:** CHILDREN'S HOSP OF PHILADELPHIA
- **Principal Investigator:** ROBERT Thomas SCHULTZ
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2020
- **Award amount:** $774,965
- **Award type:** 5
- **Project period:** 2019-01-01 → 2023-11-30

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/9842013

## Citation

> US National Institutes of Health, RePORTER application 9842013, Predicting Autism and Social Functioning from Computer Vision Analyses of Motor Synchrony During Dyadic Interactions (5R01MH118327-02). Retrieved via AI Analytics 2026-05-22 from https://api.ai-analytics.org/grant/nih/9842013. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*