# Multimodal communication and cognition: The role of gesture in language processing and word learning in individuals with traumatic brain injury

> **NIH NIH F31** · VANDERBILT UNIVERSITY · 2022 · $31,575

## Abstract

PROJECT SUMMARY
Communication is multimodal, containing speech and gesture. When people talk, co-speech gestures
(spontaneous movements of the hands and arms) can visually depict information conveyed in speech but often
communicate unique information not conveyed in speech. For example, a speaker might say, “I searched for a
new recipe,” while making a typing gesture, conveying only in gesture that the speaker searched online rather
than through a cookbook. Listeners must bind linguistic information from speech and visuospatial information
from gesture to generate an integrated representation of a message. The benefits of gesture for
communication and cognition are well-documented in neurotypical individuals. For example, gesture improves
comprehension and memory for spoken information and facilitates word learning, abilities critical for academic
and vocational success. However, gesture has not yet received the same attention in clinical populations with
cognitive-communication disorders, such as traumatic brain injury (TBI). In this proposal, we examine whether
the benefits of gesture extend to individuals with TBI, or if the very nature of their deficits prevent gesture’s
facilitatory role in communication and cognition. Using a novel approach combining methods and theory from
speech-language pathology, gesture studies, psycholinguistics, and neuropsychology, we test the ability of
individuals with TBI to use gesture during multimodal language processing and word learning across three
experiments. Aim 1 (Experiments 1 and 2) investigates the ability of individuals with TBI to integrate
information from speech and gesture during multimodal language processing. Experiment 1 tests the effect of
observing a narrator’s gestures on subsequent retellings of stories to determine whether individuals with TBI
report information provided uniquely in gesture and integrate it into their representation of the stories across
time. Experiment 2 uses eye-tracking to determine if individuals with TBI can use information from gesture to
resolve referential ambiguity in a visual-world paradigm during rapid on-line language processing. Aim 2
(Experiment 3) investigates whether individuals with TBI benefit from observing and producing gesture during
word learning. Exploratory Aim 3 examines the relation between speech-gesture integration and working
memory abilities to explore individual differences in gesture processing and inform future confirmatory studies.
Studying gesture along with speech is critical for providing ecologically valid assessments of language that
more closely approximate the real-world communication contexts that characterize and enrich everyday life.
The proposed research will directly advance the study of gesture in clinical populations by providing new
insight into the ability to integrate speech and gesture in the context of multimodal language processing and
testing whether gesture can be leveraged to support new learning in individuals wi...

## Key facts

- **NIH application ID:** 10466175
- **Project number:** 1F31DC020388-01
- **Recipient organization:** VANDERBILT UNIVERSITY
- **Principal Investigator:** Sharice Clough
- **Activity code:** F31 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2022
- **Award amount:** $31,575
- **Award type:** 1
- **Project period:** 2022-01-01 → 2024-12-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10466175

## Citation

> US National Institutes of Health, RePORTER application 10466175, Multimodal communication and cognition: The role of gesture in language processing and word learning in individuals with traumatic brain injury (1F31DC020388-01). Retrieved via AI Analytics 2026-05-26 from https://api.ai-analytics.org/grant/nih/10466175. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*