# Neural substrates of optimal multisensory integration

> **NIH NIH R01** · UNIVERSITY OF PENNSYLVANIA · 2024 · $564,464

## Abstract

Project Summary
When communicating face-to-face, humans receive information in two sensory modalities: visual information
from the face of the talker; and auditory information from the voice of the talker. The pandemic has brought
into sharp focus the importance of audiovisual speech: mask wearing obscures the talker's face while muffling
the voice, a double whammy that hinders communication. The popularity of video conferencing software
attests to the importance placed by people on seeing a talker's face as well as hearing their voice. Although
audiovisual speech perception is very important, we know little about the neural mechanisms for this uniquely
human ability. We will remedy this gap in knowledge using most powerful techniques in human neuroscience:
computational modeling; behavioral studies; ultra-high-field (7 tesla) functional MRI (fMRI); and the
examination of epilepsy patients who have electrodes implanted in their brain for the treatment of medically
intractable epilepsy, a technique referred to as intracranial electroencephalography (iEEG). The anatomical
focus of the proposal is the posterior superior temporal sulcus/gyrus (STS/G), known since the time of
Wernicke to be important for speech perception. The behavioral focus of our proposal is the new discovery that
a classic audiovisual speech illusion, known as the McGurk effect, can produce dramatic, long-lasting changes
in auditory-only speech perception, turning a ba into a da. This phenomenon, termed fusion-induced
recalibration (FIR), provides a tool to advance computational and neural studies of speech perception. The first
aim will develop computational models of speech perception and test against behavioral data. Different models
will be fit to speech perception data before, during and after exposure to the McGurk effect. Fitted models will
be compared using held-out behavioral data. Because the models instantiate different theoretical constructs,
model comparison will determine which explanatory constructs are essential. These results will provide a solid
theoretical grounding for future studies, including those in Aims 2 and 3: searching for a neural correlate of an
unjustified construct is likely to be fruitless. The second aim will examine speech perceptin through the lens of
patterns of activity in STS/G measured with 7 tesla fMRI. We expect to observe reliable changes in STS/G
response patterns before and after exposure to the McGurk effect, reflecting modification of speech
representations (in contrast, in cortical areas driven solely by acoustic features, McGurk exposure should not
change fMRI response patterns.) The third aim will use iEEG to record broadband high-frequency activity
(BHA) from small populations of STS/G neurons with high temporal resolution. Responses to the auditory-
only component of the McGurk speech (but not control speech) are predicted to show sustained decreases after
successive blocks of audiovisual McGurk exposure, in lockstep with the pe...

## Key facts

- **NIH application ID:** 10930095
- **Project number:** 5R01NS065395-13
- **Recipient organization:** UNIVERSITY OF PENNSYLVANIA
- **Principal Investigator:** Michael S Beauchamp
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2024
- **Award amount:** $564,464
- **Award type:** 5
- **Project period:** 2010-02-01 → 2028-06-30

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10930095

## Citation

> US National Institutes of Health, RePORTER application 10930095, Neural substrates of optimal multisensory integration (5R01NS065395-13). Retrieved via AI Analytics 2026-05-24 from https://api.ai-analytics.org/grant/nih/10930095. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*