# Mechanisms for Crossmodal Recovery of Degraded Speech Information in Auditory Cortex

> **NIH NIH F32** · UNIVERSITY OF MICHIGAN AT ANN ARBOR · 2020 · $65,310

## Abstract

Recent research suggests that listeners use visual speech cues to recover degraded spectrotemporal signals
in auditory speech, suggesting that visual speech may provide richer information about auditory speech signals
than current models suppose. The proposed research uses a combination of intracranial electrocorticography
(ECoG) and functional neuroimaging (fMRI) to test a novel neurobiological account of these crossmodal
enhancements. Because speech-sensitive sites along the superior temporal gyrus (STG; i.e., auditory cortex)
exhibit spatially-organized selectivity for spectrotemporal features, visual speech may improve auditory speech
perception by enhancing the distributed encoding of these features in auditory cortex. This hypothesis will be
tested by examining whether visual speech improves the ability to reconstruct auditory speech spectrograms
from multi-electrode ECoG signals in auditory cortex (Aim 1). Auditory sentence stimuli will be filtered to
selectively degrade spectral or temporal information, and the information provided by visual speech will be
indexed by improvements in the ability to reconstruct the filtered (i.e., missing) information. Spectral and
temporal restoration effects will be situated within the broader functional organization of the auditory system
using fMRI-based representational similarity analysis (Aim 2). We hypothesize that crossmodal restoration
effects observed under different filtering conditions (spectral vs. temporal) will reflect known differences in
spectrotemporal tuning across cortical subfields, cortical regions, hemispheres, and anterior-posterior position
along the STG. Finally, crossmodal effects on temporal, spectrotemporal, phonetic feature, and phoneme-level
representations of noise-degraded speech will be characterized and compared to assess their respective
contributions to crossmodal enhancements of speech coding (Aim 3). The fellowship will support the
applicant’s development towards becoming a world-class independent researcher by providing training in the
techniques best-suited to test neural hypotheses generated on the basis of his graduate research using
psychophysics and digital signals processing. In addition to personalized training in these techniques, the
sponsoring team and institution will provide access to a collaborative network for ECoG data collection, funds
for neuroimaging, and multiple opportunities for ongoing training in the design and optimization of ECoG and
neuroimaging experiments. This training will support the applicant’s long-term goals of using multiple
state-of-the-art methods to understand the neural mechanisms of crossmodal facilitations, with the goal of
informing the design of perceptual training regimes and prosthetic devices for the treatment of hearing deficits.
By understanding the brain’s internal mechanisms for inter-areal sensory enhancement, the candidate aims to
facilitate the design of biologically-inspired interventions to enhance sensory coding...

## Key facts

- **NIH application ID:** 9994727
- **Project number:** 5F32DC018199-02
- **Recipient organization:** UNIVERSITY OF MICHIGAN AT ANN ARBOR
- **Principal Investigator:** John Plass
- **Activity code:** F32 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2020
- **Award amount:** $65,310
- **Award type:** 5
- **Project period:** 2019-09-01 → 2022-08-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/9994727

## Citation

> US National Institutes of Health, RePORTER application 9994727, Mechanisms for Crossmodal Recovery of Degraded Speech Information in Auditory Cortex (5F32DC018199-02). Retrieved via AI Analytics 2026-05-23 from https://api.ai-analytics.org/grant/nih/9994727. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*