CAREER: Democratizing the Pretraining of Vision Foundation Models: A Developmentally Plausible Framework

NSF Award Search · 01002930DB NSF RESEARCH & RELATED ACTIVIT · $599,999 · view on nsf.gov ↗

Abstract

Vision foundation models (VFMs) are artificial intelligence systems for "all-purpose" understanding of images and videos. They are currently extremely expensive to create. This high cost restricts their creation to a few highly resourced institutions and leaves independent researchers and the public unable to fully explore how these systems learn. This project seeks to democratize this research by creating a highly efficient training method inspired by how human infants learn. A human child acquires foundational visual skills from a limited number of waking hours compared to the massive amount of data used by current VFMs. By using longitudinal video and audio recorded from the viewpoint of infants, this project develops a training process that is affordable for university budgets. Innovating and understanding how to train these systems efficiently using this infant-inspired approach will increase accessibility to artificial intelligence research for the broader public. Furthermore, the project provides unique educational opportunities for students and offers insights that can be transferred to specialized industries, such as medical imaging and vocational training, where data is often limited. Expanding community involvement in building these models will ultimately promote artificial intelligence safety, enhance transparency, and build public trust. The technical goal of this project is to formalize a developmentally plausible, data-efficient pretraining framework for VFMs. First, the team of researchers will establish a core framework by curating longitudinal, egocentric audiovisual recordings of human infants and designing a suite of evaluation benchmarks strictly aligned with early cognitive milestones. Second, the project bridges inherent sensory and temporal gaps in the recordings. This involves employing model ensembling to simulate tactile and gustatory senses from audiovisual cues and utilizing a meta-learning formulation to optimally mix heterogeneous d

Key facts

NSF award ID
2540851
Awardee
Trustees of Boston University (MA)
SAM.gov UEI
THL6A6JLE1S7
PI
Boqing Gong
Primary program
01002930DB NSF RESEARCH & RELATED ACTIVIT
All programs
Artificial Intelligence (AI), CAREER-Faculty Erly Career Dev, ROBUST INTELLIGENCE
Estimated total
$599,999
Funds obligated
$371,191
Transaction type
Continuing Grant
Period
09/01/2026 → 08/31/2031