CAREER: Addressing Hallucinations for Trustworthy Large-scale Multi-Modality Learning

NSF Award Search · 01002829DB NSF RESEARCH & RELATED ACTIVIT · $499,728 · view on nsf.gov ↗

Abstract

The rapid rise of large-scale multimodal models (LMMs) has promoted their advancement in numerous fields, including social media analysis and healthcare. However, while LMMs have shown outstanding performance, they can produce misleading outputs because these models do not know what they do not know, raising concerns about their reliability. The inaccurate, irrelevant, or unintelligible output produced by LMMs is often called hallucinations. Addressing the hallucination problem in LMMs is an important topic in the next five years since it will improve their trustworthiness and impact in many other fields. This project will develop novel hallucination mitigation approaches to improve the trustworthiness and applications of LMMs in healthcare, including sample use cases of tobacco advertisement prevention and autism behavior prediction. Outcomes from the research will impact the field by providing a foundational and practical study needed for future research. It will also train students to conduct and use research to improve community health and mental health outcomes. There are three primary factors leading to hallucinations in LMMs. First, it is widely understood that biased data distribution causes significant challenges in data-driven responsible AI approaches. However, biased distribution influence on predictions is also a leading cause of hallucinations in LMMs. Second, the misalignment between input modalities could result in the LMMs overconfidently relying on a part

Key facts

NSF award ID
2442295
Awardee
University of Arkansas (AR)
SAM.gov UEI
MECEHTM8DB17
PI
Khoa Luu
Primary program
01002829DB NSF RESEARCH & RELATED ACTIVIT
All programs
CAREER-Faculty Erly Career Dev, INFO INTEGRATION & INFORMATICS, EXP PROG TO STIM COMP RES
Estimated total
$499,728
Funds obligated
$305,002
Transaction type
Continuing Grant
Period
07/01/2025 → 06/30/2030