# Project 3: Multispectral Imaging and Robotic Bronchoscopy Devices for Precision Lung Tumor Detection

> **NIH NIH P01** · UNIVERSITY OF PENNSYLVANIA · 2024 · $325,703

## Abstract

Project 3: Multispectral Imaging and Robotic Bronchoscopy Devices for Precision Lung Tumor
Detection
Project Summary
One of the most significant advances in surgery has been the development of minimally invasive devices and
technologies for endoscopic, laparoscopic, and robotic assisted surgical and diagnostic procedures. In comparison
with conventional or “open” surgery, a major problem in minimally invasive surgery is however that the surgeon
loses his or her sensory/tactile feedback and cannot visualize the full field of surgical view. This problem arises
because the incisions are often so small, that surgeons cannot use their hands/fingers to directly inspect or search
for tumor nodules and lesions. Thus, there are still major technical challenges and unmet clinical needs in
developing miniaturized devices for surgical navigation and guidance under minimally invasive conditions, especially
for robotic in-body manipulation and navigation, and for real-time capturing of both anatomical and molecular
information. To address these challenges, we have assembled an academic and industrial team of senior
investigators with interdisciplinary expertise in optoelectronic imaging chip design, miniaturized instrumentation,
software engineering, and medical robotics. The overall goal is to integrate multispectral sensors, flexible robotic
manipulation, and clinically applicable targeted tracers for image-guided biopsy and surgery of undiagnosed lung
nodules and non-small-cell lung cancer (NSCLC). Specifically, we will develop a multispectral imaging sensor
comprised of vertically stacked photodetectors with pixelated multispectral filters, with low readout noise (less than
one electron) and individually controlled exposure for both color RGB (red-green-blue) and near-infrared (NIR)
pixels in a complementary metal-oxide-semiconductor (CMOS) architecture. We will also develop hologram
wearable goggle devices for providing three-dimensional (3D) visualization and augmented reality display during
surgery. The hologram-based goggles will provide an augmented or mixed-reality display during surgical settings
because the surgeons can observe the patient with their natural eyesight while viewing a 3-D hologram display of
cancer tissue locations acquired by the bio-inspired multispectral imaging sensor. Furthermore, through an
academic-industrial partnership, we will collaborate with Johnson and Johnson to develop and incorporate near-
infrared molecular imaging into a robot-enabled flexible bronchoscope to locate malignant deposits in lung lesions
and lymph nodes. In summary, the overall rationale and strategies are to improve the sensitivity and specificity of
intraoperative tumor imaging during minimally invasive procedures by combining tumor targeted near-infrared
molecular tracers with bio-inspired multispectral imaging devices and three-dimensional (3-D) holographic displays.
The proposed work is expected to yield cutting-edge molecular imaging systems for s...

## Key facts

- **NIH application ID:** 10907445
- **Project number:** 5P01CA254859-03
- **Recipient organization:** UNIVERSITY OF PENNSYLVANIA
- **Principal Investigator:** SHUMING NIE
- **Activity code:** P01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2024
- **Award amount:** $325,703
- **Award type:** 5
- **Project period:** 2022-06-16 → 2027-05-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10907445

## Citation

> US National Institutes of Health, RePORTER application 10907445, Project 3: Multispectral Imaging and Robotic Bronchoscopy Devices for Precision Lung Tumor Detection (5P01CA254859-03). Retrieved via AI Analytics 2026-05-22 from https://api.ai-analytics.org/grant/nih/10907445. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*