# A scalable non-intrusive image annotation method using eye tracking for training deep learning models in radiology

> **NIH NIH R21** · UTAH STATE HIGHER EDUCATION SYSTEM--UNIVERSITY OF UTAH · 2021 · $152,500

## Abstract

PROJECT SUMMARY/ABSTRACT
Machine learning (ML) and artiﬁcial intelligence have recently emerged as powerful techniques that can augment
radiology interpretations and show promise for improving patient outcomes. One of the ways for ML to make a
signiﬁcant impact on health care is in improving the evaluation of high-volume, low-cost exams for early signs
of a wide variety of diseases.The routine chest x-ray is an ”opportunity for screening” for diseases, including
cancer, chronic obstructive pulmonary disease (COPD), pneumonia and congestive heart failure. For instance,
lung cancer is the most common cause of cancer death in the US, and is typically diagnosed at a higher stage
than most other cancers leading to low survival rates. The National Lung Screening Trial reported that low dose
computed tomography (LDCT) screening resulted in a 20% reduction in lung cancer mortality; however, few eli-
gible people actually undergo LDCT screening. Meanwhile, chest x-rays continue to be the most common form
of imaging worldwide. Improved detection from x-rays can direct patients to LDCT. COPD is another important
disease that is often under-diagnosed. People with COPD are at increased risk of lung cancer and respiratory
infections, or exacerbations, which are associated with higher morbidity and mortality. Furthermore, a chest x-ray
may show poorly-deﬁned regions of consolidation that are concerning for pneumonia. Medical attention is re-
quired to treat an infection or evaluate for other cause. More generally, methods to detect disease on chest x-rays
can be extended to cardiomegaly, pulmonary edema and pleural effusions which are seen in congestive heart
failure. Improved detection can direct patients to medical care. Convolutional neural networks (CNN), a highly
successful ML model, can be applied to chest x-ray images. However, few annotated medical datasets exist
that are sufﬁciently large to train CNNs. Furthermore, it has been shown that bounding boxes used to localize
disease can be incorporated into the training of CNNs and signiﬁcantly increase their accuracy. Unfortunately,
medical datasets with such localized annotations are even rarer and are very limited in the number of cases due
to the time-consuming process of creating bounding boxes by radiologists. We propose an innovative integrated
approach using eye tracking, speech recording and novel vision and language models to create localized annota-
tions in a manner that is non-intrusive to the workﬂow of the radiologist. The novelty of our approach is in the use
of eye tracking during routine radiological reading. The challenge is to overcome the relatively ambiguous nature
of eye tracking information compared to bounding boxes which provide deﬁnitive information about abnormalities.
To address this challenge, we will also design new CNN architectures and learning algorithms that can use eye
tracking and additional information such as pupil dilation and ﬁxation duration. The proposed methodolog...

## Key facts

- **NIH application ID:** 10133070
- **Project number:** 5R21EB028367-02
- **Recipient organization:** UTAH STATE HIGHER EDUCATION SYSTEM--UNIVERSITY OF UTAH
- **Principal Investigator:** Tolga Tasdizen
- **Activity code:** R21 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2021
- **Award amount:** $152,500
- **Award type:** 5
- **Project period:** 2020-04-01 → 2023-01-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10133070

## Citation

> US National Institutes of Health, RePORTER application 10133070, A scalable non-intrusive image annotation method using eye tracking for training deep learning models in radiology (5R21EB028367-02). Retrieved via AI Analytics 2026-05-23 from https://api.ai-analytics.org/grant/nih/10133070. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*