# Autonomous Gaze Coaching for Virtual Reality Telesurgical Simulators

> **NIH NIH R21** · CASE WESTERN RESERVE UNIVERSITY · 2024 · $222,813

## Abstract

Project Summary
The proposed project aims to develop the first-ever data-driven autonomous gaze coach for robot-assisted
minimally invasive surgical (RAMIS) virtual reality simulation training. A surgeon’s skill level is correlated with
patient outcomes. It is known that expert surgeons adopt different gaze patterns from novices, and that gaze
training is a proven, but labor-intensive coaching method that improves a trainee surgeon’s surgical skills,
especially under stress. Our proposed coaching system will increase the accessibility of high-quality gaze
coaching, even in areas where there is a shortage of expert instructors, and will free up valuable time for surgical
experts for clinical and higher-order instructional work. The acceleration of skill acquisition, will, in turn, decrease
the time and cost for training competent surgeons across the country.
The project will comprise three specific aims. In Aim 1 we collect gaze and performance data from experts and
novices in six virtual reality RAMIS drills. We will analyze the data to find good candidate drills in which to develop
our autonomous gaze coaches by identifying those in which expert performance and gaze patterns differ most
from those of novices. In Aim 2, we use the data collected in Aim 1 to train expert gaze-synthesizing neural
networks that can generate a fixation point or a gaze map given surgical video streams. We validate the networks’
abilities to replicate expert gaze using a held-out test set. In Aim 3 we develop approaches to display the gaze
synthesized by the neural networks from Aim 2 and test the effectiveness of our gaze coaching system to improve
the surgical skills of novices. Outcome skill measures of this user performance study are the surgical skill
measures such as economy of motion and the completion time, and task-specific penalties. These include
instrument and apparatus collisions, dropping of apparatus, instruments out-of-view, durations of excessive force
application, wrong energy application, dissection outside of defined zones, and repeated needle pierces.
Outcome gaze measures include the increase in the amount of overlap between the novice gaze and expert
gaze after training, and the increase in time spent focusing on expert-defined areas of interest in the surgical
scene. The results of this development and validation of the system in virtual reality simulation will inform our
long-term goal of developing autonomous gaze coaching systems for use in more complex clinical-like training
scenarios, such as cadaver or porcine models.

## Key facts

- **NIH application ID:** 10952931
- **Project number:** 1R21EB036161-01
- **Recipient organization:** CASE WESTERN RESERVE UNIVERSITY
- **Principal Investigator:** Zonghe Chua
- **Activity code:** R21 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2024
- **Award amount:** $222,813
- **Award type:** 1
- **Project period:** 2024-06-01 → 2027-03-31

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10952931

## Citation

> US National Institutes of Health, RePORTER application 10952931, Autonomous Gaze Coaching for Virtual Reality Telesurgical Simulators (1R21EB036161-01). Retrieved via AI Analytics 2026-05-25 from https://api.ai-analytics.org/grant/nih/10952931. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*