# Point and Listen: Augmented Reality Interfaces for the Visually Impaired

> **NIH NIH R01** · SMITH-KETTLEWELL EYE RESEARCH INSTITUTE · 2023 · $432,221

## Abstract

Point and Listen: Augmented Reality Interfaces for the Visually
Impaired
Summary
CamIO (“Camera Input-Output”) is a project designed to make physical objects, including documents,
maps and 3D objects such as architectural models, fully accessible to people who are blind or visually
impaired. It makes these objects accessible by means of an audio-based Augmented Reality (AR)
interface: the user points to a specific location on an object of interest and hears audio information
about this location read aloud (or sees an enhanced image of the selected location if the user has low
vision). Compared with other approaches to making objects accessible to people who are blind or
visually impaired, CamIO offers several advantages: (a) there is no need to modify or augment existing
objects (e.g., with braille labels or special touch-sensitive buttons), requiring only a smartphone and
optional wearable camera device; (b) CamIO is accessible even to those who are not fluent braille
readers; and (c) it permits natural exploration of the object with all fingers to enable audio-haptic
interactions with the object.
 Building on our past work on the prototype CamIO iPhone app, we plan to develop powerful
enhancements and new functionality, including finger tracking as an alternative to the use of a pointing
stylus, and support for wearable options such as the camera-enabled Envision Glasses that will offer a
far superior user interface to what is possible with a stand-alone smartphone app. New authoring
capabilities will make it easy for blind and visually impaired users and sighted assistants to create audio
labels for objects. We will also pursue two exciting applications of CamIO in depth: using CamIO to
make tactile maps fully interactive and easier to understand, and the development of interactive
installations such as a 3D scale model of a large playground that helps visually impaired playground
visitors familiarize themselves with the playground’s layout and features. Finally, people who are blind
or visually impaired – the target users of the CamIO system – will be involved in all aspects of the
proposed research, to maximize the impact of the research effort.

## Key facts

- **NIH application ID:** 10691442
- **Project number:** 5R01EY025332-06
- **Recipient organization:** SMITH-KETTLEWELL EYE RESEARCH INSTITUTE
- **Principal Investigator:** JAMES M COUGHLAN
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2023
- **Award amount:** $432,221
- **Award type:** 5
- **Project period:** 2016-03-10 → 2026-02-28

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10691442

## Citation

> US National Institutes of Health, RePORTER application 10691442, Point and Listen: Augmented Reality Interfaces for the Visually Impaired (5R01EY025332-06). Retrieved via AI Analytics 2026-05-21 from https://api.ai-analytics.org/grant/nih/10691442. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*