# Point and Listen: Augmented Reality Interfaces for the Visually Impaired

> **NIH NIH R01** · SMITH-KETTLEWELL EYE RESEARCH INSTITUTE · 2023 · $200,350

## Abstract

“Point and Listen: Augmented Reality Interfaces for the Visually Impaired”
Project Summary
This is an Administrative Supplement to Enhance Software Tools for Open Science application for the parent
R01 grant, “Point and Listen: Augmented Reality Interfaces for the Visually Impaired.” The focus of the parent
grant is the CamIO (“Camera Input-Output”) project, intended to make physical objects, including documents,
maps and 3D objects such as architectural models and appliances, fully accessible to people who are blind or
visually impaired (BVI). It makes these objects accessible by means of an audio-based Augmented Reality
(AR) interface: the user points to a specific location on an object of interest and hears an audio label about this
location read aloud.
Substantial progress has been made on the parent grant, including a nearly completed permanent installation
of CamIO at the Magical Bridge Playground in Palo Alto to make a 3D bronze map of the playground fully
accessible to BVI visitors. However, the project would benefit from wider adoption of, and experimentation with,
CamIO’s AR approach by other researchers and developers in the accessibility community. To this end, we
propose a Supplement project to enhance the CamIO software to lower the barriers to researchers who wish to
incorporate the software in their own projects or experiment with it. Specifically, we plan two enhancements to
the CamIO software: (a) restructure the software with the goal of making it easier for others to incorporate in
their own accessibility projects; and (b) create SKIstream, a rapid prototyping and debugging tool for use with
CamIO and similar accessibility projects.
With these enhancements to the CamIO software, we envisage institutions such as museums, medical
facilities and universities developing their own custom versions of CamIO to make 3D maps and other models
fully accessible. We note that CamIO requires no physical modifications to make a model accessible, provided
a camera can view the surface of the model that users will interact with; this means that virtually any 3D model
can be made accessible. Other researchers can experiment with modifications to the CamIO interface and
conduct user studies to evaluate the usability and effectiveness of these modifications. Finally, since the
enhanced CamIO software and SKIstream tool will be shared freely as open source code, other researchers
can contribute their own enhancements to the code, including improvements and entirely new functionality.

## Key facts

- **NIH application ID:** 10839155
- **Project number:** 3R01EY025332-06S1
- **Recipient organization:** SMITH-KETTLEWELL EYE RESEARCH INSTITUTE
- **Principal Investigator:** JAMES M COUGHLAN
- **Activity code:** R01 (R01, R21, SBIR, etc.)
- **Funding institute:** NIH
- **Fiscal year:** 2023
- **Award amount:** $200,350
- **Award type:** 3
- **Project period:** 2016-03-10 → 2026-02-28

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10839155

## Citation

> US National Institutes of Health, RePORTER application 10839155, Point and Listen: Augmented Reality Interfaces for the Visually Impaired (3R01EY025332-06S1). Retrieved via AI Analytics 2026-05-24 from https://api.ai-analytics.org/grant/nih/10839155. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*