# Development of Vision-Guided Shared Control for Assistive Robotic Manipulators

> **NIH VA I01** · VETERANS HEALTH ADMINISTRATION · 2022 · —

## Abstract

Veterans who use powered mobility devices including those with high-level spinal cord injury (SCI),
amyotrophic lateral sclerosis (ALS), and multiple sclerosis (MS) often experience serious upper extremity
impairments. In addition, many older Veterans and Veterans with hemiplegia may also suffer from functional
loss in their upper extremities. Management and care of upper extremity impairments often involve a range of
assistive solutions. However, product availability and technological advancement for manipulation assistance
fall far behind those for mobility. Many of these individuals, despite their independent mobility, cannot reach for
a glass of water, make a simple meal, and pick up a tooth brush. They still require assistance from a personal
caregiver for essential activities of daily living (ADLs) involving reaching and object handling/manipulation. With
the rapid advancement of robotics technology, assistive robotic manipulators (ARMs) emerge as a viable
solution for assisting Veterans with upper extremity impairments to complete daily tasks involving reaching,
object handling, and manipulation. ARMs are often equipped with many degrees of freedom (DOF), but users
cannot control all of the DOFs at the same time with a conventional joystick, and need to switch modes quite
often to complete even simple manipulation tasks, especially when an ARM gets close to the target and need
to be aligned appropriately for manipulation. Thus existing ARMs suffer from the lack of efficiency and
effectiveness especially in an unstructured environment.
The goal of this project is to combine vision-guided shared (VGS) control with two types of environment
modifications to address the effectiveness and efficiency of ARMs for real-world use. The two types of
environment modifications include using commercial or custom adaptive tools (e.g., a holder that can hold a
bottle or jar so an ARM can open it), and adding fiducial markers (similar to QR codes) to objects or adaptive
tools to make vision-based tracking robust and reliable for real-world applications. Built upon the environment
modifications, the VGS control will allow a user to initiate any task by moving an ARM close to a tagged object,
and the ARM to take over fine manipulation upon detecting the target. The specific aims are to (1) to develop,
implement, and bench-test the VGS control with fiducial markers and adaptive tools; (2) to evaluate the new
control among powered wheelchair users who will use a wheelchair-mounted ARM to complete a set of
everyday manipulation tasks; (3) to explore the potential benefits and limitations of ARMs enhanced by the
VGS control with fiducial markers and adaptive tools in a one-month pilot home trial.
We expect to improve manipulation functions of Veterans with upper limb impairments through a more practical
and usable implementation of vision-based robotic control and human-robot interaction technologies. The
mission of the VA Prosthetic & Sensory Aids Service (PSAS)...

## Key facts

- **NIH application ID:** 10407465
- **Project number:** 5I01RX003242-03
- **Recipient organization:** VETERANS HEALTH ADMINISTRATION
- **Principal Investigator:** Dan Ding
- **Activity code:** I01 (R01, R21, SBIR, etc.)
- **Funding institute:** VA
- **Fiscal year:** 2022
- **Award amount:** —
- **Award type:** 5
- **Project period:** 2020-07-01 → 2024-06-30

## Primary source

NIH RePORTER: https://reporter.nih.gov/project-details/10407465

## Citation

> US National Institutes of Health, RePORTER application 10407465, Development of Vision-Guided Shared Control for Assistive Robotic Manipulators (5I01RX003242-03). Retrieved via AI Analytics 2026-05-24 from https://api.ai-analytics.org/grant/nih/10407465. Licensed CC0.

---

*[NIH grants dataset](/datasets/nih-grants) · CC0 1.0*