AI Smartglasses & Smartphones to Bridge Hearing and DHH Communities

Using AI Smartglasses and Smartphones to Reduce Communication Barriers between the Hearing and the Deaf and Hard of Hearing (DHH) Communities

AI advances in speech recognition and in sign language recognition are enabling new interfaces that can reduce communication barriers. At Georgia Tech in 2013, we used Google Glass to show live transcription to help people who are hard of hearing better follow spoken conversations. However, the use of Glass often distracted from the conversation. Since then, we have run many experiments to determine the best practices for creating practical smartglasses that can be used unobtrusively for captioning while in group conversation.

To make American Sign Language learning more accessible to hearing parents of deaf infants, we have been creating games for smartphones that use sign language recognition as part of the game play. PopSignAI Preview, available for both Android and iPhone, is our first example. Soon we will be releasing the Sign Language Recognition and Gesture ToolKit (SLRGTK) that provides the tools to collect examples of sign, annotate them, process the video with Google's MediaPipe hand tracker, train a recognition system, and integrate the recognizer into the skeleton of a smartphone game. The hope is that SLRGTK will enable educational games to be made for the world's 150+ major sign languages.

Thad Starner is a Georgia Tech Professor and a wearable computing pioneer. In 1990, Starner coined the term "augmented reality" to describe the types of interfaces he envisioned for the future. In 1997, Thad was a founder of the annual ACM International Symposium on Wearable Computers (ISWC), now in its 28th year. From 2010-2018 Dr. Starner was a Technical Lead on Google's Glass, which was named a "50 Most Influential Gadget of All Time" by Time Magazine. Professor Starner has been inducted into the CHI Academy in 2017 and AWE's XR Hall of Fame in 2024. He has over 100 issued United States utility patents on wearables, artificial intelligence and interfaces.

View the speaker's youtube channel.

Host or Publisher UNSW Computer Science and Engineering