Real-time, real-world captioning comes to Google Glass
October 4, 2014
Google Glass is fitted with a microphone and installed with speech-recognition software, which allows the wearer to talk to the device to give it instructions. This software has been refined to a point where it is now reasonable to expect very few errors — but what Google Glass is not so good at is picking up sounds from farther away.
The wearable head-up display’s potential for the hearing impaired, though, did not go unnoticed by a team of researchers at Georgia Institute of Technology. They found a way to circumvent the limitation’s of Glass’ microphone to create an app that captions conversations in real-time.
Captioning on Glass adds an Android smartphone to the mix. The speaker talks directly into the smartphone’s microphone; the free CoG Android app translates the speech into text using Google’s own speech recognition software and sends it to the free Captioning on Glass Glassware.
The app was developed by the team when School of Interactive Computing Professor Jim Foley found he was having trouble hearing — and thought that Google Glass might be able to help.
“This system allows wearers like me to focus on the speaker’s lips and facial gestures,” Professor Foley said. “If hard-of-hearing people understand the speech, the conversation can continue immediately without waiting for the caption. However, if I miss a word, I can glance at the transcription, get the word or two I need and get back into the conversation.”
Although the added smartphone might seem unwieldy to some, the project’s leader, Professor Thad Starner, said that it was actually helpful in that holding a physical object made people think more carefully about what they were saying and how, reducing the number of speech disfluencies. He also noted that it was far superior than relying on the Google Glass microphone in terms of sound quality.
“Glass has its own microphone, but it’s designed for the wearer,” Professor Starner said. “The mobile phone puts a microphone directly next to the speaker’s mouth, reducing background noise and helping to eliminate errors.”
The team is working on a similar app designed to translate spoken language using an Android app on a smartphone, sending the translation to Google Glass. So far, the project has two-way translations for English, Spanish, French, Russian, Korean and Japanese. This app will be launched in the near future.
“For both uses, the person wearing Glass has to hand their smartphone to someone else to begin a conversation,” said Professor Starner. “It’s not ideal for strangers, but we designed the program to be used among friends, trusted acquaintances or while making purchases.”