Microsoft’s HoloLens has an impressive ability to quickly sense its surroundings, but a group of scientists found it’s really good at helping blind people find their way through buildings and offering a better sense of where objects are around them.
The researchers, at the California Institute of Technology, created a new guiding app for Microsoft’s HoloLens by taking advantage of the device’s real-time room and object mapping capability, as well as speakers that can make audio seem to be coming from different points in three-dimensional space. They used these features to map a complicated path through a campus building and created a virtual guide that helps a blind person navigate it, according to work recently published on the bioRxiv website.
The researchers recruited seven blind people to test it out. They were given a brief intro but no training, and then asked to accomplish a variety of tasks. The users could reliably locate and point to objects from audio cues, and were able to find a chair in a room more quickly than they normally would, and avoid obstacles easily as well.
Then they were asked to navigate from the entrance of a building to a room on the second floor by following the headset’s instructions. A “virtual guide repeatedly says: “follow me” from a virtual distance of a meter ahead, while also warning when stairs were coming, where handrails were and when the user had gone off course.
All seven users got to their destinations on the first try, and much more quickly than if they had had to proceed normally with no navigation.
They designed the system around audible feedback. Every major object and feature can tell the user where it is, either via voice or sound. Walls for instance make a hiss sound like white noise, as the user approaches them. And the user can scan the scene, with objects announcing themselves from left to right from the direction in which they are located. A single object can be selected and will repeat its callout to help the user find it.
The video below demonstrates a female voice directing a HoloLens-wearing study subject, who is blind, by saying things like “Railings on both sides,” “Up stairs,” and “Right turn ahead.” The man follows the commands, walking easily from a first-floor lobby up a set of staircases, around several corners, and past a few doorways until he arrives at a room on the second floor. He’s one of seven subjects who tried the application. All got to their destination on the first try.
https://drive.google.com/file/d/1v5Wdbi2WWXAMQXyLmVU6ogWDQfpyINQu/view
Markus Meister, a professor at Caltech and coauthor of the study, thinks the research could eventually lead to a device that could be offered to visually impaired visitors at places like hotels or malls, helping them get around unfamiliar areas more easily. There are already some tools that can be used this way outdoors, such as turn-by-turn mapping apps—but indoors, as Meister notes, there aren’t as many options.
The potential for a system like this is enormous, but this is just a prototype. As systems like HoloLens get more mobile and more powerful, they’ll go from controlled lab studies to off the shelf products.
For now, any routes from one point to another must be scanned in advance, and there isn’t a way to track other people who might walk through the space as the HoloLens wearer is navigating it.
