Visually impaired individuals can now leverage the power of two apps, developed by a team at the University of Michigan, to enhance their everyday experiences using iPhones. These apps, VizLens and ImageExplorer, provide valuable features such as virtual labels for control panels and the ability to explore images through touch and audio feedback.
How does VizLens enable visually impaired users to interact with control panels?
VizLens acts as a real-world screen reader, allowing users to read labels on control panels by pointing at buttons of interest using their fingers. By utilizing smartphone cameras, users can comprehend and operate various interfaces found in their surroundings, including home appliances and public kiosks. By taking a picture of an interface, VizLens employs optical character recognition to automatically detect text labels. Users familiarize themselves with the layout on their smartphone touchscreen and can then move their finger across the physical control panel, with the app vocalizing the button beneath their finger.
What features does ImageExplorer provide to help visually impaired users explore images?
ImageExplorer, the second app developed by the team, enables visually impaired individuals to gain a deeper understanding of image content. By integrating advanced object detection and segmentation models, such as Meta’s Detectron2 visual recognition library, Google OCR, and image analysis models, ImageExplorer allows users to explore objects within an image and understand their relationships. When an image is uploaded, ImageExplorer provides a comprehensive analysis, offering a general overview of the objects detected, relevant tags, and a descriptive caption. The app also features a touch-based interface, enabling users to explore the spatial layout and content of the image by interacting with different areas.
How does ImageExplorer stand out in terms of detail and description?
ImageExplorer sets itself apart by providing an unparalleled level of detail in its descriptions of objects within an image. It goes beyond basic identification and provides users with comprehensive information, such as the type of clothing a person is wearing, their activities, and the precise position of objects within the image. This rich level of description empowers visually impaired users to form a vivid understanding of the image’s content, despite their visual impairment.
How is user feedback shaping the development and deployment of these apps?
Hundreds of visually impaired individuals have participated in user testing, providing valuable feedback to the team behind VizLens and ImageExplorer. This feedback is instrumental in further refining the tools to ensure optimal performance and user satisfaction. Although ImageExplorer is a more recent concept than VizLens, which made its debut in 2016, the team is continuously improving the apps based on user input. While some aspects, like the identification of clothing items, require further refinement, the overall results indicate that ImageExplorer enhances users’ ability to assess the accuracy of AI-generated captions. With public deployment, the team anticipates gathering additional feedback to adapt these tools to better suit users’ needs.
VizLens and ImageExplorer exemplify the tremendous potential of technology in fostering inclusivity for visually impaired individuals. These apps provide virtual labels for control panels and enable detailed exploration of images, granting users greater agency and access to information. As these apps continue to evolve, driven by user feedback and advancements in AI models, they hold the promise of bridging the gap between visual content and accessibility, enhancing the lives of visually impaired users worldwide.
Download VizLens on the AppStore
Download ImageExplorer on the AppStore
