EyeMusic: New App Enables The Blind To ‘Hear’ Images

In the world of Marvel Comics, blind superhero Daredevil is able to fight crime using his other senses, which were heightened to super-human levels due to his blindness. While that may be complete fiction, it turns out that you can use other senses, such as sound, to help the blind “see” their surroundings.

In order for the blind to perceive colors or shapes, Hebrew University researchers use a sensory substitution device (SSD) that transforms images into pieces of music, enabling the blind to “hear” pictures. With the new EyeMusic SSD app (available on the App Store) one hears pleasant musical notes, which convey information about colors, shapes and location of objects in the world.

Related articles

• Israel’s Project RAY Creates Advanced Smartphone And Apps For The Blind
• OrCam Will Help The Visually Impaired Read Anywhere

The system uses different pitch level notes and specific timing to convey the images. The higher the pitch is – the higher the pixels on a vertical level. On the horizontal plane, the notes played first represent the left side of the image and the notes towards the end represent the right side of the image. Color is transformed into different instruments, so soft vocals would be white, a trumpet is blue and silence means black.

Regain “sight” within three hours

In recent studies, blind and blindfolded sighted users were shown to correctly perceive and interact with objects using EyeMusic. Participants were able to recognize different shapes and colors or reach for a specific beverage:

[youtuber youtube=’http://www.youtube.com/watch?v=r6bz1pOEJWg’]

Professor Amir Amedi, Head of the Center for Human Perception and Cognition at the Hebrew University of Jerusalem, developed the technology for the blind and visually impaired. Amedi says that using the SSD and a unique training program, the blind are able to achieve various advanced visual-linked abilities in two to three hours of training.

“The human brain is more flexible than we thought,” says Amedi. “These results give a lot of hope for the successful regaining of visual functions using cheap non-invasive SSDs or other sight restoration approaches. They suggest that in the blind, brain areas have the potential to be ‘awakened’ to processing visual properties and tasks even after years or maybe even lifelong blindness, if the proper technologies and training approaches are used.”

Tasks, not senses

Past studies have validated that the blind can characterize sound-conveyed images into complex object categories such as faces, houses and outdoor scenes, plus everyday objects and could locate people’s positions, identify facial expressions and read letters and words.

The Hebrew University research has shown that many brain areas are characterized by their computational task, meaning they can be activated using senses other than the one commonly used for a certain task. These areas can be awakened even in people who were never exposed to “original” sensory information at all, such as a person who’s blind from birth.

Helping the blind and understanding the brain

When processing “visual information” conveyed through SSD, the researchers found that congenitally blind people who learned to read by touch using the Braille script or through their ears with sensory substitution devices use the same areas in the visual cortex as those used by sighted readers.

The researchers hope EyeMusic can become a tool for future neuroscience research and holds great promise for the field of visual rehabilitation.

“The utilization of the EyeMusic and its added color information in the field of neuroscience could facilitate exploring several questions in the blind with the potential to expand our understanding of brain organization in general,” says Amedi.

The work of Amedi and his colleagues is patented by Yissum, the Hebrew University’s Technology Transfer Company.

Photo: Young Man With Headphones Listening To Music by Bigstock