Deaf people trained to lip-read show activity in auditory cortex. The brain uses 'hearing' circuits for visual speech processing; blurring the line between senses.
Haskins Laboratories, CC BY-SA 3.0, via Wikimedia Commons
Seeing sounds
Your brain constructs a single, unified perception of the world that rarely feels like a patchwork of different senses. Speech perception shows this seamless integration. You don't just hear words; you see them, too. This phenomenon is vividly demonstrated by the McGurk effect, an illusion first described by Harry McGurk and John MacDonald in 1976. If you watch a video of a person mouthing the syllable "ga" while the audio plays the syllable "ba," you will most likely perceive a completely different sound: "da". Your brain fuses the conflicting visual and auditory information to create a new, coherent perception.
This integration happens in a part of the brain called the superior temporal sulcus (STS). The STS acts as a convergence zone, positioned between the visual and auditory cortices. It receives inputs from both areas and combines the sight of mouth movements with the sound of a voice. Research at institutions like Haskins Laboratories in New Haven, which moved to the city in 1970 and has long-standing affiliations with Yale University, has deeply explored these mechanisms. Since the 1950s, Haskins researchers have been at the forefront, developing early speech synthesis technology and the influential "Motor Theory of Speech Perception," which posits that we perceive speech by referencing our own systems for producing it.
A repurposed cortex
The brain's flexibility in processing speech is apparent in individuals who are deaf. Functional magnetic resonance imaging (fMRI) studies show that when a congenitally deaf person lip-reads, their auditory cortex—the part of the brain that processes sound—becomes active. Specifically, areas like the posterior superior temporal gyrus (pSTS) and even parts of the secondary auditory cortex (BA42) show greater activation in deaf participants compared to hearing ones during silent speechreading. It seems the brain, deprived of auditory input, repurposes its "hearing" circuits to process visual speech information. This neural reorganization demonstrates that the function of a brain area is not rigidly fixed.
This cross-modal activity is not limited to those who cannot hear. In hearing individuals, lip-reading silent speech also activates the auditory cortex, though often to a lesser degree. The brain essentially "fills in" the missing sound, synthesizing auditory features from visual cues alone. Magnetoencephalographic (MEG) recordings show that when someone is lip-reading, their auditory cortex activity synchronizes with the rhythm of the unseen, silent speech. This suggests a predictive mechanism where visual information helps the brain anticipate and decode the auditory signal faster and more accurately, an advantage in noisy environments. Studies using sEEG (stereoelectroencephalography) have found that the response in the STS to audiovisual speech is 40% faster and 18% larger than to sound alone.
💡Fun Facts
The McGurk effect is not equally strong in all languages and can be influenced by your familiarity with the speaker's face.
The brain's ability to integrate senses is so robust that if the left superior temporal sulcus is damaged by a stroke, the right one can take over to preserve audiovisual speech perception.
Haskins Laboratories developed one of the first working prototypes of a reading machine for the blind.
Researchers are studying the brain's "aperiodic" signals—a form of neural noise—to understand why speech processing can be less efficient in some individuals with autism.
Haskins Laboratories is a private research institute and is not open for public tours. Affiliated researchers from Yale University and the University of Connecticut often give public lectures.
Admission
Not applicable. Check Yale and UConn event calendars for related public talks.
Accessibility
The research facilities are in modern university buildings and are generally accessible.
