Look at what I'm saying

University of Utah engineers show brain depends on vision to hear

2013-09-05

(Press-News.org) University of Utah bioengineers discovered our understanding of language may depend more heavily on vision than previously thought: under the right conditions, what you see can override what you hear. These findings suggest artificial hearing devices and speech-recognition software could benefit from a camera, not just a microphone.

"For the first time, we were able to link the auditory signal in the brain to what a person said they heard when what they actually heard was something different. We found vision is influencing the hearing part of the brain to change your perception of reality -- and you can't turn off the illusion," says the new study's first author, Elliot Smith, a bioengineering and neuroscience graduate student at the University of Utah. "People think there is this tight coupling between physical phenomena in the world around us and what we experience subjectively, and that is not the case."

The brain considers both sight and sound when processing speech. However, if the two are slightly different, visual cues dominate sound. This phenomenon is named the McGurk effect for Scottish cognitive psychologist Harry McGurk, who pioneered studies on the link between hearing and vision in speech perception in the 1970s. The McGurk effect has been observed for decades. However, its origin has been elusive.

In the new study, which appears today in the journal PLOS ONE, the University of Utah team pinpointed the source of the McGurk effect by recording and analyzing brain signals in the temporal cortex, the region of the brain that typically processes sound.

Working with University of Utah bioengineer Bradley Greger and neurosurgeon Paul House, Smith recorded electrical signals from the brain surfaces of four severely epileptic adults (two male, two female) from Utah and Idaho. House placed three button-sized electrodes on the left, right or both brain hemispheres of each test subject, depending on where each patient's seizures were thought to originate. The experiment was done on volunteers with severe epilepsy who were undergoing surgery to treat their epilepsy.

These four test subjects were then asked to watch and listen to videos focused on a person's mouth as they said the syllables "ba," "va," "ga" and "tha." Depending on which of three different videos were being watched, the patients had one of three possible experiences as they watched the syllables being mouthed:

The motion of the mouth matched the sound. For example, the video showed "ba" and the audio sound also was "ba," so the patients saw and heard "ba."

The motion of the mouth obviously did not match the corresponding sound, like a badly dubbed movie. For example, the video showed "ga" but the audio was "tha," so the patients perceived this disconnect and correctly heard "tha."

The motion of the mouth only was mismatched slightly with the corresponding sound. For example, the video showed "ba" but the audio was "va," and patients heard "ba" even though the sound really was "va." This demonstrates the McGurk effect -- vision overriding hearing.

By measuring the electrical signals in the brain while each video was being watched, Smith and Greger could pinpoint whether auditory or visual brain signals were being used to identify the syllable in each video. When the syllable being mouthed matched the sound or didn't match at all, brain activity increased in correlation to the sound being watched. However, when the McGurk effect video was viewed, the activity pattern changed to resemble what the person saw, not what they heard. Statistical analyses confirmed the effect in all test subjects.

"We've shown neural signals in the brain that should be driven by sound are being overridden by visual cues that say, 'Hear this!'" says Greger. "Your brain is essentially ignoring the physics of sound in the ear and following what's happening through your vision."

Greger was senior author of the study as an assistant professor of bioengineering at the University of Utah. He recently took a faculty position at Arizona State University.

The new findings could help researchers understand what drives language processing in humans, especially in a developing infant brain trying to connect sounds and lip movement to learn language. These findings also may help researchers sort out how language processing goes wrong when visual and auditory inputs are not integrated correctly, such as in dyslexia, Greger says.

INFORMATION:

Equipment used for these experiments was purchased as part of a Defense Advance Research Project Agency project to control and communicate with prostheses, in particular for recording signals to control a prosthetic limb. Additional funding for the study was provided by the University of Utah Research Foundation.

A video demonstrating the McGurk effect is available by contacting Aditi Risbud at aditi.risbud@coe.utah.edu.

ELSE PRESS RELEASES FROM THIS DATE:

Only known hibernating primate has unique sleep patterns

2013-09-05

Fat-tailed dwarf lemurs are the only primates known to hibernate, and new research suggests their sleep patterns during hibernation are markedly different from those of ground squirrels, which also hibernate at similar temperatures. The results are published September 4 in the open access journal PLOS ONE by Andrew Krystal and colleagues from Duke University School of Medicine and the Duke Lemur Center. During hibernation, dwarf lemurs experience periods of rapid eye movement (REM) sleep at relatively high ambient temperatures, but no non-REM sleep. Ground squirrels, ...

Ashtray availability, signage may determine success of smoke-free legislation

2013-09-05

Signs banning smoking may not have as much of an impact on secondhand smoke concentrations as the presence of ashtrays or ashtray equivalents, according to research published September 4 in the open access journal PLOS ONE by Constantine Vardavas from the Harvard School of Public Health and colleagues from other institutions. The authors measured the success of a non-enforced, nationwide smoke-free legislation in Greece by testing levels of secondhand smoke before the ban and for two years afterward. Following the 2010 legislation, secondhand smoke concentrations dropped ...

Northeastern US forests transformed by human activity over 400 years

2013-09-05

Forests in the northeastern US have been radically transformed over the last four centuries by human activity, and their relationship with climate factors like rainfall weakened, according to research published September 4 in the open access journal PLOS ONE by Jonathan Thompson and colleagues from the Smithsonian Institution. Though the land use history of the northeastern United States is well documented, its ecological consequences remain poorly understood. Based on more than 300,000 references to individual trees within colonial-era property records, the researchers ...

Relationship of kidney function estimates to risk improves by measuring cystatin C in the blood

2013-09-05

A new, international study from the Chronic Kidney Disease Prognosis Consortium found that use of blood levels of cystatin C to estimate kidney function—alone or in combination with creatinine—strengthens the association between kidney function and risks of death and end-stage renal disease. The findings, reported in the September 5 edition of the New England Journal of Medicine, suggest that the use of cystatin C as a measurement of kidney function could lead to better staging and risk classification of chronic kidney disease. They also suggest that the measurement ...

Experimental compound reverses Down syndrome-like learning deficits in mice

2013-09-05

Researchers at Johns Hopkins and the National Institutes of Health have identified a compound that dramatically bolsters learning and memory when given to mice with a Down syndrome-like condition on the day of birth. As they report in the Sept. 4 issue of Science Translational Medicine, the single-dose treatment appears to enable the cerebellum of the rodents' brains to grow to a normal size. The scientists caution that use of the compound, a small molecule known as a sonic hedgehog pathway agonist, has not been proven safe to try in people with Down syndrome, but say ...

New laser-based tool could dramatically improve the accuracy of brain tumor surgery

2013-09-05

ANN ARBOR, Mich. — A new laser-based technology may make brain tumor surgery much more accurate, allowing surgeons to tell cancer tissue from normal brain at the microscopic level while they are operating, and avoid leaving behind cells that could spawn a new tumor. In a new paper, featured on the cover of the journal Science Translational Medicine, a team of University of Michigan Medical School and Harvard University researchers describes how the technique allows them to "see" the tiniest areas of tumor cells in brain tissue. They used this technique to distinguish ...

Training the older brain in 3-D: Video game enhances cognitive control

2013-09-05

Scientists at UC San Francisco are reporting that they have found a way to reverse some of the negative effects of aging on the brain, using a video game designed to improve cognitive control. The findings, published this week in Nature, show how a specially designed 3-D video game can improve cognitive performance in healthy older adults. The researchers said it provides a measure of scientific support to the burgeoning field of brain fitness, which has been criticized for lacking evidence that such training can induce lasting and meaningful changes. In the ...

Back of pack health warnings make little impact on teen smokers

2013-09-05

Back of pack picture or text warnings depicting the dangers of smoking, make little impact on teen smokers, particularly those who smoke regularly, suggests research published online in Tobacco Control. Pictorial warnings work better than text alone, but if positioned on the back of the pack are less visible and less effective, say the researchers. In 2008 the UK became the third European Union country to require pictorial health warnings to be carried on the back of cigarette packs. In only five out of the 60 countries worldwide that have introduced this policy do ...

Chlamydia and gonorrhoea infections linked to pregnancy complications

2013-09-05

Becoming infected with chlamydia or gonorrhoea in the lead-up to, or during, pregnancy, increases the risk of complications, such as stillbirth or unplanned premature birth, indicates research published online in the journal Sexually Transmitted Infections. The researchers analysed the birth records of more than 350,000 women who had had their first baby between 1999 and 2008 in New South Wales, Australia's most heavily populated state. The researchers wanted to find out if infection with either chlamydia or gonorrhoea in the lead-up to, or during, pregnancy, had any ...

Queen Mary scientists uncover genetic similarities between bats and dolphins

2013-09-05

The evolution of similar traits in different species, a process known as convergent evolution, is widespread not only at the physical level, but also at the genetic level, according to new research led by scientists at Queen Mary University of London and published in Nature this week. The scientists investigated the genomic basis for echolocation, one of the most well-known examples of convergent evolution to examine the frequency of the process at a genomic level. Echolocation is a complex physical trait that involves the production, reception and auditory processing ...

LAST 30 PRESS RELEASES:

The American Society of Tropical Medicine and Hygiene and the Bill & Melinda Gates Foundation announce new research fellowship in malaria genomics in honor of professor Dominic Kwiatkowski