(Press-News.org) Researchers at UC San Francisco and UC Berkeley have developed a brain-computer interface (BCI) that has enabled a woman with severe paralysis from a brainstem stroke to speak through a digital avatar.
It is the first time that either speech or facial expressions have been synthesized from brain signals. The system can also decode these signals into text at nearly 80 words per minute, a vast improvement over commercially available technology.
Edward Chang, MD, chair of neurological surgery at UCSF, who has worked on the technology, known as a brain computer interface, or BCI, for more than a decade, hopes this latest research breakthrough, appearing Aug. 23, 2023, in Nature, will lead to an FDA-approved system that enables speech from brain signals in the near future.
“Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others,” said Chang, who is a member of the UCSF Weill Institute for Neuroscience and the Jeanne Robertson Distinguished Professor in Psychiatry. “These advancements bring us much closer to making this a real solution for patients.”
Chang’s team previously demonstrated it was possible to decode brain signals into text in a man who had also experienced a brainstem stroke many years earlier. The current study demonstrates something more ambitious: decoding brain signals into the richness of speech, along with the movements that animate a person’s face during conversation.
Chang implanted a paper-thin rectangle of 253 electrodes onto the surface of the woman’s brain over areas his team has discovered are critical for speech. The electrodes intercepted the brain signals that, if not for the stroke, would have gone to muscles in her, tongue, jaw and larynx, as well as her face. A cable, plugged into a port fixed to her head, connected the electrodes to a bank of computers.
For weeks, the participant worked with the team to train the system’s artificial intelligence algorithms to recognize her unique brain signals for speech. This involved repeating different phrases from a 1,024-word conversational vocabulary over and over again, until the computer recognized the brain activity patterns associated with the sounds.
Rather than train the AI to recognize whole words, the researchers created a system that decodes words from phonemes. These are the sub-units of speech that form spoken words in the same way that letters form written words. “Hello,” for example, contains four phonemes: “HH,” “AH,” “L” and “OW.”
Using this approach, the computer only needed to learn 39 phonemes to decipher any word in English. This both enhanced the system’s accuracy and made it three times faster.
“The accuracy, speed and vocabulary are crucial,” said Sean Metzger, who developed the text decoder with Alex Silva, both graduate students in the joint Bioengineering Program at UC Berkeley and UCSF. “It’s what gives a user the potential, in time, to communicate almost as fast as we do, and to have much more naturalistic and normal conversations.”
To create the voice, the team devised an algorithm for synthesizing speech, which they personalized to sound like her voice before the injury, using a recording of her speaking at her wedding.
The team animated the avatar with the help of software that simulates and animates muscle movements of the face, developed by Speech Graphics, a company that makes AI-driven facial animation. The researchers created customized machine-learning processes that allowed the company’s software to mesh with signals being sent from the woman’s brain as she was trying to speak and convert them into the movements on the avatar’s face, making the jaw open and close, the lips protrude and purse and the tongue go up and down, as well as the facial movements for happiness, sadness and surprise.
“We’re making up for the connections between the brain and vocal tract that have been severed by the stroke,” said Kaylo Littlejohn, a graduate student working with Chang and Gopala Anumanchipalli, PhD, a professor of electrical engineering and computer sciences at UC Berkeley. “When the subject first used this system to speak and move the avatar’s face in tandem, I knew that this was going to be something that would have a real impact.”
An important next step for the team is to create a wireless version that would not require the user to be physically connected to the BCI.
“Giving people the ability to freely control their own computers and phones with this technology would have profound effects on their independence and social interactions,” said co-first author David Moses, PhD, an adjunct professor in neurological surgery.
Authors: Additional authors include Ran Wang, Maximilian Dougherty, Jessie Liu, delyn Tu-Chan, and Karunesh Ganguly of UCSF, Peter Wu and Inga Zhuravleva of UC Berkeley, and Michael Berger of Speech Graphics.
Funding: This research was supported by the National Institutes of Health (NINDS 5U01DC018671, T32GM007618), the National Science Foundation, and philanthropy.
About UCSF: The University of California, San Francisco (UCSF) is exclusively focused on the health sciences and is dedicated to promoting health worldwide through advanced biomedical research, graduate-level education in the life sciences and health professions, and excellence in patient care. UCSF Health, which serves as UCSF's primary academic medical center, includes top-ranked specialty hospitals and other clinical programs, and has affiliations throughout the Bay Area. UCSF School of Medicine also has a regional campus in Fresno. Learn more at https://ucsf.edu, or see our Fact Sheet.
###
Follow UCSF
ucsf.edu | Facebook.com/ucsf | YouTube.com/ucsf
END
How artificial intelligence gave a paralyzed woman her voice back
2023-08-23
ELSE PRESS RELEASES FROM THIS DATE:
Wildlife overflow enriches biodiversity beyond park boundaries
2023-08-23
New research has discovered the power of large national parks to not only enhance bird diversity inside their borders but boost mammal diversity in nearby unprotected areas.
The University of Queensland’s Dr Matthew Luskin said the study, which involved using more than 2,000 cameras and bird surveys across Southeast Asia, reveals for the first time the benefit of expanding protected land areas around the globe beyond park boundaries.
“Protected area expansions are often a difficult and expensive process, but our results show they are absolutely worth it,” Dr Luskin said.
“We ...
Graphene: More magic starts when flatness ends
2023-08-23
Researchers from The University of Manchester and the University of Warwick finally solved the long-standing puzzle of why graphene is so much more permeable to protons than expected by theory.
A decade ago, scientists at The University of Manchester demonstrated that graphene is permeable to protons, nuclei of hydrogen atoms. The unexpected result started a debate in the community because theory predicted that it would take billions of years for a proton to permeate through graphene’s dense crystalline structure. This had led to suggestions that protons permeate not through the crystal lattice itself, ...
Researchers fully sequence the Y chromosome for the first time
2023-08-23
What was once the final frontier of the human genome — the Y chromosome — has just been mapped out in its entirety.
Led by the National Human Genome Research Institute (NHGRI), a team of researchers at the National Institute of Standards and Technology (NIST) and many other organizations used advanced sequencing technologies to read out the full DNA sequence of the Y chromosome — a region of the genome that typically drives male reproductive development. The results of a study ...
New research shows how cancer rewires a key immune pathway to spread
2023-08-23
A study led by researchers at Memorial Sloan Kettering Cancer Center (MSK) and Weill Cornell Medicine discovered a new relationship between cancer cells and the immune system, and shows how cancer can selfishly hijack a normally helpful immune pathway.
Usually, activation of this key immune pathway — called the STING pathway — triggers a strong inflammatory response that protects the body from foreign and unhealthy cells. But prolonged activation of the same pathway leads to a desensitization and ultimately to a “rewiring” of cellular signaling, which aids and abets cancer’s spread, the researchers found.
“You might think of ...
Graphene discovery could help generate hydrogen cheaply and sustainably
2023-08-23
Researchers from The University of Warwick and the University of Manchester have finally solved the long-standing puzzle of why graphene is so much more permeable to protons than expected by theory.
A decade ago, scientists at The University of Manchester demonstrated that graphene is permeable to protons, nuclei of hydrogen atoms.
The unexpected result started a debate in the community because theory predicted that it would take billions of years for a proton to permeate through graphene’s dense crystalline structure. ...
Adding immunity to human kidney-on-a-chip advances cancer drug testing
2023-08-23
By Benjamin Boettner
(Boston) — A growing repertoire of cell and molecule-based immunotherapies is offering patients with indomitable cancers new hope by mobilizing their immune systems against tumor cells. An emerging class of such immunotherapeutics, known as T cell bispecific antibodies (TCBs), are of growing importance with several TCBs that the U.S. Food and Drug Administration (FDA) approved for the treatment of leukemias, lymphomas, and myelomas. These antibody drugs label tumor cells with one of their ends, and attract immune cells with another end to coerce them into tumor cell killing.
One major challenge in the development of TCBs and other immunotherapy ...
Fungus gnats as pollinators not pests
2023-08-23
Many plants and crops rely on insects to pollinate them so they can reproduce. A new study has shown that several flowering plants from the group Euonymus are pollinated by fungus gnats, a dipteran insect. Specifically, they pollinate Euonymus plants which have red-petaled flowers with short stamens and yogurt-like scent. Although fungus gnats are known to pollinate hundreds of plant species, this study shows that the particular traits of red Euonymus flowers were likely to have been acquired via pollination syndrome, evolving over a process of natural selection to be pollinated specifically by fungus gnats. This research highlights the important role of Diptera, which are commonly regarded ...
Solar powered irrigation: a game-changer for small-scale farms in sub-Saharan Africa
2023-08-23
In sub-Saharan Africa 80% of agricultural production is from smallholder farmers, who face constraints on increasing farm productivity resulting in a large yield gap. Extensive rain-fed agriculture (90% of all cropland) under unpredictable and erratic rainfall pattern is a leading cause of the low productivity and food insecurity in Africa, together with a low degree of mechanization. This has been reinforcing a persistent poverty trap, triggered by cyclical famines that are jeopardizing local development opportunities.
In a new IIASA-led study as part of the research project Renewables for African Agriculture (RE4AFAGRI), an international ...
Looking out for kids: a case for better pediatric trauma interventions
2023-08-23
In the weeks following the onset of the COVID-19 pandemic, public health directives called for masking, social distancing, social isolation, and stay-at-home orders. Apart from the severe medical consequences as a direct result of the pandemic, the ensuing social isolation had far-reaching impacts on children. The pandemic control measures affected the pediatric population by increasing mental distress, limiting physical activity, changing sleep patterns, and reducing emergency room visits. Unfortunately, the scope of pediatric trauma during the pandemic remains ...
SwRI investigates the efficiency impact of smart-technology-enabled vehicles
2023-08-23
SAN ANTONIO — August 23, 2023 — A Southwest Research Institute project funded by the U.S. Department of Energy (DOE) has demonstrated an average of 15% energy savings when vehicles outfitted with connected and automated vehicle systems, or CAVs, are introduced into traffic.
CAVs use wireless smart technology to communicate with other CAVs and traffic infrastructure. SwRI’s eco-driving framework uses custom software and predictive powertrain algorithms to enable human drivers to make more efficient driving ...