Brain-computer interface creates text by decoding brain signals associated with handwriting

(Press-News.org) PROVIDENCE, R.I. [Brown University] -- Researchers with the BrainGate collaboration have, for the first time, used an implanted sensor to record the brain signals associated with handwriting, and used those signals to create text on a computer in real time.

In a study published in the journal Nature, a clinical trial participant with cervical spinal cord injury used the system to "type" words on a computer at a rate of 90 characters per minute, more than double the previous record for typing with a brain-computer interface. This was done by the participant merely thinking about the hand motions involved in creating written letters.

The research team is hopeful that such a system could one day help to restore people's ability to communicate following paralysis caused by injury or illness.

The new study is part of the BrainGate clinical trial, directed by Dr. Leigh Hochberg. Hochberg is a critical care neurologist and a professor at Brown University's School of Engineering and Carney Institute for Brain Science. Frank Willett, a research scientist at Stanford University and the Howard Hughes Medical Institute (HHMI), led the study, which was supervised by Krishna Shenoy, a Stanford professor and HHMI investigator, and Dr. Jaimie Henderson, a professor of neurosurgery at Stanford.

"An important mission of our BrainGate consortium research is to restore rapid, intuitive communication for people with severe speech or motor impairments," said Hochberg, who also directs the Center for Neurotechnology and Neurorecovery at Massachusetts General Hospital and the VA RR&D Center for Neurorestoration and Neurotechnology at the Dept. of Veterans Affairs Providence Healthcare System. "Frank's demonstration of fast, accurate neural decoding of handwriting marks an exciting new chapter in the development of clinically useful neurotechnologies."

The BrainGate collaboration has been working for several years on systems that enable people to generate text through direct brain control. Previous incarnations have involved trial participants thinking about the motions involved in pointing to and clicking letters on a virtual keyboard. That system enabled one participant to type 40 characters per minute, which was the previous record speed.

For this latest study, the team wanted to find out if asking a participant to think about motions involved in writing letters and words by hand would be faster.

"We want to find new ways of letting people communicate faster," Willett said. "This new system uses both the rich neural activity recorded by intracortical electrodes and the power of language models that, when applied to the neurally decoded letters, can create rapid and accurate text."

The trial participant, a 65-year-old (at the time of the study) man, was paralyzed from the neck down by a spinal cord injury. As part of the clinical trial, Henderson placed two tiny electrodes about the size of a baby aspirin in a part of his brain associated with the movement of his right arm and hand. Using signals the sensors picked up from individual neurons when the man imagined writing, a machine learning algorithm recognized the patterns his brain produced with each letter. With this system, the man could copy sentences and answer questions at a rate similar to that of someone the same age typing on a smartphone.

The system is so fast because each letter elicits a highly distinctive activity pattern, making it relatively easy for the algorithm to distinguish one from another, Willett says.

The new research is the latest in a series of advances in brain-computer interfaces (BCIs) made by the BrainGate collaboration, which includes researchers from Brown University, Massachusetts General Hospital, Harvard Medical School, the Providence VA Medical Center, Stanford University, and Case Western Reserve University.

In 2012, the team published landmark research in which clinical trial participants were able, for the first time, to operate multidimensional robotic prosthetics using a BCI. That work has been followed by a steady stream of refinements to the system, as well as new clinical breakthroughs that have enabled people to directly control tablet apps and even move their own paralyzed limbs. Most recently, the team demonstrated the first human use of a wireless intracortical BCI that can transmit neural data at full bandwidth.

Hochberg said he's grateful to clinical trial participants for making these breakthroughs and future ones possible.

"The people who enroll in the BrainGate trial are amazing," Hochberg said. "It's their pioneering spirit that not only allows us to gain new insights into human brain function, but that leads to the creation of systems that will help other people with paralysis."

INFORMATION:

The research was funded by the National Institute of Neurological Disorders and Stroke and the NIH BRAIN Initiative (UH2NS095548, U01NS098968), the National Institute on Deafness and Other Communication Disorders (R01DC009899, U01DC017844, R01DC014034), Howard Hughes Medical Institute, the U.S. Department of Veterans Affairs (A2295R, N2864C), L. and P. Garlick, S. and B. Reeves, the Wu Tsai Neurosciences Institute at Stanford and the Simons Foundation Collaboration on the Global Brain (543045).