Man, from all walks of life, has become dependent on modern technology. While some do express resistance to this change, it also comes with its own set of perks and benefits. Doctors, for instance, have searched for ways to help the sick. In this instance, they may have exactly what they need.
Researchers at UC San Francisco have successfully developed a “speech neuroprosthesis.” This kind of technology allowed a man with severe paralysis to communicate in comprehensible sentences. It does this by translating signals from his brain to the vocal tract directly into words, and these words are translated as text on a screen.
This achievement did not happen overnight. As with many breakthroughs, this was built from more than a decade of effort by UCSF neurosurgeon Edward Chang. It was his desire to develop a technology that allows people with paralysis to communicate even when their bodies and brains are limited to do such things.
Chang, the senior author of the study, shared, “To our knowledge, this is the first successful demonstration of direct decoding of full words from the brain activity of someone who is paralyzed and cannot speak. It shows strong promise to restore communication by tapping into the brain’s natural speech machinery.”
Thousands of people lose the ability to speak each year. This is brought about by unforeseen circumstances such as a stroke, an accident, or a result of a disease. With deeper and more elaborate developments made, the approach made in this study could one day enable the paralyzed individuals to fully communicate with the people around.
The Ability to Translate Brain Signals into Speech
The previous work in the field of communication neuroprosthetics was focused on restoring communication through spelling-based approaches. This requires the user to type out letters one at a time. This is where Chang’s study is critically and significantly different: his team is translating signals intended to control muscles of the vocal system for speaking words, rather than signals to move the arm or hand, which requires typing.
Chang’s new approach focuses on the natural and fluid aspects of speech, which promises more rapid and organic communication for the user. He notes that the spelling-based approach that makes use of typing, writing, and controlling a cursor is definitely more laborious. It is also much slower. He shared, “With speech, we normally communicate information at a very high rate, up to 150 or 200 words per minute. Going straight to words, as we’re doing here, has great advantages because it’s closer to how we normally speak.”
Over the last ten years, Chang’s progress toward this goal was facilitated by patients at the UCSF Epilepsy Center. They were those who underwent neurosurgery in order to pinpoint the origins of their seizures using electrode arrays that were placed on the surface of their brains. These said patients, all of whom had normal speech, volunteered to have their brain recordings analyzed to see where speech-related activity happens. The early findings of the study were successful and the volunteers were able to pave the way for the current trial for patients who suffer from paralysis.
Previously, Chang and his colleagues in the UCSF Weill Institute for Neurosciences had mapped the cortical activity patterns associated with vocal tract movements that produce every consonant and every vowel. In order to translate those findings into speech recognition of full words, David Moses, PhD, a postdoctoral engineer in the Chang lab, had come up with new methods for real-time decoding of those patterns and statistical language models in order to improve its accuracy.
While their success in decoding speech in participants who were able to speak didn’t guarantee that the technology could actually work in a person whose vocal tract had been paralyzed, Moses believes in it and said, “Our models needed to learn the mapping between complex brain activity patterns and intended speech. That poses a major challenge when the participant can’t speak.”
Moreover, the team didn’t know whether brain signals controlling the vocal tract could still be in working condition especially if they haven’t been able to move their vocal muscles for many years. Moses further explained, “The best way to find out whether this could work was to try it.”
The First 50 Words
The researchers also wanted to further look into the potential of this kind of technology in patients who are paralyzed. Hence, Chang partnered with colleague Karunesh Ganguly, an associate professor of neurology, to launch a another study which they aptly named BRAVO, which is an acronym for Brain-Computer Interface Restoration of Arm and Voice.
The first participant in the trial was a man in his late 30s. He had suffered a devastating brainstem stroke more than 15 years before. The incident severely damaged the connection between his brain and his vocal tract, as well as his limbs. Since his injury, he has had extremely limited head, neck, and limb movement. He was forced to communicate using a pointer that was attached to a baseball cap in order to poke letters that were found on a screen.
The participant, who asked to be referred to as BRAVO1, worked closely with the researchers. They had managed to create a 50-word vocabulary that Chang’s team could recognize from the brain activity. They had utilized advanced computer algorithms to make this possible. The vocabulary included words such as water, family, and good. This was comprehensive enough to allow them to create hundreds of sentences expressing concepts that were commonly used in BRAVO1’s daily life.
In order to pull this off, Chang surgically implanted a high-density electrode array. He had placed this over BRAVO1’s speech motor cortex. Once the patient had fully recovered, his team recorded 22 hours of neural activity in this brain region. This needed time. A total of 48 sessions and several months were accumulated in the study. As for each session, BRAVO1 endeavored to say each of the 50 vocabulary words many times over as the electrodes recorded the brain signals stemming from his speech cortex.
The Act of Translating Attempted Speech into Text
To make translating the patterns of recorded neural activity into specific intended words possible, the other two lead authors involved in the study used custom neural network models. These are different forms of artificial intelligence. When BRAVO1 attempted to speak, these networks worked to distinguish the subtle patterns in the brain activity to detect speech attempts and to identify which words he was striving to speak out loud.
They then tested their approach. But first, the team presented BRAVO1 with short sentences that had been constructed from the 50 vocabulary words. They asked him to try saying them at several tries, and as he made his attempts, the words were decoded from his brain activity, one at a time on a computer screen. Then the team switched their methods of prompting him by asking questions such as, “How are you today?” and “Would you like some water?” As done before this, BRAVO1’s attempted speech appeared on the screen. His answers involved sentences such as “I am very good,” and “No, I am not thirsty.”
The findings were hopeful because the team found that the system was able to decode words from brain activity at rate of up to 18 words per minute and with up to 93 percent accuracy (with a 75 percent median).
Much of the success can be credited to the language model Moses had applied, and one that implemented an “auto-correct” function, similar to what has been commonly used by consumer texting and speech recognition software. Moses characterized the early trial results as proof that the principle does work. His statement and the details of the study appear in the New England Journal of Medicine. He said, “We were thrilled to see the accurate decoding of a variety of meaningful sentences. We’ve shown that it is actually possible to facilitate communication in this way and that it has potential for use in conversational settings.”
Looking towards the future, Chang and Moses said they will expand the trial. They aim to include more participants affected by severe paralysis and communication deficits. At this point in time, the team is currently working to increase the number of words in the available vocabulary of the technology. They’re also looking into improving the rate of speech.
Chang and Moses also said that while the study focused on a single participant and a limited vocabulary, those limitations don’t diminish the accomplishment they had made. Moses explained, “This is an important technological milestone for a person who cannot communicate naturally, and it demonstrates the potential for this approach to give a voice to people with severe paralysis and speech loss.”
Recommended For You
