“Overnight, everything was taken from me,” Ann wrote, thinking back on the stroke that had left her paralyzed and voiceless almost two decades ago. Ann was 30 when the stroke struck, leaving her life forever altered, but now her strength and resilience have given rise to a revolutionary medical breakthrough: a brain-to-voice neuroprosthesis that allows real-time speech synthesis from neural activity. This research technology is upending potential for people with severe paralysis, holding the promise to give back their voice and access to the world.
The UC Berkeley and UC San Francisco BCI developed by scientists is an engineering and neuroscience marvel. Artificial intelligence (AI) is used to decode brain neural signals in the motor cortex region of the brain that generates speech into words. With 80-millisecond snapshots of brain activity, the system decodes thoughts into words as quickly. That, said co-principal investigator Gopala Anumanchipalli, “Our streaming approach brings the same rapid speech decoding capacity of devices like Alexa and Siri to neuroprostheses.” “Using a similar type of algorithm, we found that we could decode neural data and, for the first time, enable near-synchronous voice streaming.”
That answer overcomes a decades-long barrier to speech neuroprostheses: latency. Other systems, having been used earlier, had to wait until sentences were constructed prior to output being made, a process that was eight seconds or longer. Interrupting pauses marred the communication process and usually annoyed people into abandoning the machine. The new streaming method, however, gives output after thoughts have been set up in one second, and thus flowing and uninterrupted communication. As co-author of the research Kaylo Littlejohn describes, “We can see relative to that intent signal, within 1 second, we are getting the first sound out. And the device can continuously decode speech, so Ann can keep speaking without interruption.”
Scientists worked hard to train the AI software in the system. The process was making the subject gradually attempt to read sentences from a 1,024-word typical word lexicon with the electrodes placed inside her brain picking up brain waves. The AI system translated the signals into phonemes the speech constructing blocks. The process meant that the system would only have to learn 39 phonemes to translate any English word better and faster. The researchers also recorded Ann’s voice before she was injured so that her synthesized speech would be recognizable as hers a very personal touch that helped her feel embodied.
The implications of this technology extend far beyond Ann’s individual case. The system has proven to be adaptable with a variety of brain-sensing interfaces, including noninvasive ones like surface electromyography (sEMG). “By demonstrating accurate brain-to-voice synthesis on other silent-speech datasets, we showed that this technique is not limited to one specific type of device,” Littlejohn said. This flexibility has the potential for additional uses for millions of people afflicted with such conditions as ALS, traumatic brain injury, and locked-in syndrome.
While an dazzling achievement, the system has limitations. Even though the system can process 90.9 words per minute with a limited vocabulary, the system’s accuracy decreases if there are additional vocabularies or unknown words. When, for example, the system was tested with new vocabularies using the NATO phonetic alphabet, the accuracy was 46%. Researchers remain hopeful that they can still enhance the technology. “This proof-of-concept framework is quite a breakthrough,” co-senior author Cheol Jun Cho maintained. “We are optimistic that we can now make advances at every level. On the engineering side, for example, we will continue to push the algorithm to see how we can generate speech better and faster.”
Future research will attempt to restore expressiveness to the synthesized voice, preserving tone, pitch, and loudness and the emotion and personality that these impart in real speech. “That’s ongoing work, to try to see how well we can actually decode these paralinguistic features from brain activity,” Littlejohn said. Such an advance would close the gap to full and total naturalism, restoring not just the ability to communicate but the richness of human interaction.
To Ann, the technology is not just an engineering feat it’s a recovery of self and agency. Her daughter, who had never known her mother’s voice as anything but a British accent through a computer, can now hear speech that sounds like her own. As Anumanchipalli explained, “Hearing her own voice in near-real time increased her sense of embodiment.”
While the system, as yet experimental and limited to the lab, is promising in its potential, it is but a beginning. With additional funding and refinement, scientists envision homes within a decade with the technology. For the millions of men and women incapacitated by paralysis, the technology promises not only a voice but a new life to life beyond their paralysis.
