Neuroprosthesis for paralysis enables simultaneous speech and body language.
An NIH-funded team at the University of California, San Francisco, reported a brain-computer interface that let people with paralysis convey speech and upper-body gestures at the same time through a full-body avatar.
A team funded by the National Institutes of Health has shown that a brain-computer interface (BCI) can allow people with paralysis of the vocal tract and body to communicate through speech and upper-body gestures simultaneously. According to NIH, earlier BCIs enabled one or the other, and this system is the first to enable both at once.
Researchers at the University of California, San Francisco, used machine learning to decipher the brain activity underlying concurrent speech and physical gesture in three patients. The BCI translated the thoughts of two participants into commands that controlled the expressions of a full-body virtual avatar.
The team had previously implanted a thin strip of sensors, called an electrocorticography array, on the motor cortex of several patients. Computer models known as decoders translated those brain signals into commands for a digital head and face. In the new study, participants were connected to an avatar of a complete body. Data were collected while they attempted to verbalize specific phrases or perform common gestures such as a hand wave or thumbs-up, both separately and together.
The researchers found that the brain signals for simultaneous speech and gesture were not simply a combination of the two separate signals. Despite some overlap, the pattern was very different. Decoders were better at reading mixed expressions when they had been trained on data recorded during simultaneous speech and gesture rather than isolated attempts.
"Conversation is about much more than the words being spoken," said corresponding author Edward Chang, M.D., a professor of neurological surgery at UCSF. He described the work as a proof of concept. The system used in the study was wired, connecting implanted sensors to external processing units. Chang said his team will soon test a fully implantable, wireless version that may be better suited to long-term use.
The findings were published September 14, 2026, in Nature Neuroscience.
At a Glance
- Three patients took part in the study, and the BCI translated the thoughts of two into commands for a full-body avatar.
- Edward Chang, M.D., of the University of California, San Francisco, was the corresponding author. The work was funded by NIH.
- Decoders trained on simultaneous speech and gesture data performed better on mixed expressions than those trained on isolated data.
- The study appeared in Nature Neuroscience on September 14, 2026.
- The current system is wired. A fully implantable, wireless version is planned for testing.
Sources
This brief was written by Brightside Report from the reporting linked above. Spot an error? Request a correction.