A brain implant has enabled people with paralysis to speak while gesturing, such as nodding or waving. These sorts of exchanges, done via an avatar, could help them communicate with more nuance.
鈥淚n our everyday life, gestures play an important role in communication,鈥� says at the University of California, San Francisco. 鈥淪aying 鈥榤aybe鈥� and nodding your head has a very different connotation than if you shake your head.鈥�
Experimental brain implants have boosted the ability to either speak or move among people with paralysis, but simultaneously haven鈥檛 been very successful. To achieve this, implants need to cover a large part of the sensorimotor cortex, a brain region involved in speech and movement, says Brosler.
Advertisement
Now, Brosler and her colleagues have tested an iPhone-sized implant in this part of the brain in two people with paralysis. The first participant, who the researchers call Bravo-1r, became paralysed after a stroke, while the second, Bravo-6, had amyotrophic lateral sclerosis, the most common form of motor neuron disease.
Both participants had lost almost all movement in their limbs. They also couldn鈥檛 speak and could only make unintelligible sounds.
First, the team recorded the participants鈥� brain activity while they repeatedly attempted to say 10 phrases 鈥� such as 鈥渉ello鈥�, 鈥渘ice to meet you鈥� and 鈥渉ow鈥檚 it going?鈥� 鈥� or perform 10 gestures, like waving, clapping and shaking their head. In some cases, the participants attempted to do these separately, while in others they tried both simultaneously.
The team then used each of the participants鈥� brain recordings to train customised machine learning models to predict their intended phrases and gestures.
To put the models to the test, the participants attempted to repeat various pairs of phrases and gestures that were displayed on a screen. The models interpreted their brain activity and used that to control the movement of an avatar resembling the participants, while the speech predictions flashed up on the screen.聽
The team chose to use avatars, rather than attempting for the individuals themselves to speak and gesture, because this offered a relatively simple way to prove that both actions could be simultaneously decoded from brain activity, says Brosler.
Bravo-1r鈥檚 intended gestures and speech were predicted with 88 and 84 per cent accuracy, respectively, while Bravo-6鈥檚 were predicted with 66 and 70 per cent accuracy. 鈥淚f those models were operating on complete chance, the accuracy would be like 9 per cent,鈥� says Brosler. 鈥淚t was really exciting.鈥�
鈥淚t鈥檚 a very nice piece of work, with robust results,鈥� says at the University of Lausanne in Switzerland. But it鈥檚 based on a very limited set of phrases and gestures, and these still need improving, he says. 鈥淔or daily use without frustration, there needs to be higher accuracy.鈥�
Brosler says the team is working to address this by further training the models and tweaking the algorithms. She also says that future models could move a person鈥檚 own body 鈥� if their joints and muscles are in a good enough condition 鈥� and produce a synthetic voice, rather than having to do this via an avatar.
Nature Neuroscience
