Natural human communication combines speech with accompanying gestures. Now imagine losing both. Patients with certain neurological disorders lack this very rudimentary ability. Amyotrophic lateral sclerosis (ALS) and stroke patients face this burden, losing both speech and nonverbal movement, which reduces their ability for natural communication.

Treatment plans available now include implants, also known as brain computer interfaces (BCIs), which translate neurological activity from the sensorimotor cortex — a region of the brain that takes sensory input to motor commands — into commands for external devices. However, these treatment plans exclusively focus on decoding speech and body movements separately. Allowing an implant to simultaneously decode both speech and movement would return natural communication and some normality into the patient's lives.

Researchers from the UCSF Chang Lab were aware of this issue and wanted to see if a single brain-computer interface implant could reliably decode both speech and gestures when attempted simultaneously. Unfortunately, there were certain constraints identified in this experiment. Through previous studies attempting to tackle this issue, they found that these behaviors can interfere with each other neurally. For example, speech degraded cursor control when performed simultaneously, as well as other detailed neural functions. With this in mind, the researchers decided to approach this issue by using a different approach: high density electrocorticography (ECoG) implant arrays. These implants were expected to reduce this overlap beyond interfaces patients had previously used.

The researchers recorded neural activity from 3 patients (Bravo-1r, Bravo-3, and Bravo 6) with severe paralysis. Bravo-1r and Bravo-3 had severe vocal tract and limb paralysis (brainstem stroke) while Bravo-6 had severe vocal tract paralysis and moderate limb paralysis (ALS). They each had an ECoG implant over the sensorimotor cortex (region of the brain that carries sensory input to control into executive motor functions). During the recording process, Bravo-1r silently attempted speech and minimally attempted other movements while Bravo-3 also silently attempted speech and attempted other movements. Since 1 and 3 had severe bodily paralysis, there was no observable output. Bravo-6 attempted speech with some vocalization but imagined other body movements without any output.

The researchers found that these implants can pick up simultaneous signals from hand, head, eye, and speech-related orofacial movements. A single ECoG implant was able to maintain multiple body systems at once, as the electrodes from the implant mimicked the natural neural mapping of the brain and organized themselves in this manner across the “grid” of the brain. The hand/arm electrodes clustered in one region while the head/eye electrodes were in another, and speech/orofacial electrodes were in a third region– consistent with the brain’s natural organization.

The coverage of the implant was quite successful, as it picked up both output and input functions. For the test part, the researchers ran three task types: speech only, gesture only, and simultaneous speech and gesture. They measured which electrodes activated during each task and found there was a consistent zone of overlap, as some electrodes lit up for both speech and nonverbal movement. However, this could create confusion within the signals and create false alarms or commands as was done in previous studies.

Instead, the researchers trained the implants in an isolated and simultaneous manner using the three tasks to create “hybrid” models. This allowed the implant to learn clearer signals to lean away from the overlapping electrodes to more specific oriented ones. Once training was completed they found that the accuracy for receiving and recognizing these simultaneous actions were 85% (gestures) and 75% (speech) for Bravo-6 and 100% (for both) for Bravo-1r and Bravo-3.

Ultimately, the researchers found that concurrent speech and gesture decoding from a single implant is possible but not automatic. The implant requires training in a multi-effector communication environment. While not a perfect solution, this raises potential for treating neurological disorders: One simultaneous step — and word— at a time.