A brain implant decodes speech and gesture at the same time

The University of California, San Francisco reported in Nature Neuroscience on 14 September 2026 that one brain implant can decode speech and upper-body gesture at the same time, driving a virtual avatar that talks and moves together. The paper, “Simultaneous speech and gesture decoding for multimodal communication in paralysis”, comes from the laboratory of the neurosurgeon Edward Chang.

Three participants with severe paralysis received electrocorticography arrays — a thin strip of electrodes laid on the surface of the motor cortex rather than pushed into it. Two of them controlled the full-body avatar, with parallel decoders reading speech and gesture from the same recording.

The finding underneath the demonstration is about the cortex rather than the hardware. Brain activity during combined speech and gesture is not the sum of the two performed separately, and decoders trained on the combined behaviour outperformed decoders trained on each in isolation. “Conversation is about much more than the words being spoken,” Chang said. “It’s a multilayered, dynamic process involving the whole motor cortex.”

The system is wired. A fully implantable, wireless version — which is what use outside a laboratory would require — has yet to be tested.