Visible Coalitions of Neuronal Activities in Brain-to-Text Communication via Handwriting

The brain's cortex features complex networks composed of many individual neurons [1, 2]. Various studies have revealed that the connectivity among neurons may vary in relation to behavioral events [3-7]. In a recent study, Willett et al. [8] demonstrated decoding of imagined handwriting movements from neural activity in the motor cortex of a paralyzed patient. We analyzed their data^* by representing all neurons as raster displays and trained convolutional neural network (CNN) models to classify different brain states as the characters that the subjects imagined. Our binary classification models had an average accuracy of 96%, which we then fine-grained by training a multi-class CNN on all 31 different characters. This achieved a high success rate of 86% accuracy. Finally, we applied Grad-CAM [9] to explore the emergence of spatiotemporal patterns which are likely to be involved in determining which character the subject was imagining. Our results support the notion that dynamic neuronal correlations are involved in encoding the different characters.