Using brain waves to reconstruct human activity

Technology that creates the world

Brain waves^[1] are weak electrical signals generated when the brain is active. The development of brain-machine interfaces (BMIs), which use brain waves or cranial nerve-derived signals to connect the brain to a machine and operate it at will, is well underway around the world. Professor Natsue Yoshimura is using AI to decipher and reproduce body movements and heard speech that are encoded in brain waves. Welcome to the realm of technology that sifts through vast amounts of biological signals to unravel the activity of the brain.

(Yoshimura) "BMI technology, which measures brain activity signals and uses this information to connect the brain to a computer, can be divided into two types in terms of invasiveness, i.e., burden on the body: invasive types that involve craniotomy and non-invasive types that do not require surgery. Invasive types can measure signals directly from a specific region of the brain, and thus have higher precision in extracting brain information. There is a lot of competition in their development, especially in Europe and the United States. In contrast, non-invasive types pose low risk to the human body, but are not precise enough to decipher detailed information. Non-invasive BMIs utilize methods such as MRI, near-infrared spectroscopy, and electroencephalography (EEG) for measurement. But measurement and analysis are generally more difficult than with invasive methods, and it has been considered almost impossible to extract detailed brain information from scalp EEG in particular.

However, I have been conducting research to expand the possibilities of non-invasive methods in the hopes of making it easier for everyone to monitor their own brain condition. The main method used is to estimate neural activity in the brain from EEG using machine learning and extract information. As a result of persistent, repeated analyses, in 2021 we succeeded in reproducing the pronunciation of two vowels, 'a' and 'i,' heard once and recalled in the brain by a human, as sounds recognizable by a third party, from EEG recorded using electrodes attached to the scalp."

The human brain is composed of approximately 100 billion neurons. When stimulated by sensory organs, neurons transmit electrical signals to each other to process information, and brain waves are the result of this electrical activity. It is one of the human biological signals, like pulse, heartbeat, electrocardiogram, and electromyogram –which are signals emitted by muscle contractions. In 1875, British scientist Richard Caton first reported electrical activity in the animal brain, and in 1924, German psychiatrist Hans Berger finally recorded an active current from the human cerebral cortex. Five years later, Berger published his work, naming this brain activity an electroencephalogram.

EEG has a relatively fast response time after the body is stimulated, and has the advantage that it can be measured from the scalp. There have been many attempts to analyze the data, but it has not gotten as far as practical use. The reason lies in the difficulty of non-invasive EEG measurement and analysis. When measuring EEG from electrodes attached to the scalp, large noises that are difficult to remove during the measurement phase, such as muscle activity and radio waves from surrounding electronic devices, are included, preventing the extraction of these very small amplitude signals. Furthermore, the EEG waveform rules are not as clear-cut as those for pulse and heartbeat, making it extremely difficult to discern EEG that represents specific information. Since the 2000s, there has been a movement to use AI to analyze EEG, but the aforementioned barriers still stand in the way of researchers.

As a pioneer in the analysis of non-invasive brain data, Yoshimura has worked on decoding information on movement, hearing, speech, and emotion, and has successfully extracted muscle activity, finger movements, and many other physical behaviors from EEG. In particular, Yoshimura's announcement in 2021 of "a technology to reproduce sounds someone has heard or recalled from EEG" suggests the possibility of advancing brain science toward elucidation of functional brain areas and paving the way for development of future applications such as BMIs. In this study, a novel technique was presented that uses machine learning and deep learning to estimate parameters that represent the frequency characteristics of speech from EEG when the two vowels "a" and "i" are heard, and then reconstructs the sound from these parameters. At the time, there were almost no examples of speech reconstruction that were clear enough for a third party to hear and distinguish, even with the invasive methods. So it is an astounding finding that speech reconstruction, at about 80% accuracy, is possible using non-invasive methods.

"The low spatial resolution^[2] that had been a problem in extracting information from EEG was improved through AI-powered computational processing, resulting in the reproduction of speech. And regarding the need to provide a large number of samples to use deep learning, it is also possible to employ new data expansion techniques that generate fluctuations^[3] from a single source data. In the past, it was common to simply identify whether the restored sound was an 'a' or an 'i.' However, the possibility of reproducing the speech waveform itself and capturing it as a distinct sound has been verified, and this could lead to the reproduction of speech in the mind, which differs from person to person. We are currently working on developing a machine learning model that can reproduce speech sounds other than just 'a' and 'i.' Furthermore, the deep learning we employed this time is not a black box with unknown analysis principles. It can be used to determine where in the brain the pathway to hearing and recalling speech is located. Although the language areas used for human speech has been largely identified, there are large individual differences, making it difficult to pinpoint the exact location of these areas. However, the advantage of non-invasive measurements that can broadly estimate signals throughout the brain should allow us to evaluate auditory, speech and language processing in the brain in the future." (Yoshimura)

The medical field is leading the way on applied research on BMIs. From the early stages of their emergence, BMIs have been used to support the movement and rehabilitation of quadriplegic patients, and as a means of communicating intention for amyotrophic lateral sclerosis (ALS) patients. Meanwhile, braintech, which utilizes BMIs in various industries to open up new business domains, has rapidly attracted attention. Braintech is a coined word that combines brain and technology. Brain science is becoming more accessible every day. But most are invasive BMIs and other devices implanted in the brain. The non-invasive technology developed by Yoshimura will benefit not only medicine but society as a whole because of its safety.

"I imagine a future in which anyone can easily check the condition of their brain by applying information extraction from EEG, which we have been researching – to enable individuals to detect early on a decline in brain function and to care for it in a game-like manner. The extraction of motion information could also be applied to the entertainment domain. We actually get requests from people who want to control VR with their brains instead of a controller." (Yoshimura)

Yoshimura Lab does not reject any ideas from students, but rather transforms that inner curiosity into energy. The intellectual and organic connections they make are like the EEG signals they study. They emerge and intersect, and the full picture is unfathomable. The possibilities for the future are tremendous.

Published: March 2024