Prospective Students

Using brain waves to reconstruct human activity

Professor Natsue Yoshimura, School of Computing

Using brain waves to reconstruct human activity - Professor Natsue Yoshimura, School of Computing

Technology that creates the world

Brain waves[1] are weak electrical signals generated when the brain is active. The development of brain-machine interfaces (BMIs), which use brain waves or cranial nerve-derived signals to connect the brain to a machine and operate it at will, is well underway around the world. Professor Natsue Yoshimura is using AI to decipher and reproduce body movements and heard speech that are encoded in brain waves. Welcome to the realm of technology that sifts through vast amounts of biological signals to unravel the activity of the brain.

Reproducing a sound in your mind from brain waves

(Yoshimura) "BMI technology, which measures brain activity signals and uses this information to connect the brain to a computer, can be divided into two types in terms of invasiveness, i.e., burden on the body: invasive types that involve craniotomy and non-invasive types that do not require surgery. Invasive types can measure signals directly from a specific region of the brain, and thus have higher precision in extracting brain information. There is a lot of competition in their development, especially in Europe and the United States. In contrast, non-invasive types pose low risk to the human body, but are not precise enough to decipher detailed information. Non-invasive BMIs utilize methods such as MRI, near-infrared spectroscopy, and electroencephalography (EEG) for measurement. But measurement and analysis are generally more difficult than with invasive methods, and it has been considered almost impossible to extract detailed brain information from scalp EEG in particular.

However, I have been conducting research to expand the possibilities of non-invasive methods in the hopes of making it easier for everyone to monitor their own brain condition. The main method used is to estimate neural activity in the brain from EEG using machine learning and extract information. As a result of persistent, repeated analyses, in 2021 we succeeded in reproducing the pronunciation of two vowels, 'a' and 'i,' heard once and recalled in the brain by a human, as sounds recognizable by a third party, from EEG recorded using electrodes attached to the scalp."

The desire to know the brain was my starting point and driving force

"The brain processes stimuli from the outside world to move the body, think, and generate emotions. How do we interpret these mechanisms and reproduce them with technology? It is a field without limits that could change the way we live."

Natsue Yoshimura

Natsue Yoshimura

Natsue Yoshimura

Department of Computer Science, School of Computing,
Professor

Laboratory Website(External site)

Researcher Profile(External site)

After receiving her undergraduate degree, she worked for several companies before completing the master's program at the Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University in 2006. In 2009 she received her doctorate in engineering from the University of Electro-Communications in 2009. After joining Tokyo Tech, she served as Assistant Professor at the Gender Equality Center and Associate Professor at the Precision and Intelligence Laboratory and Institute of Innovative Research before assuming her current position in 2023. She is a board member of the Japanese Society for Motor Control and a member of the Japanese Society for Medical and Biological Engineering and the Japan Neuroscience Society.

(Yoshimura) "While most BMI research has focused on establishing technology that allows people with arm or leg impairments to operate machines with their brains and obtain higher quality of life, we believe that non-invasive BMIs can be applied to the health care of persons without impairments, because of their low risk. If men and women, young and old, can monitor their brains with the ease of a thermometer and quickly detect problems with their bodies and minds, it will contribute significantly to the creation of an environment in which people can live independently in an era of 100-year lifespans.

My motivation for research is the urge to see inside the brain, to see what has never before been seen. With this single-minded determination, I left industry and decided to learn brain data analysis and programming from scratch. It was 10 years after I graduated from college. Naturally, I had some difficulties. But in a sense, it was precisely because I was an outsider and had no preconceived notions about the difficulty of extracting information from EEG that I was undaunted by the challenge, which led to the results we are seeing today. I will continue to follow my intellectual desires without making assumptions about my own potential, and will continue to unravel the information processing of the brain, which is still an unknown world."

Four ways to understand how the brain processes information

MRI scan [Non-invasive method that places less burden on the body]

Using a type of MRI called functional magnetic resonance imaging (fMRI), the head is scanned to visualize brain activity. This method is used at Yoshimura Lab.

Using a type of MRI called functional magnetic resonance imaging (fMRI), the head is scanned to visualize brain activity. This method is used at Yoshimura Lab.

Using a type of MRI called functional magnetic resonance imaging (fMRI), the head is scanned to visualize brain activity. This method is used at Yoshimura Lab.

Using a type of MRI called functional magnetic resonance imaging (fMRI), the head is scanned to visualize brain activity. This method is used at Yoshimura Lab.

An opening is made in the skull and a millimeter-scale electrode is implanted in a specific part of the brain to measure the electrical signals emitted from neurons.

Electrode sheet placed on the surface of the brain [Invasive method involving surgery]

The skull is surgically opened and a sheet several centimeters in size with numerous electrodes is placed on the surface of the brain to measure signals from the cerebral cortex.

Humans have been searching for a "map of the brain" to determine which parts of the brain are responsible for which activities. Information processing in the brain is the last huge frontier of the human body.

The human brain is composed of approximately 100 billion neurons. When stimulated by sensory organs, neurons transmit electrical signals to each other to process information, and brain waves are the result of this electrical activity. It is one of the human biological signals, like pulse, heartbeat, electrocardiogram, and electromyogram –which are signals emitted by muscle contractions. In 1875, British scientist Richard Caton first reported electrical activity in the animal brain, and in 1924, German psychiatrist Hans Berger finally recorded an active current from the human cerebral cortex. Five years later, Berger published his work, naming this brain activity an electroencephalogram.

EEG has a relatively fast response time after the body is stimulated, and has the advantage that it can be measured from the scalp. There have been many attempts to analyze the data, but it has not gotten as far as practical use. The reason lies in the difficulty of non-invasive EEG measurement and analysis. When measuring EEG from electrodes attached to the scalp, large noises that are difficult to remove during the measurement phase, such as muscle activity and radio waves from surrounding electronic devices, are included, preventing the extraction of these very small amplitude signals. Furthermore, the EEG waveform rules are not as clear-cut as those for pulse and heartbeat, making it extremely difficult to discern EEG that represents specific information. Since the 2000s, there has been a movement to use AI to analyze EEG, but the aforementioned barriers still stand in the way of researchers.

Reproducing sound from EEG

As a pioneer in the analysis of non-invasive brain data, Yoshimura has worked on decoding information on movement, hearing, speech, and emotion, and has successfully extracted muscle activity, finger movements, and many other physical behaviors from EEG. In particular, Yoshimura's announcement in 2021 of "a technology to reproduce sounds someone has heard or recalled from EEG" suggests the possibility of advancing brain science toward elucidation of functional brain areas and paving the way for development of future applications such as BMIs. In this study, a novel technique was presented that uses machine learning and deep learning to estimate parameters that represent the frequency characteristics of speech from EEG when the two vowels "a" and "i" are heard, and then reconstructs the sound from these parameters. At the time, there were almost no examples of speech reconstruction that were clear enough for a third party to hear and distinguish, even with the invasive methods. So it is an astounding finding that speech reconstruction, at about 80% accuracy, is possible using non-invasive methods.

"The low spatial resolution[2] that had been a problem in extracting information from EEG was improved through AI-powered computational processing, resulting in the reproduction of speech. And regarding the need to provide a large number of samples to use deep learning, it is also possible to employ new data expansion techniques that generate fluctuations[3] from a single source data. In the past, it was common to simply identify whether the restored sound was an 'a' or an 'i.' However, the possibility of reproducing the speech waveform itself and capturing it as a distinct sound has been verified, and this could lead to the reproduction of speech in the mind, which differs from person to person. We are currently working on developing a machine learning model that can reproduce speech sounds other than just 'a' and 'i.' Furthermore, the deep learning we employed this time is not a black box with unknown analysis principles. It can be used to determine where in the brain the pathway to hearing and recalling speech is located. Although the language areas used for human speech has been largely identified, there are large individual differences, making it difficult to pinpoint the exact location of these areas. However, the advantage of non-invasive measurements that can broadly estimate signals throughout the brain should allow us to evaluate auditory, speech and language processing in the brain in the future." (Yoshimura)

Progress in EEG measurement through "EEG caps"

EEG cap

When measuring EEGs from the scalp, a cap with multiple electrodes, called an EEG cap, is often used. Yoshimura Lab primarily uses EEG caps with 64 electrodes, and a special gel is inserted between the electrodes and the scalp to ensure contact and improve the quality of the EEG signal to be measured. Not long ago, preparation involving scraping the sebum from the scalp was essential to obtain an accurate signal. But now, taking into account the comfort of the experiment participants, a variety of EEG caps have been developed to make them easier to use. Research is also underway to miniaturize devices while maintaining measurement precision, and there are remarkable technological innovations in hardware as we move toward a future in which individuals will be able to easily utilize EEG.

Progress in EEG measurement through "EEG caps"

The promise of non-invasive BMIs

The medical field is leading the way on applied research on BMIs. From the early stages of their emergence, BMIs have been used to support the movement and rehabilitation of quadriplegic patients, and as a means of communicating intention for amyotrophic lateral sclerosis (ALS) patients. Meanwhile, braintech, which utilizes BMIs in various industries to open up new business domains, has rapidly attracted attention. Braintech is a coined word that combines brain and technology. Brain science is becoming more accessible every day. But most are invasive BMIs and other devices implanted in the brain. The non-invasive technology developed by Yoshimura will benefit not only medicine but society as a whole because of its safety.

"I imagine a future in which anyone can easily check the condition of their brain by applying information extraction from EEG, which we have been researching – to enable individuals to detect early on a decline in brain function and to care for it in a game-like manner. The extraction of motion information could also be applied to the entertainment domain. We actually get requests from people who want to control VR with their brains instead of a controller." (Yoshimura)

Yoshimura Lab does not reject any ideas from students, but rather transforms that inner curiosity into energy. The intellectual and organic connections they make are like the EEG signals they study. They emerge and intersect, and the full picture is unfathomable. The possibilities for the future are tremendous.

"The evolution of EEG analysis" has expanded the options for experimental tasks

In EEG experiments, researchers have struggled with how to eliminate noise in the experimental phase due to how small the signals are. In the past, it was customary to perform simple experimental tasks while having the participant avoid blinking. But with the development of analytical methods to separate noise from EEG, this practice is becoming a thing of the past. Yoshimura Lab is currently working to understand the information processing mechanisms in the brain related to motor control by examining EEG during juggling training.

Exercise experiment using VR goggles
Exercise experiment using VR goggles

Exercise experiment using liquid crystal shutter goggles
Exercise experiment using liquid crystal shutter goggles

Monitoring with sensors attached to the body
Monitoring with sensors attached to the body

Terms

[1] Brain waves : Brain waves are wave-like signals produced by the brain. They can be broadly classified into two types: those measured from the cerebral cortex and those measured from the scalp. The latter are measured with electroencephalography (EEG). When people refer to "brain waves," they are typically referring to EEG.

[2] Spatial resolution (in EEG measurement) : A measure of how precisely signals generated from specific brain regions or sources of neural activity are reflected in signals measured on the scalp.

[3] Data fluctuation : Predictive uncertainty or variability among data. Here it refers to data with different patterns within the range that the AI can recognize as the same type of information.

School of Computing

School of Computing
—Creating the Future Information Society—

Information on School of Computing inaugurated in April 2016

School of Computing

Schools, Departments, and Institute for Liberal Artsouter

Published: March 2024

Contact

Public Relations Division, Tokyo Institute of Technology

Email pr@jim.titech.ac.jp