The process of ascertaining a speaker’s emotions from their voice is called speech emotion recognition, and it is something we all do unconsciously in everyday conversation. For example, if someone says “You idiot” in a strong, decisive tone, the emotion conveyed is completely different than when the same words are uttered in a slightly pampering tone. Conversely, if we pay attention to both the meaning of words and the way they are spoken, we can more accurately understand the emotions of others. Nagase recalls that he has been interested in how people use speech to express emotions ever since he began taking courses in the teacher training program.

For Nagase, who is a member of a choir, choral instruction also affords him with an opportunity to consider emotional expression.

Since the voice is sound, and sound travels in a wave, it can be quantified using a frequency analysis. Therefore, the next step for Nagase was to use deep learning and other methods to analyze the combination of what is spoken and what is meant. After taking this class in the second year of his undergraduate program, which was taught by his current advisor, Professor Yamashita of the Spoken Language Laboratory, Nagase discovered the path he should pursue.

Human speech can be broken down into acoustic information such as pitch, resonance, and voice quality. On the one hand, spoken content is transmitted to the listener as linguistic information. In speech emotion recognition, acoustic and linguistic (i.e., semantic) information are combined to let the emotion recognition device (computer) estimate the emotional state of the speaker. You understand what someone is saying by combining the meaning expressed in words with the sounds transmitted by speech. This is a process that we do unconsciously to facilitate communication even in normal dialogue. If this process can be analyzed by computer in some form, the results could be applied to robots that interact with humans. Nagase explains:

Mr. Nagase began this research in earnest in 2021, and the first topic he tackled was speech emotion recognition using both acoustic and linguistic information. So, how can you make the emotion recognition device accurately understand speech that is easily misrecognized based on acoustic information alone? For example, when emotions are classified into the four categories of joy, sadness, anger, and calmness, acoustic information alone can easily lead to the misrecognition of joyful and angry emotions.

If humans are listening and making judgments, then the first thing you need to do is ensure that the speech that you want the emotion recognition device to correctly estimate does not contain any mistakes. To do this, Nagase compared the integrated processing of acoustic and linguistic information, which he had not yet exhaustively verified, under the same conditions to identify the best combination. This was an original approach, and Nagase's findings were accepted for publication in Acoustical Science and Technology, the journal of the Acoustical Society of Japan.

The next topic that Nagase addressed was speech emotion recognition by estimating emotional label sequences, that is, the recognition of detailed expressions of emotions that change subtly in a series of utterances. For example, take the utterance, "I'll let it slide this time, but I won't let you off the hook next time.” In conventional speech emotion recognition, generally, only one emotion label can be attached to one utterance. Therefore, if the preceding utterance were to be classified in one of the four categories of joy, sadness, anger, or calmness, it would be tagged as anger.

First off, the human voice is divided into voiced and unvoiced sounds according to the vibration of the vocal cords when pronouncing sounds. The Japanese vowels a, i, u, e, and o are basically voiced sounds that entail vocal cord vibrations. Consonants, on the other hand, are divided into voiced and unvoiced sounds. However, previous studies on speech emotion recognition focused only on voiced sounds, and even then, they did not consider the difference between vowels and consonants.

Nagase’s research paper was accepted for entry at Interspeech, the world's largest conference in the field of spoken language processing, and he also won the Outstanding Research Award from the Graduate School of Information Science and Engineering at Ritsumeikan University and the IEEE SPS Tokyo Joint Chapter Student Award. The next theme that Nagase tackled was speech emotion recognition that transcribes descriptions of the emotions conveyed by speech, an innovative method of writing that literally translates a speaker’s emotions into text.

Specifically, for the utterance, "Yay, I did it!", the device writes the following description of the emotion: "I feel excited and satisfied that I won.” In pursuing this idea, Nagase developed a new emotion recognition device that predicts emotion captions from speech by adding descriptions of emotions to existing speech emotion data and performing deep learning of them. This method transforms the very way emotions are predicted by transcribing them into text rather than relying on conventional forms of evaluation. Nagase presented the results of this research at the Spring Meeting of the Acoustical Society of Japan in March 2024.

Nagase's research is built on novel ideas, like his perspective that focuses on the differences between vowels and voiced and unvoiced consonants, which is something had never been considered in conventional speech emotion recognition studies, and his method of expressing emotions in writing. So, how did he come to take these kinds of perspectives? They key appears to lie in his attitude of placing value on error analysis.

If you simply view correct answers and errors on the same dimension, certain perspectives will go unnoticed. It is only when you elevate your judgment to the next dimension that you can gain new insights like these. When you take a fresh look at your research from a higher dimension, things that have been overlooked will emerge. This is what led Nagase to divide sounds into more granular categories and what gave him the idea to put emotions into writing rather than just labeling them. In the future, Nagase's research findings might be used to facilitate smooth communication between humans and machines in conversation robots and automatic response systems.

Sometime in the future, after Nagase's research has been further developed, perhaps people will be able to understand each other better and the conflicts that arise from minor differences in communication will be a thing of the past.