Possibility of machines that support human mind through mutual man-machine exchange of information

Have you ever felt as if you were "united" with someone beside you, perhaps even whose name you do not know, when you attended a live concert of your favorite artist, or watched a sports team play in a stadium, or accomplished a difficult task together? Although the mind is intangible and we cannot literally connect our minds, we sometimes feel an affinity to the person with whom we share an emotional connection, as if our minds took a few steps closer to each other. With this forethought, Shima Okada, Professor at the College of Science and Engineering at Ritsumeikan University aims to develop a communication robot by quantifying the state of mind of people when they communicate with one another to encourage the creation of relationships with other people and groups. As a step toward this goal, Okada is currently working on a "kokoro (mind) distance measure" that quantifies the intimacy or distance a person feels towards another person.

The idea of using machines to support people in need is not new. Assistive suits, which help people with limited muscle strength to lift heavy objects, and robotic arms, which allow people with paralyzed limbs to move like their own using nerve signals, are already being utilized in medical and welfare fields. According to Okada, such machines that support human motor functions have been made possible by observing, measuring, and quantifying human movements and reactions. For example, to reproduce the motion of grasping an object, it is necessary to capture the visual sensory signals indicating the presence of the object, the central nervous system's decision to pick it up, the instructions transmitted through the motor nerves, as well as the muscle movement, using multiple sensors including electroencephalography, magnetic resonance (MR), electromyography, and inertial sensors. In other words, a thorough understanding of the functions of living organisms makes it possible to design a machine that systematically reproduces their actions.

Now that cybernetics research has shown that machines can support the human body, is there a possibility that machines can also support the human mind? Artificial intelligence, which has made remarkable progress in recent years, can be interpreted as an attempt to "create machines with human minds" by having machines reproduce the information processing and thought processes of the human brain. However, Okada is concerned that simply creating human-like machines will not allow humans and machines to be truly compatible with each other. "Cybernetics is about making machines work based on biological signals to support or replace the human body; artificial intelligence is about using information to make machines learn, with the goal of making them behave as if they were human. In both cases, the exchange of information is not bidirectional. How can we achieve a two-way exchange of information from person to machine and vice versa and create a machine that supports the human mind like cybernetics supports the human body," ponders Okada.

Quantifying the degree of empathy by the synchronization of pulse rhythms

Okada admits that the kind of logic driving the human mind cannot be ascertained, but it is possible to observe the state of the mind at any given moment and process the results numerically. "Vital signals such as heartbeat, facial capillary contraction, and sweating change with the state of mind. Prior research indicates that when people participate in certain religious ceremonies together, their heartbeat rhythms become synchronized," Okada points out. She hypothesized that the synchronization of the autonomic nervous system of the participants in an activity could be used as an indicator to measure whether they empathize with one another, or whether they feel getting closer to each other in terms of mental distance. This is the idea behind the kokoro distance measure.

To prove this hypothesis, Okada called 10 students from Ritsumeikan University who met each other for the first time that day. She asked them to wear wearable sensors and camp together in her absence. She also instructed them to participate in three activities during camping: setting up a tent, cooking, and lighting a campfire, and monitored their heartbeats during those activities. "We focused on the correlation of rhythms as an indicator of the extent to which the pulse rhythms of the participants are synchronized," explains Okada. "A correlation coefficient value approaching plus one indicates that the pulses are in sync, while a correlation coefficient close to minus one indicates that the pulses are out of sync. We examined the pulse correlation of all possible participant pairs and compiled them into a two-dimensional self-organization map." On the map, the participants with stronger correlations are plotted closer to each other and those with weaker correlations are plotted farther apart.

One of the 10 participants was from a different campus, and another was of a different age than the others. Those two students were plotted on the map at a significant distance from the other eight when they had just started camping and pitched their tents. One of the eight students, however, was a friend of the student from the other campus. They bonded well while cooking and eating together. The older student, who was more communicative with others, was plotted closer to the other members of the group because the student had joined them for dinner. By the time of campfire, the two students with different backgrounds were able to join others, and the 10 participants were plotted as one group. According to Okada, the interesting aspect of this experiment is that information on the students' location was not collected throughout the process. After interviewing the students to determine what they did during each activity, it was confirmed that the students who were plotted nearby on the self-organization map were indeed working together at that time. Based on these results, Okada proposes that by looking at the scatter of correlations on the self-organization map, it may be possible to measure the actual empathy and emotional distance between subjects.

Since heartbeats are involuntary movements and cannot be intentionally hidden or changed like facial expressions, this method could also be used to quantify the state of mind of people belonging to cultures that discourage the expression of emotions facially or by attitude. Another advantage is that the correlation coefficient is not affected by the gender or age of the subjects. Okada has also conducted experiments in which elderly and young people are asked to have conversations with sensors worn, and by comparing the pulse data and transcripts, she can determine what topics brought them closer together and conversely, what topics were less likely to interest their partner. She is also planning to involve pupils from the elementary school affiliated with Ritsumeikan University in an experiment in the near future.

Potential for "robot caretakers" to support building successful relationships

Now that it seems possible to detect mind distance using pulse synchronization, Okada would next like to utilize this information with machines or applications such as robots and chatbots. "Quantification is a means, not a goal," stresses Okada. "The obtained data is currently being used by human caretakers to support the subjects in building successful relationships with one another. They offer emotional support by, say, speaking warmly to the depressed subject or advising other subjects to do the same. They can support their subjects more efficiently beforehand by understanding numerically the conditions under which their minds are more likely to approach each other." Eventually, she aims to enable robots to take on this role. As for the data acquisition, Okada also hopes to develop a wearable multi-sensor that can measure not only the heartbeat but also the electrocardiogram, respiration rate, perspiration, amount of activity, and vasoconstriction response.

"I wish that I can contribute to realize a society in which deployable robots can help people communicate with one another, say for example, in educational settings, these robots can help eliminate bullying, strife, division, and isolation," says Okada. "Even among close friends and family members, there are times when we cannot understand others' feelings or find ourselves at cross purposes. We are also likely to have fundamental differences in the way of thinking with people from other countries, with different ideologies and cultures. There are also people with whom it is impossible to communicate verbally in the first place, such as babies. Some say that it is impossible for people and machines to get along when it is difficult even for people to fully understand each other. However, if we can learn what kind of words or actions make our counterpart open up to us or deteriorate the relationship, it will help us communicate with each other in a more positive way, even in situations where there is no time to get to know each other."

The COVID-19 pandemic caused more people to work from home or attend classes online, which led to a lack of human interaction and a sense of isolation. Okada believes that by developing technologies that support the building of relationships or serve as a buffer for conflicts between people, and by applying these technologies not only to relationships between individuals but also to those between larger groups, unnecessary deterioration of relationships and friction caused by cultural, religious, and other differences may decrease, possibly prompting the creation of a society with empathy and creative communication.