Interaction through Integration of Speech and Image
Satoru Hayamizu
Machine Understanding Division,Electrotechnical Laboratories
Up to now the Electrotechnical Laboratories has conducted researches on speech, image and natural language at each research section and the Real World Computing (RWC) Project Team was established as an organization that spans over those research areas for the research on information integration and learning.
The research I introduce here aims at the integration of speech and image information in interactions with particular emphasis on "the information integration and real-world-ness", inheriting the research result of individual sections and the RWC Project Team. It is timely that the Electrotechnical Laboratories introduces the laboratory system when RWC program is moving from the first half into the second half.
Many problems need to be solved before accomplishing a natural interaction between humans and artifacts (artificial systems). Artifacts assumed here is to simulate the expressions and behavior of humans using computer graphics to synthesize humans' faces and bodies. It is a personified agent by simulating human eyes, ears and mouths with image and speech recognition and speech synthesis.
The technological target is to implement natural interactions with humans by integrating speech recognition, speech synthesis, image recognition and image synthesis.
When such artifacts capable of carrying out interaction with humans is implemented , these artificial systems can conduct such simple jobs of receptionists and looking for people to deliver messages.
In our daily life, we often use speech and image information when we communicate information. Voice is used to deliver verbal information and gestures and expression non-verbal information. Image information used by humans for communicating information is to show objects themselves or pointing to objects with fingers.
We are now conducting research on how speech and image information should be used in interactions between humans and artifacts.
The target of our research and development involves how to integrate information in an interaction using speeches and images and how to deal with information that is hard to describe in words to carry out a natural interaction.
There are a number of types for integrating speech and image information. For instance, we could develop new functions or improving the performance by combining two kinds of information.
Human gestures and facial expressions may be difficult to communicate to others in words. Rhythm and timing of utterances are also hard to express in words. One of the important problems is to develop a technology to be used for properly handling such interactions.
Another study is required to deal with how to carry out these interactions in conversation using language information in order to accomplish a natural conversation.
Fig. 1 shows the configuration of the prototype multimodal interaction system we developed in last fiscal year which integrated speech recognition, speech synthesis and image recognition and image synthesis. It is an integration of spoken dialogue function, face and object recognition function and presentation function for facial expression and gesture by upper half of the body. With these functions the system can recognize and talk to people, allow them to leave messages such as time to return when they are out, or notify them of the arrival of e-mail.
To implement a natural interaction between humans and a system using speech and image recognition, developers do not need to write everything the system must do. Instead, they need to instruct the system how speeches and images are used in the dialogue through interactions consisting of speech and image information. In other words, it will be necessary to use speeches and images acting as mutual supervising signals to instruct the other of the use of speeches by images, or vice versa.
We call this "Intermodal learning". It is a key concept for a natural interaction using speech and image information. It is also important to make it possible to use patterns as they are in addition to symbolizing speech and image information.
A technology for recognizing animated images is required for handling human gestures and facial expressions, as well as an action to show things. The proper handling of "time" is also important for the integration of speech and image information.
Researches are being conducted for a theoretical analysis of the information integration and building an experimental system as a testbed and evaluation of the effectiveness with actual data and interaction processing for a natural interaction using speeches and images.
For evaluation of the effectiveness with actual data, we need to collect a large amount of speech and image information found in such interactions. We are now building RWC multimodal database by setting up the Multimodal Database Committee under the Database Workshop with the cooperation from related research organizations.
We will make efforts to develop technology that will be actually useful through such a demonstrative research using a large amount of actual data.