Information Integration Interactive Laboratory of RWCP Tsukuba Research Center (TRC) produced the following two press releases on recent research results:
(1) Development of a Method of Summarizing Topics in Conversations and
Meetings by "Browsing"
(2) Real-Time Integration System for Speech and Gestures of Simultaneous
Speakers
The text of these press releases is given below.
Development of a Method of Summarizing Topics in Conversations and Meetings by "Browsing"
Technical Capability
We often record daily conversations, meetings, and discussions on tape. The intention behind these recordings is to replay the contents later in order to summarize them or retrieve important information. However, when the recording time is too long, or the contents of various recordings are too substantial, it may be difficult to find out what was said within which time period, and the recording is left unreviewed. In addition, important information that has been recorded is often easily overlooked. This contrasts with written documents, which can be browsed to find out what has been written where-even when there is a large amount of text.
In order to overcome this inconvenience--however long the recording tape may be--RWCP has developed a way of finding out within what time periods which major topics have been mentioned, and how many minor topics are contained under each major topic. This method enables users to reconstruct summaries of major and minor topics by patching together the relevant portions of the recording into a shorter discourse. It can be seen as a way of "browsing" recorded speech. The method eliminates difficulties in post-processing long recordings.
In addition, listening to summarized speeches also allows easy retrieval of images from videotapes when the topic sounds important.
The method is not restricted to Japanese, but can be applied to any language--including English, Chinese, or Arabic. Even mixed language sets (e.g., English and Chinese), often encountered when interpreters are present, can be dealt with.
The technique is currently being used in real time to allow users to listen instantaneously to a summary of what has been said during or after meetings or discussions.
New Scheme Used in This Research
TRC has recently developed a scheme, termed "Incremental Reference Interval-Free Continuous Dynamic Programming (IRIFCDP), capable of processing recorded conversations or speeches during meetings.
Assuming that the same thing was said twice within one minute during the meeting, for example, the method recognizes that the same words have been spoken twice, connects them with a line, and places this line on a time axis (Fig. 1-1).
When this process is repeated, line segments are stacked on the time axis (Fig. 1-2).
When looking at the way in which the line segments are stacked, there are peaks and dips, depending on the numbers of line segments that are stacked. High peaks correspond to major topics, and low peaks to minor topics. Dips indicate where one topic breaks off (Fig. 1-3).
The fact that each peak corresponds to each topic is based on experiences and experimental observations; it is very likely that the same words or phrases will appear twice within one minute when one or more persons are talking with a single topic in mind. IRIFCDP is designed to be capable of automatically detecting the same words, phrases, sentences, or any speech phenomenon, that appears twice within one minute.
The method also locates the boundaries between topics by automatically detecting peaks and dips. Fig. 1-4 shows a sample output from the computer. As shown in the figure, four topics were mentioned during a period of 15 minutes, separated by major dips. Also shown are minor dips corresponding to subtopics under major topics.
Topics can be summarized by connecting the sections of speeches that form peaks (Fig. 1-5).
Listening to just this portion may suggest what is all about. A written summary can also be produced by passing this portion to a speech recognition process. The written summary can then be used as a search key to retrieve important information from a large database, even during a meeting so that it can be offered to participants.
The technique can be used flexibly to process speeches in everyday conversation, meetings, or discourse, in order to provide users with information in any form that is requested. It is expected that the technique will become essential in a multimedia society, allowing significant speech data to be used effectively even when handling it may be difficult.
Prospects
The research uses a method of extracting linguistic information contained in recorded speech without direct speech recognition. This avoids various difficulties that are often encountered in speech recognition, and it therefore acquires a very advanced type of functions of speech recognition.
The idea is one implementation of our goal of "flexible information processing." The important aspect of this is that the approach is effective in using pattern information to generate symbolic information. This is an idea that should be further generalized in the future.
Real-Time Integration System for Speech and Gestures of Simultaneous Speakers
Technical Capability
It is a well-known fact that human beings communicate their intentions to others using speech and gestures. We need to take in other people's ideas or information, and either insist on or adapt our ideas in order to reach a certain level of mutual agreement. Can we use computers in such situations to help us reach consensus of opinions smoothly? Can computers help people to cooperate in creative intellectual activities?
Needless to say, computers are no substitutes for people--and especially not for people involved in a debate. Computers might, however, be useful during discussions to carry out tasks that people are not good at, but computers are. Specifically, computers are good at retrieving any information that is required and displaying the results, particularly displaying earlier proceedings or agreements made during discussions, using color graphics.
However, it is quite difficult for current technology to link such computer operations to human discussions that use speech and gestures.
TRC has developed a system which can solve these difficulties. The system (Fig. 2-1) is described below.
In Fig. 2-1, two users are discussing how a house should be designed, using conversational speech and gestures to express their intentions. The computer recognizes the speech and gestures of both parties, and presents in color graphic form what they have agreed about the design of the house at various points in the discussion. Both parties can then examine the stored images in order to confirm the current agreement status and proceed to arrive at a final design of the house.
Fig. 2-2 shows a sample computer output. Four pictures in this figure indicate how the design has changed, starting from (a) upper-right, through (b) lower-right, (c) lower-left, to (d) upper-left, with:
(a) One-story Japanese-style house with a pond, a rock, a carport, cedar
trees, a flower bed in the garden, a western-style gate, and a concrete
wall.
(b) The house has changed to a two-story type.
(c) The wall has
changed to brick.
(d) The cedar trees have been removed.
This is how the conversation proceeds:
A: What type of house would you like?
B: Well, I'd like a one-story
Japanese-style house.
A: How about a wall?
B: A concrete wall might be
OK.
A: I'd like to have a pond, and I'll need a parking space for the car,
preferably a carport type.
B: O.K. I think a big rock should be brought in
near the pond.
A: And a flowerbed, too.
<< (a) displays up to this
point. >>
B: The house should be a two-story type to give us more
space.
<< changed to (b). >>
A: O.K. The wall ought to be
brick rather than concrete, which doesn't look so good.
<< changed to
(c). >>
B: It looks a bit cramped around the parking space, so the
trees --- [should be removed--expressed by the gesture].
<< changed to
(d). >>
And so on.
In this way, the screen changes as the conversation proceeds. An agent (face) on the screen indicates the screen changes to the users via synthesized speech, giving the two users the impression that the conversation is being carried out among three people.
New Schemes Used in This Research
Four new schemes have been used to implement this system:
(1) Real-time spotting recognition of arbitrary speech.
(2) Real-time spotting recognition of intentions expressed by gesture, observed as animated images.
(3) Representation of a knowledge database as a network, and representation of the progress of the conversation in the form of images, derived from the recognition of words in the input speech and gesture in the animated images.
(4) Real-time processing in which the output images can change in each frame by combining the functions (1) through (3) above at frame intervals of 8 msec or 30 msec.
The recognition schemes described in (1) and (2) above allow speakers to use conversation and gestures with complete freedom. This is because only those words and gestures that are necessary to the progress of the topic represented on the network are recognized and extracted from the speech and gestures involved in the conversation.
In this system, the computer graphic images are updated every 8 or 30 msec, keeping pace with the speakers' thinking speed. This allows the system to have a real-time influence on the speakers' thinking as the images change. The images are driven by speech and gestures, as well as by the knowledge represented in the underlying database. A fundamental characteristic of the system is its real-time use of the knowledge in the database.
Prospects
There are currently limitations to the vocabulary range and the various types of gesture that can be processed in real time. Further research and development are needed to identify better features from animated images of gestures, and to improve the automatic configuration of the network to be used in the formal experiment based on a large-scale knowledge database.