Prototyping "Gesture-Speech Interface PC"
-Interaction via Interractive Multimedia-

Ryuichi Oka

Theory and Novel Functions Laboratory
TRC

In early March, 1996, the Information Integration Laboratory, TRC, RWCP, published one of its recent research results in the newspapers. This report was written based on the news releases.

What was Prototyped?

Gesture-Speech Interface PC (GSI-PC) (see Fig.1)
This software can also run under Windows of Microsoft Inc.

Fig,1 Prototype Gesture-Speech Interface PC(GSI-PC)

What Was Made Possible?

Natural dialog with PC (see Figs. 2 and 3)

Fig.2 Screen Presented by PC Software
An agent(face shown in upper right corner) to which the user talk and a CG presentation of what the agent understands of agrees between the user and agent(images are updated counterclockwise at a certain time interval as shown on the left)

Ordinary PCs with application software developed in our laboratory can be operated via human gestures and speech, without large-scale computers which have massive processing capacity or high-performance workstations.

The PC has an agent, a virtual human with a face, which answers by voice and presents graphically what it understands on the screen to serve as a mediator between the PC and human user to make it more friendly. It can deal with various tasks simply by changing the application software. Figs. 2 and 3 show an example of using the application software for designing a house.

Using such application software enables the user to communicate his or her intentions to the PC via redundant conversations and/or gestures. Real-world information is redundant and incomplete, like the information supplied by humans. If a PC can handle such information, it should be more familiar to users.

A user can easily register words which the PC can understand as speeches and gestures. Although there are typically few gestures involved, the words used in speech may vary widely. The number of words handled by one task is normally limited to 100.

The PC recognizes in real time only those words which have previously been registered from the flow of redundant speeches and gestures supplied by the user. Of course speeches and gestures can be used simultaneously.

The PC then integrates the recognition result of speeches or gestures with the database and provides the result as a CG or answers by speech to enable the user to easily understand. The CG can be updated every 16 msec. Speech is recognized every 16 msec, and gestures are recognized every 60 msec. They are processed on a first-recognized first-served basis.

What's New?

Unlike conventional human interfaces on PCs, the Gesture-speech Interface PC has the following features:

How Was It Built?

Methods of implementing the following were developed:

These technologies were developed as an achievement of RWC research. Previously, each of these technologies required a separate high-performance workstation, requiring at least 3 such workstations for the entire process. Our prototype demonstrated that they can be implemented on a single PC and run in real time.

Fig.3 Sample Dialog(application software for designing a house). A and B indicate human speeches and getures.<....> denotes a PC screen, underlined texts are spottingly recognized by the PC, and double-underlined texts denote getures.

(NOTE) The term "spotting" means the automatic recognition of previously registered words from a continuous flow of speeches or gestures without extracting them, where "recognition without extraction" means determination of whether any registered word appears or not at any time. This is close to humans' intuitive understanding of the whole from spottingly understood words in a flow of redundant speeches or gestures.

Challenges and Future Direction

This PC uses two DSP boards which support CPU processing on the motherboard of the PC. The DSP boards in this PC will be eliminated sooner or later because the performance of the CPU on the motherboard will soon be greatly increased, at which time this PC is expected to penetrate all of society as a hardware platform.

As for the dialog function with gestures and speeches, the software used on this PC cannot replace word processors with a function of typing and document presentation. Therefore, the dialog function with gestures and speeches will only be achieved:

(1) when an application area has been set (i.e., words can be restricted and a virtual human serves as the interface with the user) and,
(2) by general PC operations (i.e., selection with a simple keyboard or mouse operation).

    The enhancement of (2) is essential for the dissemination of PC usage. Hence, operations with gestures and speeches will be incorporated into conventional PC's mainly at this stage in a natural way. Because a virtual human will also serve as the interface with the user at this point, (2) is considered to be an example of (1). It will not take several years to reach this stage.

    The prototyping of this PC was done with a view to future progress in hardware and required software.

    Configuration of the PC