Prototype of automatic translation system of sign language
During the last fiscal year we constructed an automatic translation system for sign language.
This system translates spoken Japanese into Japanese sign language. The translated sign language is presented as a 3D graphics animation to a hearing-impaired person. Conversely, the system translates sign language gestures input from a sensor glove worn by a hearing-impaired person into Japanese and the result is output as synthesized voice and text.
Words in sign language are recognized by detecting the gesture components such as the shape, direction, and position of a hand during manual motion separately and integrating the results. Stochastic knowledge is included in the template of gestures which are referenced upon recognition, thus enabling flexible recognition.
Our system can recognize simultaneous communication (Signed Japanese) and Japanese sign language.
Sign language words are represented according to Japanese syntax in simultaneous communication. Particles are usually omitted in communication using simultaneous communication, which may result in missing grammatical information. Signed Japanese is mainly used for communicating between hearing-impaired and hearing persons; this sign language is taught by sign language groups.
Japanese sign language is a native language of the congenitally hearing-impaired. Japanese sign language involves the whole upper body as well as manual motion. Japanese sign language can also convey grammatical information accurately to the other party. Regarding manual motion, the shape, direction, and position of the hand, as well as the direction, position, speed, and size of the motion are used to convey grammatical information. Regarding non-manual motion, the shape of the mouth, facial expression, eye-brow movement, jaw movement, gazing direction, nodding and shaking of the head, and body direction are used to convey grammatical information. For instance, sentences start with a raising of the eye-brows and end with a nodding. The past tense is indicated by shaping the mouth to form the syllable "Pa" at the end of a sentence, and a negative tense is indicated by shaping the mouth to form the syllable "Pi". Thus, Japanese sign language conveys intentions by using multimodal information.
The congenitally hearing-impaired think in Japanese sign language, but few sign language interpreters can understand Japanese sign language. Because of this, the congenitally hearing-impaired are forced to switch from Japanese sign language to signed Japanese when conversing with other people, including sign language interpreters.
Given the above description and need for a welfare system, Japanese sign language recognition technologies, automatic interpreting systems, and education systems that use these technologies need to be developed. In addition to simply reacting to these needs, to encourage technological development, Japanese sign language recognition is a suitable real world subject for multimodal information integration and recognition technologies. From either perspective, the development of multimodal information integration technologies is the key challenge. Currently, we are conducting a basic study on a multimodal information integration method focused on the manual motion and non-manual motion of Japanese sign language, especially the processing of grammatical information represented in manual motions.
Our research focuses on the development of multimodal information integration technologies, and we shall construct an automatic translation system for sign language dealing with Japanese sign language. The construction of such a system involves the downsizing and the input and recognition of images, as well as the implementation of developed technologies. Trial operation of the system in the real world is planned in FY1999.
The Multimodal Function region has been organized to focus on vision. We shall take full advantage of this organization and study image input and recognition in collaboration with other research entities. We have already started to supply relevant data to other research entities.
We have created a multimodal sign language database as an intellectual resource. This database will contain data related to words, sentences, and spoken conversation, including:
(1) Filmed image data of faces and upper bodies taken from the front and the side with and without wearing a sensor glove
(2) Waveform data from when a sensor glove is worn
(3) Labeling data to show at what time what types of manual or non-manual motions are performed
This database is characterized by the correlation of image data, waveform data from a sensor glove, and labeling data.
We will use this database mainly for acquiring knowledge about Japanese sign language. The image data may not need to be high quality for such usage. However, we will gather image data appropriate for research on image processing so that such data can be effectively used as an intellectual resource. Currently, we are examining data management methods and methods for enabling a model (hearing-impaired persons) to speak sign language naturally. A sample with this database will be made public in FY1998.
The market for welfare systems is small, but the market for the technologies that are used in them may not be small. Products with new concepts will be developed by abstracting the multimodal information integration method for Japanese sign language that is developed in this project and extending it to a more general multimodal information integration methodology. We aim to put the system into practical use within five years.
Intellectual resoures