|
Hideki AsohInformation Science DivisionElectrotechnical Laboratory |
|
|
Hideki AsohInformation Science DivisionElectrotechnical Laboratory |
As human beings, we process and integrate various modalities of information from raw, perceived data to natural languages in our daily life. Reflecting this, the abilities of computers have also been gradually extended from handling just numerical data to texts for document processing and logical expressions and mathematical equations for knowledge processing. The recent multi-media boom is caused mostly by the fact that personal computers become to able to handle (animated) images, sounds, and speeches.
Although the range of modalities of information which computers can handle is approaching that of humans, computers, however, still perform only elemental processing mainly, and "Information Integration" technology like humans appropriately process various type of information remains unestablished. This technology is a key for building "Real World Intelligence Systems" which can collaborate with humans in the real world, and is one of the major goals of the Real World Computing Programs.
Information integration technologies are often divided into three categories: horizontal integration, vertical integration, and system-environment integration (Fig. 1). Horizontal integration refers to integrating information from different sources such as visual, auditory, and tactile senses. Vertical integration stands for integration of different description levels such as pattern information on the sensor level and symbolic information on the language level. System-environment integration means the integration of information within a system and information in the external environment. Although this classification is not so clear-cut, I will use it tentatively in the following for the sake of convenience.
This area is closely related to sensor fusion technology, distributed sensing technology, and multi-media technology for the integration of (animated) images, sounds, and speeches. In computer vision research, reconstructing inner representation of the external world, which cannot be directly observed, by integrating information on parallax, shading, textures, etc. has been investigated. For virtual reality technology, providing users with integrated visual, sound, and tactile information is important. These are closely related to horizontal integration technology. In recent years, the multimodal interface technology has been recognized as a near-future interface technology. The goal is to enlarge the band width of the information channel between the systems and users, from the conventional keyboard-based narrow channel to one which includes animated images and speeches in order to realize a more comfortable and more natural interface that imposes less load on both systems and users.
What are the main difficulties in integrating information with different modalities such as visual and auditory signals? The difference between their natures may not be a significant problem for integration. For instance, the Hidden Markov Model, a standard method of dealing with time-series, and the Reference Interval-Free Continuous Dynamic Programming (RIFCDP) matching technology developed by RWC, are proving to be useful both in processing speech and animated images. From my short experience in working on multimodal interfaces, I think that the biggest difficulty may lie in finding out the combinations of information which can produce useful information.
In integrating a variety of information, only a part of the information is used at every moment or in a specific situation. It is impossible and impractical to taking all combinations of information into account always in real time. Thus, it is important to focus on the useful combination of information required at the specific moment or situation. In the conventional systems, processing has been organized based on a priori modalities of information, but once the walls which separate different modalities of information is broken and interaction is allowed, it is unclear what is a good combination of information under each situation. This may lead to the conclusion that "A core of the integration is essentially a dynamic subdivision of multi-modal information." Another related issue is how we determine that the information coming from different sensors is on the same subject. A typical example is the correspondence problem in stereo vision. As the number of sensors increases, this problem becomes more difficult.
These are information segmentation problems involving extracting meaningful chunks from huge volumes of information according to situations, and at the same time the problem of "attention", focusing on meaningful information. Several R&D studies have been conducted on maintaining attention on one object, such as tracking, but the elemental units or triggers for attention has not been clarified. It is necessary to find and implement useful combinations of information by referring to studies in cognitive science on the human behaviors in doing those tasks which require processing of many types of information such as driving a car and playing real-time game, and by applying the findings from research on the brain. It is also important to conduct R&D on technologies which enable to systems to automatically self-organize the multi-sensory space in a task- and situation-dependent manner.
Another difficulty lies in the lack of data. The collection of standard data such as speeches and hand-written characters and the construction of databases have greatly contributed to the progress of pattern recognition research. However, only a few databases exist on situations involving multiple modalities. It is important to build up multi-modal databases. At the same time, technique for a system to actively gather and learn the data by itself may be necessary.
Those involved in producing real-time shooting games, as well as movies, CG, and animations are probably the most aware of these issues. Studies on engineering know-how in such areas may also be interesting.
A red apple can be expressed in many ways such as by a color photograph, a monochrome photograph, a hyper-real drawing, a drawing in a picture book, a cartoon, and the words "red apple". While horizontal integration deals with integrating various types such as the color and the taste of a red apple, vertical integration deals with integrating expressions with different "precision" (this may not be a proper term) as in the previous sentence.
Because the world is very complicated, it is not practical to describe and process everything on one level of description. That is, a process of abstraction is necessary, and many levels of description such as photographs, cartoons, and sentences are used daily. This leads to many research issues; how to construct various types of abstract expressions in order to handle such variety of abstraction levels, and how to control the process or the flow of information, given information on many levels.
An example of the latter, that is control of process utilizing multiple expressions on different description levels, is to be found in the real-time continuous speech recognition system, where spectrum information is extracted from raw speech data, which in turn is converted into vector quantized code on the phoneme level. Then it is analyzed with a word lexicon which describes the relation between phonemes and a grammar which describes the relation between words, to eventually output one sentence as the result of recognition. Also, in research on image understanding, a process has started to be used where the meaningful regions are extracted from the brightness on the pixel level, which are then analyzed using an image lexicon, which relates those regions, and image grammar, which relates images. Although, in many of these systems, the process goes from bottom to top hierarchically, the integration of top-down predictions and the bottom-up processing has always been an important issue.
A more detailed issue relating to the vertical integration is to clarify or construct a representation system with a structure where multiple representations are correlated with each other in a complex way. One study in this direction is "symbol grounding problem" in artificial intelligence. This originated mainly in studies on natural language understanding and, according to Steven Harnad who invented the term, it deals with how to ground a formal symbolic description to a semantic interpretation that is intrinsic to the system and the environment. This problem is directly related to the philosophical problem of "the meaning of symbols." On the contrary, this problem can also be viewed from sensor signals (patterns) side as "how to segment and associate sensor signal space with symbolic representations, or how to derive symbolic representations," and this is directly related to conventional "pattern recognition" and "pattern understanding" problems.
This is the most important technology when systems actually deal with the "meaning of words." This is also required, to some extent, for the systems which processes discrete internal representations and combines elemental representation. A typical example is action planning for intelligent robots. The simplest way to have a robot behave as wished is to specify the position and joint angles for every very small interval of time as found in the current playback-type industrial robots. On the contrary, to instruct it on a semi-natural language such as "take a cup", the symbol of a "cup" must be associated with information acquired from the robot's vision system, and a symbol of "take" must be associated with a sequence of control signals in its motion control system. Moreover, "take" should be interpreted into different motion control signals for a cup and a spoon, and "to take a cup" should be interpreted into different motion control signals for the different shape or orientation of a cup. Thus, in information integration, the generation, control and maintenance of complicated highly context-dependent relation-ships among complicated representations is a very difficult problem.
This is also closely related with studies on intelligent robots. It is natural to think of having a system store a model of the external world inside and to select an appropriate action using prediction or reasoning with this model in order to make this system behave intelligently. However, this leads to the problem of "what internal model should be stored." Giving too large or too precise internal model incurs computational cost for maintaining consistency with the actual external environment as well as prediction and reasoning. Giving too poor internal model may limit its intelligence by restricting the resulting actions to very reactive ones.
Rodney Brooks et al. in the MIT demonstrated the feasibility of implementing actions that are seemingly complicated and intelligent by integrating even a poor internal model with information acquired from the external environment in real time, based on the observation of intelligence in insects. An architecture proposed for implementing this system-environment integration to make systems reactive is called "subsumption architecture."
The origin of discussions on such internal models may be found as far back as "cybernetics" in the `60s. The proposed idea of "feedback control" is a control with no internal model, where the result of the controlled behavior is not predicted with an internal model, but is computed by the controlled object itself as the result of actions actually taken. A reactive system may be considered as an extension of this idea, assuming faster and denser flow of information between system and environment.
A problem here is to generate stable patterns by making information >from the internal model and the information that is supplied each time >from outside react appropriately with each other, that is, to design an appropriate information loop to integrate the system and environment.
System-environment integration is an ultimate goal in the entire information integration. That is, horizontal integration and vertical integration are used to implement this. In fact, the subsumption architecture involves considerations of horizontal integration and vertical integration. The importance of the system-environment integration in an intelligent system is also strongly expressed in the system theory known as "Autopoiesis" advocated by Maturana and Varela in more radical form. They denied the distinction between the system's inside and the outside. This issue is also related to wide variety of problems such as the cooperative formation of multi-agents in artificial intelligence, Gibson's "affordance", and Gadamer's "hermeniutics."
So far, we have reviewed the flows of research in respective categories of information integration. More general researches on information processing architecture, programming languages, and models are also carried out for information integration.
For instance, Koiti Hasida of the Electrotechnical Laboratory (ETL) considers the design of intelligent systems as the implementation of various complicated and situation-dependent information flows, and proposes a constraint-oriented design as the general design principle. He has implemented an integrated speech dialog system based on this principle. This "predescribe only the constraints among information and determines specific processing procedures on the fly while running the system," is closely related to "declarative programming." Kiyoshi Akama of the Hokkaido University has proposed a declarative programming language to demonstrate the emergence of integration of knowledge on several levels in some examples in natural language understanding. "Integrated processing with fields" proposed by Ryuichi Oka of RWCP is also constraint-oriented with different granularity of information representations.
Connectionism, the child of studies of the brain science and the cognitive science, is also related to the search for an architecture for "information integration." For instance, there are many proposals for symbol-pattern integration include Paul Smolensky's tensor product network and Takshi Omori et al.'s "PATON." In the field of artificial intelligence, as inspired by "Society of Minds" written by Marvin Minsky and the progress in distributed artificial intelligence technology based on the development of networks, a concept of "(multi) agent-based" approach has been proposed as a method of implementing complex systems. Matsuyama et al. of Kyoto University considers information integration for computer vision from this viewpoint. The "organic programming" proposed by Hideyuki Nakashima of ETL was motivated by situation theory to implement situation-dependent actions by combining situation recognition and situation-dependent processing. This reminds us of the combination of the relatively fast perception-motion determining system (cerebral and limbic system) and the slow recognition system (neocortex) in the brain. Kunihiko Kaneko of the University of Tokyo, Ichiro Tsuda of Hokkaido University, and Jun Tani of Sony CSL have been making research based on the non-linear complex dynamical system.
On the other hand, there are also old statistical techniques dealing with huge amounts of various types of incomplete information as well as "interpreting data with (internal) model". In particular, as computational-intensive probabilistic methods have been developed due to recent advances in computers, statistical and probabilistic theories and methods are likely to be exploited in information integration technology.
We have reviewed the trends in information integration-related research fields. The RWC Program has conducted R&D based on the concept of "information integration" since its inception. In the latter half of the program, R&D on multimodal dialog systems, self-organized information bases, and autonomous learning systems will be pursued as real-world intelligence systems that require sophisticated information integration. Each of these will be introduced in other occasions.
Information integration technology will explore patterns that combine a great amount of hetero-geneous representations, and focus on important and stable patterns. This technology is essential to make the interaction between information processing systems and the environment including humans more natural and to implement information processing technology that is more like humans. Extensive areas of science are related to the technology. I look forward to further progress in this research area.
Acknowledgments: The discussions in "Information Integration Workshop" and "Symbol-Pattern Integration Workshop" by the RWCP were very helpful for writing this paper. I thank the participants of these workshops, especially Yukito Iba (Institute of Statistical Mathematics), Hideyuki Nakashima (ETL), Ryuichi Oka (RWC), and Takashi Omori (Tokyo University of Agriculture and Technology).