Through the following conversation with "Ms.
N," a visitor to RWCP facilities, and myself (O), latest RWCP research
activities shall be introduced.
Through the following conversation with "Ms.
N," a visitor to RWCP facilities, and myself (O), latest RWCP research
activities shall be introduced.
N: Excuse me, but could you tell me about your latest research activities?
O: First of all, welcome to our facilities and thank you for coming. On that regard, there are a lot of issues to talk about.
N: Oh, really? Any one particular issue, then.
O: Well then, let's talk about the development of a method called IPM.
N: What is IPM?
O: It is the abbreviation for Incremental Path Method that we have derived. It is a way by which computers accumulate and compile data continuously flowing in over time, such as speech and motion images, and store data in a sort of network of its own.
N: What do you mean by terms such as "method" and "network"? What is ultimately the final goal? I would like to know that first.
O: O.K. I should have said that first.:-)
Specifying the
Goal
O: The system we are trying to build is a "multimodal retrieval/summarization system" which cabn perform inter-conversion and search between any pairs among speech, motion image, and text data." Do you understand?:-)
N: Not exactly.:-)
O: The speech data include speech signals, such as voices on TV and radio programs, and speeches made at lectures and discussions. The motion image data include data on TV programs and captured on video. The text data include newspapers, books, articles on the Internet, and office-created documents.
N: Up to this point I can understand.
O: Our system can handle these pieces of data. As far as how these data are handled, suppose we are watching TV. Suddenly, we think we saw something on screen similar to this before. Something important was said, and we want to know what exactly it was. This feeling of deja-vu does not happen all the time.:-)
At the same time we start wondering what newspaper articles were written about this feeling. We, humans, tend to "think" about too many things.:-)
N: Exactly.
O: In such cases, just by mentioning a need or expressing it with a gesture, the desired images, speech, or newspaper articles would be generated on screen or over a PC's speaker.
Just thinking is not enough, we have to express our needs by voice or gesture.:-)
N: You mean, thinking should be explicitly expressed. Naturally!
O: That's the system we are trying to create, but it is quite difficult.
N: Seems like it could be easily done.:-)
O: No, it's pretty tough.
The Types of Technology
Needed
O: That is, to actually establish such a system, the sorts of technology necessary are:
(1) Technology to search the same type of data, i.e., "speech <-> speech," "motion image <-> motion image" and "text <-> text."
(2) Technology to search and convert between any pairs of three types of data, i.e., "speech <-> motion image," "speech <-> text" and "motion image <-> text."
(3) Technology to automatically accumulate and compile a vast amount of raw speech, motion image and text data created daily and sent over the Internet.
(4) Technology to summarize data (see Fig. 1).
It is not a simple task in achieving any of this.
N: Well, I can't say I understand fully, but I guess there are many things to it.
O: Well, let me explain to you a little more so as to clarify myself.
In addition to the preceding, two of those kinds of technology [viz., (1) and (2) above] may have something in common. For instance, since speeches and motion images on TV, such as those in news programs, share the same time period, this time feature can be used for searching and converting "speech -> motion image" if "speech <-> speech" search is possible. In the same way, it is possible to search and convert "motion image -> speech" if "motion image <-> motion image" is possible.
In general, it is considered very difficult to search and convert "motion image -> text." Even the same person may express the same motion image differently in text from day to day. Therefore, by using speech recognition that may be considered as "speech -> text" conversion, the path of "motion image data -> speech" can be used to go from motion image to text, and "motion image -> text" may be achieved through by "motion image -> (speech -> ) text" with some restrictions. At this point, speech is considered an expression of a motion image interpretation in a spoken form.
Also, "text -> speech" can be handled as a "speech -> speech" problem by expressing text as a voice wave with speech synthesis. In this case, "text -> (speech ->) motion image" may also be possible by using the path of "speech -> motion image" mentioned before.
N: Are you saying there are many ways to correlate speeches, images, and texts with one another?
O: Absolutely. I ought to thank you for catching on so quickly. But it's not the end of the story.
N: What else is there?
Necessary 'Smart' Search and
'Smart' Summary
O: "Text <-> text" is considered the easiest. Search programs like Yahoo are examples of this.
N: Finally, a word I know!
O: In such programs, keywords are linked to the home page. So, if all of speech and motion image data can be labeled with keywords, motion images and speeches may be searched from texts. But, it is still questionable if a vast amount of data daily accumulated could be handled with this method, due to problems and consistency through extraction and labeling of speech and motion image data.
N: I see. That's way in Yahoo the home page can not be searched by showing images or talking to it.
O: On the other hand, even a search with "text <-> text" itself involves many problems that must be solved if a sophisticated relationship is necessary between a query (search request) and a certain target. Demand to "smart" search, which handles associated and conceptually-close words in a query similar to that of human beings, will probably strengthen in the future. We are developing a new technique called "Galaxy Net Method" to meet such a demand - I'll discuss that another time.
N: Text searching by text is still a long way off, I would assume. Right?
O: Oh, yes. As for "speech -> text" conversion, we are developing a method called IRIFC DP, which is an extended Continuous DP, to divide topics in sections, as well as technology to generate text-form speech summary using speech recognition and thesaurus dictionary with a large amount of vocabulary capacity (with more than 200K of words). It's hard to automatically distinguish between the start of one topic in raw conversation and lecture speech data with the end of another.
This is a highly sophisticated job, even for humans. It would also be a problem as difficult as or even more difficult than this to summarize and document this, even if one topic section could be cited.
The technology to summarize topics is that described before (4). I have already explained how this research was proceeding, in the past RWC NEWS (3rd issue) to some degree, and the latest status will be presented on another occasion.
N: Although speech recognition has a long history, it also seems like a rigorous field. I thought speech PCs recently available from IBM were sufficient.
O: Many people are surprised to learn that it is more difficult to make computers understand usual conversational speech than narration or lector speech.
N: I also figured it was easy to make computers understand that sort of speech, as I myself talk without thinking about it.:-)
O: That means your brain is hardly at the same level as that of a computer.:-)
N: Yeah, right!:-)
O: At any rate, let's talk about something else rather important.
N: What? We are not already talking about it? Sounds to me it's ended.
O: There's more.
Compiling a Vast Amount of
Speech and Motion Image Data
O: The technology mentioned before, (3), "Inter-conversion and search between speech and motion image data,"is needed for processing and accumulating a vast amount of raw speech, motion image and text data for later use. Without it, ever-increasing data could not be handled and there would be no clue for developing methods to accomplish the other three types of technology [(1), (2), and (4)].
So, I ought to explain about the "Incremental Path Methof" (or IPM) which solves this problem. The IPM is used to automatically create a database directly from raw speech and motion image data.
N: Now I see! You mentioned "IPM" earlier.
O: Yes, I did. IPM is one of a few self-organizing methods developed by our laboratory. Self-organization is something by which pieces of data are automatically compiled and a way upon which the data compiled are automatically improved.
N: I doubt that such things can be attained. You must be kidding.
O: No, really. But it may sound exaggerated.:-)
Anyway, IPM is for realizing technology (3). "Speech <-> speech" and "motion image <-> motion image" can be attained by "Continuous Automation," an extension to a spotting-based retrieval method called "Continuous DP," to a database created with IPM. This can be considered a partial realization of technologies (1) and (2).
As for IPM, it handles time sequence symbol data as if an input. Therefore, speech and motion image data are first divided into data sections over a certain period of time. For example, speech is divided into spectrum vectors of sound waves for 8 or 16 msec, to form a sequence of vectors expressing sounds at different tones. Motion images are divided into frame images for 30 msec, equivalent to one screen of TV, to form a frame image sequence. These are then, further processed so that spectrum vectors or frame images are quantitized, or divided into cubes which are in turn numbered.
IPM also represents the entire input for such data, as a network consisting of nodes and arks which link those nodes. Here, this network can be created automatically by IPM. Each ark is numbered with quantitized input data at a given time. If the current input with quantitization number can not be included (or can not be positioned) in the network by reusing existing nodes and arks, a new node or ark is created and attached to it. This job is designed to follow five natural predetermined rules. In automatic creation of networks, speech and motion image data have separate networks. For motion images and speech signals in a TV program sharing time, a datum quantitization number assigned to an ark will also store a parameter to indicate when the data are entered.
Depending on the IPM, the number of nodes and that of arks may be in the order of several hundred thousands. A vast amount of speech and motion image data are included into respective networks to constitute a database in a compressed form. And data entry can be continued by disallowing any increase or only allowing of limited increase in the number of nodes and arks. With this, a database is automatically created, improving as more pieces of data are entered. This is a database as well as a "model" formed by entered speech and motion image data. Therefore, IPM is also considered to automatically build a model of time sequence data network.
N: It's all Greek to me! Way too technical for me...
O: Hmm, let's take a look at the results. Fig. 2 shows what such network data look like. This was created from motion image data entry, represented in a 3-dimensional space. Frame images are attached to nodes so as to be easy to understand (frame images are actually attached to arks).
N: Right. It is called a network because images are linked by lines, isn't it?
Are things on TV I saw also stored in my brain in this manner?
O: Well, it is true that humans store images in memory and they are linked with one another, but I don't think what exactly they are has been fully grasped, in a physiological sense. Since it is also true that motion images are not memorized as they are and neural networks are indeed networks, one might consider these images similar to the network in this figure.
Touching briefly upon techniques to search this network database...
N: You still have more to say?
O: Exactly... a little more.:-)
Search Possible Even in
Ambiguous Query
O: When we want to search something, we often input to the effect of "something related to such and such." Here, "such and such" is the "query" and "something related" is the "search result." As explained thus far, in a multimodal inter-retrieval system, a "query" is a motion image and a "search result" is a speech at certain times, or "query" is a motion image and a "search result" is a text at other times. If a "query" is created while a "TV program is on the air," it may be as ambiguous as "something like speech or image which appeared within the last 5 minutes." This is when users canšt identify a "query" as a definite keyword. Therefore, we try to make it possible to search with a "free-wheel query" with no start or end time.
N: It would be good for users if they could make a query in a free-wheel manner.
Users simply could then speak continuously as they think.
O: That's right. The point is to reduce user-side load. Ideally, we want to make the user forget the actual computer is a computer.
A query entered in such a "free-wheel" manner could be searched with a method called "Continuous DP," which is developed by our laboratory.
With "Continuous DP," activities of all nodes of an IPM network are computed and associated. Each activity indicates the presence of similar paths, when a database on a network (IPM) is traveled in the previous time sequence input, at a given time in a continuous time sequence query (speech or motion image). In other words, that is, when an optimal path is searched. Higher activity means that there is information more closely related to that path in the network. Suppose nodes with activity higher than the threshold are lit in the network at this point. If they are continuously lit, then there are section data (which we are looking for) in the database - similar to the section data (which is automatically determined among the sorts of input) ending at present and up to a certain time in the past.
When the data was entered into the database is known through the time stamp at that lit position. The section indicated by continuously lit nodes is a search result (that is, speech section or motion image section). If there is no (section) data in the network database around the current time, there should be no lit node.
This indicates that we don't need to specify in advance the range of a "query" in ever-changing input time sequences. This type of search is called "spotting-based retrieval." This is what is realized in "Continuous DP" as "a standard time sequence pattern with section" (which corresponds to a database), turned into an IPM network or database on "an interlinked network" (which corresponds to a kind of standard pattern). This method is called "Continuous DP." Again, the principle of optimum in "DP" is relevant here.
N: Which means that computers should be designed to handle user queries in a "free-wheel" manner?
O: Exactly. That's one of the most profound comments you have made today. Fig. 3 shows the result of this case.
N: ...This figure obviously indicates that. But that's enough for today.(^^) Thank you for your time and elaborate explanation of the processes involved. I'm sure it was as hard to explain as it was to follow. At any rate, thank you very much.:-)
O: I wish a demonstration could convey what I have said. Unfortunately, it would simply stress what could be seen outside, thereby not disclosing any background of our research.