|
Paul Mc Kevitt Professor of the Center for PersonKommunikation(CPK), Institute of Electronic Systems(IES) Aalborg University |
|
|
Paul Mc Kevitt Professor of the Center for PersonKommunikation(CPK), Institute of Electronic Systems(IES) Aalborg University |
I am very pleased to be here today and be able to speak to the Real World Computing program. I would like to thank Oka-san and the committee for inviting me for the opportunity to speak here today. A lot of what I will be talking about in more detail will be following on from the previous speaker. A lot of the ideas that he was referring to will also come out in my presentation...and my presentation agrees very much with what he was speaking about and predicting for the future.
One of the things that I think is driving us towards the whole idea of Real World Computing and integration of natural language and vision processing by machine...it is the Information Superhighway and here's a picture of the Information Superhighway (Fig. 1).
Fig. 1
Another thing about this picture of the Information Superhighway is that we can look at it in two ways at least. One way is to look at in terms of traditional MultiMedia. Traditional MultiMedia is the presentation of images, spoken language, music, and so on and the hearing of it and the presentation of text, that is what traditional MultiMedia is. We are seeing a new phrase now called Intelligent MultiMedia. Intelligent MultiMedia is where we're looking at this picture in terms of understanding images and visual environment, understanding speech in the context of other sounds and understanding of text as well and the processing of text...that is the difference between traditional MultiMedia and Intelligent MultiMedia...one more focusing on presentation the other more focusing on understanding. ...and of course the Information Superhighway is an area where we're going to see a lot of this work being used because the Information Superhighway is about putting video, text, and spoken language on databases in different locations and, of course, retrieving it, and one way of retrieving it by being able to speak to the machine and asking it questions about the information that you want to retrieve. So that's an area which is very much pushing us forward.
What does it involve? it specifically involves speech and language processing and vision processing using parallel processing machines to do that. A lot of the papers we can see from the proceedings of this conference and the Real World Computing initiative are doing those things and integrating them together -- that is the Real World Computing initiative. And, also, there are large companies, such as Microsoft Corporation, who have large research laboratories working on speech, language, and vision and linking them together and eventually putting them into their products in the future, as MultiMedia products. Of course the European Union is very much involved in language processing because of all the multiple languages in Europe and it is spending a lot of money on speech and language processing and also in vision processing -- not so much on the integration of them yet, but that might come a little bit later in the future.
Fig. 2a
Fig. 2b
One of the problems in traditional computing and Artificial Intelligence work is that the people have had to express their emotions in strange ways. For example, you might get a smile in your E-mail but the smile isn't the right way up, it is on its side like these here (Figs. 2-a and 2-b). That would mean turning your computer screen on its side and that would be difficult sometimes. So it seems like computer scientists and people working in traditional computing have had problems when they express their emotions on computers. There are hundreds of different ways of smiling on the machine. The problem with all of the smiles is that they are sideways -- they are not the correct way up. That is a problem if you can't move your computer screen.
Fig. 3
What we really should be doing in computer science looking toward the
real world is looking at pictures more like these (Fig. 3). We are able
to do smiles and pictures of smiles. For example, sending an E-mail message
without a text smile on its side but more with a picture like this would
be more useful for expressing your intentions on E-mail, particularly if
you are doing it across cultures where people don't understand conventions
in E-mail using colons and hyphens and so on.
Taking a look at those faces again (Fig. 3), they were shown to 42 subjects,
and subjects were asked what do you think the smile means or what type of
smile it is. What came out of that using pathfinder cluster analysis was
that of course "happiness" came out and "fear" and "sadness" and they were
grouped together (Fig. 4). Of course there are lots of different types of
"happiness" : "meditative", "demure", "thoughtful", "smug","sadness": "lonely",
"regretful", "miserable", "fear": "shocked", "surprised", "frightened",
"horrified", "cold", and so on. Even more interesting was that these pictures
were also shown to some other subjects -- this time some Chinese subjects
(Fig. 5) and the last set were British subjects (Fig. 4). They came out with
similar areas of "happines", "sadnes", "fear", and "disgust". What was interesting
was that in general the British subjects and the Chinese subjects thought
that the pictures more or less meant the same thing. They more or less thought
"happiness", "fear", and "sadness" for both groups of subjects. There were
some differences in the details in the interpretation of the emotions in
the pictures but in general they were interpreted more or less in the same
way. That is good news because if you are sending those faces without labels
on them in your E-mail message between Britain and China, you shouldn't
have much of a problem. Maybe not this year but next year, it will be easier.
But of course the question is, there might be something that the British
and the Chinese have in common, the cultures might be very similar in their
interpretations of faces, so we have to check other cultures as well to
make sure that it can be used for the whole world as well. But at least
for British and Chinese that is good news for E-mail messages with pictures
in them.
Fig.4
Fig. 5
Going back a little bit now, back to 1960, there was a computer research problem where Nils Nilsson and Bertram Raphael at Menlo Park were developing a robot called SHAKY with camera and language interface. The idea was that you could tell the robot things like "Push the box off the Platform" and the robot SHAKY would go ahead and carry out that operation -- a picture of the robot here (Fig. 6) -- there is the box on the platform and the robot with the camera on top. It would have to go and move that box off the platform. It didn't always work, there was a problem of course back in the 1960's, none of these problems were solved. Most of them worked only some of the time.
Fig. 6
Sikossy was looking at language through pictures and looking at the mapping of pictures into list structures or list representations and into sentences. This was way back in the 60's as well.
In 1967, Coles at the Carnegie Institute of Technology was looking at using pictures to resolve ambiguity and that is mapping pictures into lists again and into language structures.
A lot of people have been working on solving problems on natural language processing of ambiguity like the sentence "I saw the man on the hill with the telescope." There are at least two ways in which that sentence is ambiguous. One way is where "I" have the telescope and the other is where it is the man on the hill who has the telescope. It is not clear which one it is unless you know the context. But if you have a picture showing a man with a telescope (Fig. 7), then you know that person is "I" on the hill, you understand the situation. This is one way in which vision processing or image processing of a picture together with a text will help to solve the ambiguity of that sentence. That was the idea that Coles was working towards. You could map a representation of who was seeing with the telescope into a list representation if you had vision processing of the picture along with it.
Fig. 7
Waltz in 1975 developed the Waltz algorithm for understanding line drawings of scenes with shadows and he was looking at boundary lines and the labeling of the lines and the sides of the objects in the environment. This is a simple object like a cube and you recognize there was an "arrow" on the corner (Fig. 8), there an L shape, a Y shape, and a T shape and then integrating this local information together. So, you get a picture of what the actual object was overall from integrating the local information. That was in 1975.
Fig. 8
In 1971, Shepard and Metzler did experiments on the mental rotation of shapes (Fig. 9). They found something interesting -- as the angle of rotation of the shape was increased and it took people longer to work out whether the shape was the same as the original from which it had been rotated from. The shapes look like this (Fig. 10). In this case (top) the angle of rotation is smaller, this one (middle) is greater. The object in this case (bottom) is actually different to the former picture. It takes a lot longer for people to work out whether those two (botom) are the same or not. So that was indicating that people in their heads have some sort of image representation of this information. Of course the big question was "Do people have images in their heads?" or "Do they have propositional representations or language representations in their heads?" There has been a lot of debate about this in Philosophy, Cognitive Science, and Artificial Intelligence. It hasn't been solved yet, but of course it's got implications for machines which are processing language and vision as well, because we have to work out what representations to put into those machines.
Fig. 9
Fig. 10
People came up with, Kosslyn in 1975, 76, 78, again doing research in Cognitive Science and Psychology looking at different types of images and different instructions to the subjects in the experiments and came up with the idea that people had images and verbal representations or propositional representations in their heads...and they talked about the Dual-Code Hypothesis. That was 1970's.
Then what happened was, people in Artificial Intelligence, Cognitive Science, Speech Processing, and Vision Processing went off and worked separately on vision and speech and so on because they found it too difficult to do everything together. So for about 15 years or more, most people worked on speech, text, natural language and vision processing separately because it was much easier to do...and also because they got into different departments. People in speech processing tended to be in Engineering departments sometimes Psychology or Computer Science departments, Linguists being in Language and Linguistics departments...and all working what they're best at or found easy to work on.
In 1991, Daniel Dennett wrote a book called "Consciousness Explained" and this book does not explain consciousness (Fig. 11). It tries to explain consciousness but actually nobody knows what consciousness is. So it's an attempt at explaining what consciousness is. It's a very good book. There were some interesting things in that book. One thing Dennett was pointing out for relevance here, he says, "Surely a major source of the widespread skepticism about machine understanding of natural language is that such systems almost never avail themselves of anything like a visual workspace in which to parse or analyze the input." (Fig. 12). What he is saying in 1991, is that there are people working on language processing and speech processing not taking into account the visual side or the visual workspace and they need to do that. Of course the reason why was that, as I just said, they found it too difficult to do both together. So he was right in 1991.
Fig. 11
Fig. 12
What we are finding now is that people these days since 1993, 94 are doing a lot more work on integration on speech and vision processing.
This is a program called "IntelliMedia 2000+" at the University of Aalborg in Denmark. It is a research and teaching initiative in Intelligent MultiMedia or IntelliMedia. It is bringing together 4 departments who are working on vision processing and natural language and speech processing and medical informatics, expert systems and computer science.
The Computer Science Department (Fig. 14) is working on hypermedia infrastructures, hypermedia databases, tool integration and extension of different modules working on speech and NLP and vision (Fig. 15). The area of tool integration and extension is actually very important, because it turns out that for integration one of the most important elements of integration is the actual platform for doing integration.
We see large projects like the German VerbMobil project which does speech processing and machine translation. They found a problem on which they focused on was platforms for integration. Also I think the same is happening in the Real World Computing initiative -- is the platform for doing the integration.
About a year ago, I asked Okada-san, Professor of Kyusyu Institute of Technology, when he was giving a presentation in England what he thought was the major problem in this area of integration of language and vision, and he thought it was the technical platform for the integration. Okada-san was one of the first people to continue working on spoken language, natural language, picture, and vision processing. He published a paper in 1979 while other people had separated off to work individually. It is turning out that platforms are very important.
The Computer Science department of Aalborg University has also developed a tool called "Topsy" which is a tool for knowledge representation and looking at the co-occurences of inputs, like a sound or a spoken word or a picture and linking them together (Fig. 16). Learning the co-occurences and co-exclusions over time from the input data -- that is all going to be very important for language and vision integration.
The vision department, Lab. for Image Analysis, is looking at active computer vision (Fig. 17). The basic idea is looking at how stereo vision can be done where the parameters are dynamically changed on the algorithms depending on the current context and conditions on which the camera is operating (Fig. 18). The camera becomes active in order to interpret the current object that it is trying to understand. So the idea is to be able to move and learn, getting the camera to automatically move around an object in order to learn more about the interpretation of the object (Fig. 19)...obtaining more information from other views, finding the views that are optimal to understand the given object and object manipulation with the eye-in-hand setup.
Fig. 19
This is being used with a second robot we've seen in this talk, this one is called ARVID (Fig. 20). Here is the stereo camera head on top of ARVID and that is being used in navigation in unknown environments and being able to move around in a corridor without bumping into things...again, parallel processing being a very important part of that.
Fig. 20
It is turning out that medical image analysis is an area, as well as robotics, where we are finding a lot of integration of speech and NLP and vision processing because we have got pictures of the human brain or the heart or internal body organs (Fig. 21). There is vision interpretation of X-rays or CT scans or magnetic resonance and the whole point is to do detection of anomalies or computer-aided teaching and assisting doctors in their operations or so on. You also have medical reports with these pictures describing what the problem is in a given picture and those medical reports can be interpreted by speech or an NLP system.
Fig. 21
Fig. 22
This is another area where we are seeing an integration of language and vision processing ...another area where we are seeing it as well is MultiMedia and virtual reality and video conferencing (Fig. 22). One of the applications we are working on in our IntelliMedia program is smart Video Conferencing where a person in the room will be able to speak to a Video Conferencing camera and ask it to move around, pointing it to a given person or pointing at an object or zooming in on a given object or a person in the room. That being speaking to cameras but also interpretation of gesture, hand signals and so on during Video Conferencing application. Those techniques can also be used for security surveillance and so on.
Fig. 23
Going back to the third department involved in this IntelliMedia 2000+ project, this is again a platform developed by the Center for PersonKommunikation (Fig. 23). This platform is like an operating system for developing spoken dialogue systems. It will also be used for the integration of image and vision processing into the spoken language processing system as well. So here it's got a Dialogue Description Language (DDL) tool which enables the user to build up dialogue descriptions that will be used in applications and it's got about 30 different symbols here indicating different types of action to be done during the dialogue system's interaction with the user. It is a graphical description language which the user builds and then it's automatically compiled into the operation of the dialogue manager for dialogue interaction with the users. It's got the normal linguistic natural language component as well: Grammars, Lexica, Semantic rules, Parsers and so on...and then there is a communication manager which communicates with the device drivers for mouse, display, text to speech, speech recognizer and any other applications like a vision system or a robot which need to be linked into this Generic Dialogue System (GDS). It is actually a tool for developing dialogue interaction applications with users.
It is not the only tool of this type. There are other tools being developed by Vocalis in England and the Oregon Graduate Institute (OGI) in the US and by Philips. They have developed a spoken language dialogue development tool as well. All these platforms are becoming important as well, we are seeing then being used a lot more especially for language and vision integration. Just to give an example, here's an example of a book club (Fig. 24). It says "Welcome to the automatic book club service", generate information, "please give your member number...", check number, overviewing ordering, canceling, "...you wish to continue?", "yes", "no", finishing. There are lots of different paths going through a given dialogue with a given system. Here is a dialogue description language which would be input into the system so that the system would be able to handle this type of dialogue with the user, specifying the dialogue for this application.
Fig. 24
Currently the system is being used for developing a banking application which works in real time over the telephone for doing things that you would normally do with your bank like inserting money, taking money out, transferring money, and so on...and it is being used in a number of different EU projects as well as national Danish projects. It has been used for about 14 different applications.
Another application is a flight reservation system, a typical application in this area of dialogue processing (Fig. 25). The idea here is that the system works over the telephone, the Dialogue Description Language is used for developing the dialogue, speech recognizer, linguistics synthesis and analysis are involved as well. It links back with the telecommunications network back to the flight reservation database. This system is currently being tested by Scandinavian Airways for use in Scandinavia for airline reservations.
Fig. 25
Another area we are looking at is NLP of angiogram reports which are reports of people who have medical problems with their blood vessels in their brains. For example, the picture of a brain here (Fig. 26)...we are not looking at consciousness here... we are looking at the actual blood vessels in the brain, so it is a different aspect of looking at brains. There are blood vessels in the brain and sometimes there are abnormalities which are usually caused by genetic reasons or accidents and it needs to be fixed. It can be treated if it is not such a serious problem over time. The doctor does a medical report (Fig. 27) as well as these X-rays being taken. The medical reports are usually short, saying that "There is a vascular malformation in the left chrodinal fissure extending into the thalamus supplied by the anterior and posterior chrodinal arteries and measuring about 2.5 cm." What we notice about this language is that it is very domain specific and it refers to brain regions and blood vessels and veins and arteries and there is also a lot of spatial language. It is talking about "IN the left the chrodinal fissure EXTENDING INTO the thalamus supplied BY the anterior and posterior chrodinal arteries". You find this a lot in language and vision integration processing -- you find a lot of information with language using spatial relationships. Also, you will find a lot of reference to objects in the environment or in the scene like in this case arteries and veins and brain regions. So the idea here is that we are looking at the language processing of these medical reports and integrating that with 3-D vision processing of the medical pictures from the X-rays and doing 3-D reconstruction form the X-ray pictures to locate the lesions on the medical pictures.
Fig. 26
So we are looking at representing the meaning of the medical report here, we have vascular malformation size, and also what is being fed, the superior cerebellar vermus, what it is being fed by, two small branches of the cerebellar artery, and it is on that, and whether or not treatment is being recommended by the doctor. So we have got objects, locations, topology, spatial relationships, and whether or not treatment is being recommended, for most of these medical reports...using Prolog definite clause grammars to parse the input and using island parsing to match up with input, finding topology like "on", "to", "by", modifiers like "lying", "arises", "directly", "extending", "feeding" and "into" and so on, and objects like "vascular malformations", "lesions", "cerebellar vermus", "lateral ventricle" and so on all being used in the definite clause grammar to match to the input.
Fig. 27
Turning now to dialogue and NLP, which we believe is a very large part of Intelligent MultiMedia -- what do we think the priorities are in this area. Obviously, pragmatics, that is part of the truck that we couldn't see how long it was, is also very important because visual processing and information is a big part of pragmatics in multi-agent settings, in places like Video Conferencing where you will have many agents. Development of lexicons is very important...automatic information extraction and retrieval and innovative methods for parsing. Obviously the integration of speech and text processing is also a problem as well as the integration of speech, text, and vision processing. Software engineering tools and platforms we mentioned already for doing the integration, the knowledge acquisition, and the testing, those platforms like the Generic Dialogue System (GDS) becoming very important today, so we don't always have to start from the beginning with each new application. Of course, this one, this is Real World Computing here, integration of language and vision processing bringing everything back together again.
Just to show how difficult it is to do language processing even with simple examples...when I first went to the US, in New Mexico, I saw this sign everywhere that said "Ped X ing," and I didn't know what it meant (Fig. 28). But all the Americans knew what it meant and the reason wasn't because I couldn't read it. I could read as well if not better then most Americans, but it was a matter of contextual or pragmatic information that had been provided by learning over time. Also when Americans come to the British Isles, and they go into a shop and see some nice sweet things in local bakeries, sometimes they ask for "Hot X Buns". But they are actually not "Hot X Buns", they are "Hot Cross Buns". This is actually "Pedestrian Crossing." After you know what it is, it's OK, but this is the sort of thing that is learned a lot from context and from the local environment rather than from the knowledge of language in general. You can work it out by the location of these signs giving a lot of information about its interpretation. The location of the sign means the visual environment, the visual context, and that is where you need to have vision and language integration.
Fig. 28
Let's talk a little bit about recession. Everybody knows what a recession is...even Japanese people know what a recession is. We have them in every country in the world at some time or other. Here is a dialogue between a journalist and an economist (Fig. 29). The journalist says, "How do you know we are in a recession?" The economist says, "The last three months have shown a downturn in GNP." The funny thing about GNP is that everybody knows it is bad if it goes down and good if it goes up but most people don't know what it is. So this is a smart journalist and he is trying to help people saying what is GNP. The economist says what it is. And then the journalist says, "Tell me more about GNP" and the economist says more about GNP, and the journalist says, "Explain the difference between GNP and GDP" and the economist explains that...and that dialogue can be put into what we call an intention graph which shows request for information, a request for explanation, a request for elaboration, and then going back to a request for information again. That can be applied to the questions being asked by the journalist and also the answers supplied by the economist.
Fig. 29
Notice that the graph shows nothing about the meaning of what is being talked about in the dialogue and it shows nothing about the structure of language used by the journalist and the economist. It only has information about the pragmatic content or why a given utterance is being used in the dialogue and also the sequence of those intentions in the dialogue.
Fig. 30
Also we can give meaning to those intentions (Fig. 30). For example, a request for information would be an intention, or a plan to achieve a goal, "How do I cook this dish?" "How do I print this file?" Description would be "What is a UNIX?" "What is a pipe?" "What is a filter?" Instruction would be "Has the file been printed?" Elaboration, "Tell me more..." about something, Confirmation, "Can I remove a directory with files in it?" Explanation, "What does cp -r mean?" ...and Guidance, and so on. We can give names to these intentions and definitions in a dialogue situation.
Fig. 31
Fig. 32
What we find is that subjects who have not used UNIX before, or have not used a particular operating system before, ask a lot of questions over in this (top left) region of the pie chart (Fig. 33-b). There are questions for guidance, explanation, instruction, confirmation, description, elaboration, and repetition. Over here (Fig. 33-a), where people are more expert, there are less requests for those types of information in a dialogue. ...and we do some statistical analysis of the data using T-tests and Chi-squared tests across the two sets of subjects (Fig. 34), more experienced and less experienced. We get significant differences in the frequencies of the intention types (Fig. 35).
Fig. 33
Fig. 34
Something even more interesting is the following...if we build up the intention graphs for those subjects, these are subject numbers, 3,4,5, and so on (Fig. 36-a,b) ...this one up here is for the more experienced subjects with UNIX(Fig. 36-a). This is the one for the less experienced subjects which have not used UNIX much before (Fig. 36-b)...what we find is that individual subjects like subject 4 has gone..5 times from "information" requests to "explanation" requests. Subject 4 is looping on "explanation" requests, subject 7 is going 5 times to "confirmations" from "information". Subject 8 is looping onto "confirmations", subject 1 is going 3 times from "information" to "instruction", subject 5, 3 times. 1 and 7 are looping onto "instructions" requests here, and notice there are less loops, there are only 3 loops up here (Fig. 36-a). Down here (Fig. 36-b) there are 5 loops, on this one. So what this is showing is that individual subjects have high frequencies of intention types or request types in a dialogue with a system or during a Wizard-of-Oz experiment. That is very good news because that means we can put that information into computer programs which are processing that.
Fig. 35
Fig. 36a
Fig. 36b
What we find is that people who are more expert in a dialogue in a system will ask for information on descriptions most of the time, and they won't need to go too far down on this (Fig. 37). They won't need to go too much down on this, to "explanations" or "guidances", or "repetition" requests. There you will find the less experienced subjects, asking a lot of questions. So we can put that into a computer program, this is another computer program, it is natural language dialogue system which has been implemented for answering questions on the computer operating systems UNIX and MS-DOS in real-time (Fig. 38). It is written in Quintus Prolog, and it's got your usual data component for representing data about operating systems, and a generator for generating back English output, or input English, and dialogue component here. This is the most interesting part that we are focusing on at the moment.
Fig. 37
Fig. 38
What we can do is, we can built an intention matrix here (Fig. 39). This
is measuring the frequencies of the intentions coming in on the input, and
particularly the pairs, the sequencing of the intentions on the inputs,
and it is measuring the satisfaction and dissatisfaction based on these
intention frequencies in the input. So, here, 'How does he see my file?'
- 'More Fig. 39 Fig. 40 That is looking at intention analysis in language understanding and dialogue.
It is also going to be very important for vision processing and integration
of vision and language processing, and recognition of intentions, syntax,
semantics, pragmatics, representation of intentions, and the use of intentions
for cooperative user responses again in the dialogue, or even if you are
doing it for a vision system in the visual environment. Fig. 41 Fig. 42 Just a little advertisement...at Aalborg University in Denmark, we have
got a 1 and 1/2 year Master degree course in Intelligent MultiMedia, bringing
all that work together from the different departments, computer science,
and so on (Figs. 41 and 42). In Denmark, education is free, even for non-Danish
and non-EU citizens -- that is good news for a lot of people. Of course
they don't pay for food, accommodation, and drinks, only for education.
So people who are interested in that, they can look up the Web address that
I showed you (p.38, Fig. 13). One of the big problems that is being referred to a lot in the field,
the philosophical problem that has been talked about by Steven Harnad and
by John Searle is the Symbol Grounding Problem (Fig. 43), or the Chinese
Room Problem (Fig. 44). That is where they talked a lot about how to solve
grounding symbols in computer programs like "dogs" and "cats", and primitives
in natural language processing programs. Harnad and a lot of people believe
that the answer to that is by doing what we are talking about here, today
-- that is, integrating language and vision. The vision provides the grounding
for the language information. People are asking philosophical questions.
This is a duck (Fig. 45), but it is also a rabbit (turning the slide)...
and people have been asking the philosophical question: is that ambiguity
in that picture, the same as the ambiguity in the language that I showed
at the beginning of my talk. What is the difference between ambiguity and
vision, and ambiguity and language, if there is any difference? Fig. 43 Fig. 44 Fig. 45 Just to finish up then, advertisement again...there are some books on
this area of integration of natural language and vision processing published
by Kluwer Academic Publishers (Figs. 46 and 47). Volume I deals with computation
models and systems...this deals with computer programs and computational
models for doing language and vision processing, and integration. Oka-san
has got a couple of papers in here, one giving a general overview of RWC,
and another one in more detail...and then number II is focusing on Intelligent
MultiMedia. That is looking more at the MultiMedia side rather than the
intelligent side, and that is Volume II (Fig. 46). Volume III is looking
at theories and grounding representations (Fig. 47). This one is looking
more at the philosophical and computer science issues, and looking at symbol
grounding and solutions for the Chinese Room Problem, and so on. There is
also a 4th book...I don't have a picture...it is a red book...and that's
on recent advances in this area of language and vision integration, and
Okada-san has got a paper in that one. There is also a book on more cognitive
science aspects to this whole Real World Computing initiative, going back
to consciousness, which we heard about a lot in the previous session. This
book "The Search for Mind, A New Foundation for Cognitive Science" published
in 1995 (Fig. 48). This looks at it from a computer science and a philosophical
point of view at integration of language and vision processing. So what
we are talking about is language and vision integration, and also a lot
of people are working on processing culture and society information and
particularly emotional information is very important. Going back to the beginning,
to the faces...face recognition, gesture recognition, and body language
recognition...a lot of work in Japan on emotion processing...I notice alot
of papers on that. Fig. 46 Fig. 47 Fig. 48 Last slide now (Fig. 49)...as a formula for the future...this CS stands
for Cognitive Science and for Computer Science as well...both, equal to
I cubed (I**3), this information here meaning we are going to be looking
more at qualitative theories of information, rather than quantitative theories
as talked about by Shannon...that means looking at semantics of information,
and pragmatics of information, as well as syntax. Integrating information
that means integrating visual information, and language information, text
information, and spoken information. Finally, something that I stressed
a lot during my talk is particularly from the language processing point
of view -- a large part of the information is about the processing of intentions,
because most of the philosophers of language like Searle and Austin argue
that when we say things or do things in the world, it is because we have
got intentions in the world, and we want to achieve those intentions. So
a good way to understand the language or vision is to go back to processing
the intentional information in the dialogue or in the visual environment...
Fig. 49 
