Trends in Image Processing Researches and Activities of RWC Image Processing Groups
|
Yoshiaki ShiraiProfessorDepartment of Computer-Controlled Machinery Systems, Faculty of Engineering, Osaka University |
|
|
Yoshiaki ShiraiProfessorDepartment of Computer-Controlled Machinery Systems, Faculty of Engineering, Osaka University |
Researches on image processing are becoming increasingly active each year. As many as 700 presentations were given at ICPR (International Conference on Pattern Recognition) held in Vienna at the end of August this year. As ICPR became too big, ICCV (International Conference on Computer Vision) has been held since 1987. Then, ECCV was born in Europe and ACCV was born in Asia, both of which have since grown. The original ICPR itself was divided into four areas several years ago, each with its respective Program Committee. The conferences are held at the same time, and each has covered a broader scope and the proceedings of the conferences have become thicker year by year.
Since it is difficult to address image processing extensively, in this report, first the trends in overall image processing will be overviewed, then the trends in research closely related to RWC will be briefly discussed, and finally related activities of the image processing groups in RWC will be introduced. Note that the illustrations used here may not be the best; they are used mainly for their availability.
Image processing may be divided into offline processing like character recognition or microscope photo analysis and real-time (online) processing such as a robot detecting the location of an object and grabbing it. Although real-time processing should be ideal, only simple defect detection and positioning of known objects have been implemented until recently due to performance restrictions.
Now, improved performance broadens the scope of real-time processing. For instance, experimental systems which detect and track moving objects and those which recognize human facial expressions and take actions have been constructed. In real-time processing, a human can confirm or help even if a system makes mistakes occasionally, which broadens the applicable scope, such as gesture recognition and presentation of image candidates which represent specified contents from an image database.
An image does not always contain sufficient information on a scene. A more complicated processing may be conceivable for this, but this can now be resolved by obtaining more information. Typical examples are sensor fusion, which makes use of many types of information, and active vision, which attempts to obtain information with various input conditions.
Real-time image processing was mainly applied to automated factories in the past, but now its use has extended to monitoring, CG (computer graphics), and image compression (animated image compression). For instance, in CG, there have been many attempts to generate images viewed from many directions based on several images from different directions (MIT, INRIA (France), Japanese companies, etc.). Images can be generated in real time when corresponding portions in several images can be extracted. Currently, corresponding positions have to be provided manually, or human interaction is often required after automatic image processing. Researches for complete automation are under way.
In recent years, the number of studies on human face or gesture recognition has been increasing. While ICPR has sessions entitled "Looking at People" and "Face Detection and Recognition", Movement Analysis and Neural Network sessions also handle human tracking and facial expression recognition.
Although face recognition has been addressed starting from pattern recognition, it was the field of communications that first handled changes in facial expressions and faces of moving people. Once changes in partial faces are perceived, transmitted information can be compressed. Furthermore, once facial expressions or gestures can be recognized, only the results are transmitted so that recipients can synthesize images.
Then, as an interface to a computer, attempts have begun to recognize human identification from a face, the orientation of a face, facial expression, the direction of eyes, and the movement of hands. In RWC, researches on detection of eyes and mouth from a face, human identification from his/her face, and gesture recognition are in progress.
Moreover, attempts to recognize movement of human fingers with a camera instead of data gloves used in sign language recognition (MIT Media Laboratory and Hitachi) and VR (virtual reality) have also appeared. Although precision and processing speed are still to be improved, they are useful for some applications. Fig. 1 shows an example of our recognition result of the three-dimensional posture of fingers from a series of silhouette images.
When the target of image processing moves from a factory to the real world, vision systems which function under various conditions are required. For such robust visions, information integration at different levels is necessary as shown below.
Integration of Information from Multiple Cameras
For instance, while stereo vision with two cameras is based on the two-eye vision of humans, three-eye vision improves reliability. In CMU, real-time stereo vision is built using six cameras. In addition, so-called active vision has also been proposed which facilitates stereo vision by moving cameras continuously.
Also in tracking moving objects, there is a distributed camera system, where one of the vision systems can obtain an image of objects under good conditions at any point of time by positioning vision systems in many points to communicate the results of information processing of each one.
As shown in Fig. 2, in RWC, research is going on to obtain the distance information with stereoscopic vision using nine cameras. In regular stereo vision, distances are often not obtained because there are those points which can be seen from one camera but cannot be seen from the other camera where the distance is discontinuous. However, multi-eye vision enables distances to be obtained even in discontinuous portions. Once the distance can be obtained, the location of the viewer can be found and an image seen from there can be generated, as shown in Fig. 2. As perspective changes, appearance also changes, so stereoscopic vision is possible. Since discontinuous portions constitute the contours of the object, which are important to humans, it is useful to obtain this distance.
In RWC, another research is going on to obtain appropriate focal distance and iris for locations of images using multiple cameras.
Integration of Information from Various Types of Sensors
When a robot grabs an object, its behavior is made reliable by combining visual information and tactile information. Using infrared images in human identification ensures extraction of warm parts in the face and hands. These are called sensor fusion.
In RWC, brightness information and motion information are combined, or motion information and distance information are combined, to track multiple moving people. As shown in Fig. 3, tracking is possible by integrating motion information and distance information, even when one person cannot be separated with either information alone.
Information Integration at Higher Level
It is important to estimate how reliable the obtained information may be when information about an object is obtained from an image. If it is sufficiently reliable, the result serves as a conclusion; otherwise, by obtaining further information, reliability may be improved. In this case, it is more efficient if it is possible to determine what information should be obtained. When an object is well-defined such as mechanical parts, it is relatively easy to express reliability as probability. However, for outdoor scenes, a certain probability model must be assumed because of the broad nature of objects.
Fig. 4 shows an example of a mobile robot matching a street scene with a map in this way. In this example, the type of intersection was not clear from the first captured image (a), and then the robot checked to see if a road existed at right (b) based on the recognition result and found that it was indeed a crossroad.
| (c)First image and derived probability distributiion of type of intersection | (d)Second image and derived probability distribution of type intersection |
In addition, there are attempts to ensure robust recognition by creating a model of color variation in outdoor scenes susceptible to the time, weather, and shadows. Real-time Vision
The main reasons for performing image processing in real time are: 1. Experiments mandate real-time vision. As in tracking of moving objects, a camera must be moved using the vision result and images must be re-input. 2. When images must be seen many times during a job. 3. When vision is used for the human interface. 4. Vision as image compression technique for storing and transferring animated images. For instance, capturing the movement (optical flow) in images.
Finally, it is an essential technology for disseminating multimedia and for this purpose LSIs to extract optical flow at high speed have been developed. This is also of great use in conventional image processing.
Let's estimate the required computation in real-time vision. Assume that an image has 30 frames/sec, resolution of 256 x 256, operation for each pixel is performed by referring to its neighboring 10 pixels, and 10 stages are required to obtain the result. Assuming the operation consists of one multiplication and one addition, it results in 200 million multiplications and additions per second. This may be realized on a supercomputer, but it costs too much.
In the past, many special devices dedicated to image processing were built for accelerating image processing. However, they are applicable to only a limited kind of processing and if the remaining processing was performed by a general-purpose computer, this in turn became a bottleneck that hindered real-time processing. That is, only processes executable on existing devices have been processed in real time.
For instance, CMU and SRI International built real-time systems for stereo vision and animated image processing on a trial basis. These use specialized LSI, thus they are quite expensive.
Real-time processing is often realized using data cubes to enhance generality, but this is restricted to low-level processing, and thus is less flexible. Another attempt is parallel processing with transputers; transputers are general-purpose and very flexible but their cost performance is poor.
To process a great amount of uniform data such as images, multiple DSPs should be used. A system has been built which computes the optical flow inside images with 5 DSPs. By adding two more DSPs to track the region which moves in a similar way using the optical flow, moving objects may be tracked with a camera. Such flexible systems are necessary to verify proposed algorithms by experiments.
For this and for the convenience of image processing, DSP boards with image memory and a signal controller are desirable. By manufacturing boards that can be used for as many applications as possible, many researchers will benefit from it and at the same time the cost of boards can be lowered. In 1994 the RWC Image Group concurred in a Workshop on the need for DSP boards, and in the beginning of 1995 the specifications of the common board were determined and requests were solicited from researchers of image-related area from RWC and those engaged in the Grant-In-Aid for Scientific Research on Priority Area from the Ministry of Education for the first manufacturing. The board contains one DSP (TMS320C40) suitable for image processing, I/O memory, program memory, work memory, and control FPGA. Using this and a video board for AD/DA conversion of image signals enables digital processing of video signals.
DSP boards were manufactured at the end of last fiscal year, and currently systems using them are being built. For instance, the tracking of moving objects mentioned above is possible with this boards. In addition, to make use of the distance as well as the movement, a system with additional DSP boards which compute the distance with stereo (Fig. 5) is being developed.
In the future, the programming efficiency will be improved by promoting the sharing of software. The results from many studies as well as RWC image processing research may be experimented in real time with this board system.