Parallel Application Toshiba Laboratory has successfully developed a parallel data mining technology which improves the performance by a factor of 14 using a cluster consisting 16 PCs as a part of parallel data mining research.
As network businesses such as e-commerce proliferate, marketing strategies of enterprises are changing. From now on, it is increasingly important for corporate management to quickly analyze information available on networks to speed up decision-making. Data mining is highly expected as a tool to extract the data required for quick decision-making.
In general, data mining involves handling vast databases, thus the processed volume is enormous. For this, expensive large server computers or supercomputers are used, or the data volumes to be analyzed are reduced by pre-processing. However, this limited the applicability of data mining not fully exploited for analyzing data available on networks.
This new parallel data mining technology was developed on a PC cluster consisting of 16 regular PCs. PC clusters can deliver the performance equivalent to large server computers at several tenths of cost. The RWCP is developing PC clusters and basic software environments on them, and has demonstrated performance levels comparable to that of supercomputers in such applications as benchmark programs, protein analysis, and electric power transmission system simulations.
Unlike scientific calculations, data mining requires parallelization that takes disk access into consideration, so it was difficult to achieve sufficient performance using PC clusters with conventional techniques. In parallelization by our unique data mining technique, slice & dice analysis, we have successfully achieved the performance comparable to large server computers by developing a new data processing technique on PC clusters.
Decision trees are representative technique for data mining, and they allow us to extract the relationships among elements in a database in tabular format and automatically classify new data based on the acquired knowledge. For instance, customers who tend to respond to direct mail may be classified by such attributes as age, profession, and purchase history.
For parallel processing of decision trees, it is necessary to create decision tree structures efficiently. Since the load changes depending on the shape of trees, efficient parallelization had been difficult to achieve. We developed a technique to dynamically balance the load on each node of a PC cluster and a technique to minimize disk I/O operations, with both of which a cluster with 16 nodes can process a job in just 21 minutes that takes 4 hours with a single node.
Slice & dice analysis is our unique data mining technique. This partitions a multi-dimensional database and can automatically find a partition containing the data which satisfies particular conditions. For instance, we can extract goods at a supermarket with a high disposal rate by product attributes, cost, and/or recent sales trend.
In parallel processing of slice & dice analysis, the gathering and summarizing of data distributed across nodes is a bottleneck in parallel processing. To solve this problem, we developed a technique to reduce the amounts of data transferred among nodes by having each node compress the data placed on it, resulting in 14 times faster performance with 16 nodes. With this, 60 GB of data can be processed in 2 hours with 16 nodes.
Note that part of this technology was presented at an international conference (HPC ASIA) held in Peking in May.
Our next goal is 1-terabyte data mining by enhancing the system to 32 PCs. This would facilitate applications like analysis of WWW access logs or receipts of supermarkets. For instance, the access log of a popular Web site for one year with 1 million hits a day can be analyzed in just 2 hours.
We will enhance large database mining as one of the fundamental technologies for e-commerce in the future and implement it as an easy-to-use system.
This technology was introduced in the Nihon Keizai Shimbun (page 15, May 22 (Mon.)).