Adaptive and Intelligent Systems
Applications 1994-1999 - Final Report

4. Methods for demanding neurocomputing applications (MENES)

4.1 Computationally intelligent analysis of large data sets (STELLA)

Summary

The purpose of the Stella project was to develop methods to analyze complex multivariate data with computationally intelligent methods. Algorithmically this was mainly carried out using the tree-structured self-organizing map, which is a fast and robust variant of Kohonen's SOM algorithm. To understand the problem better the relationship between statistics and neural networks was included in the study. This led us into deeper investigation of how data analysis and knowledge discovery techniques are best applied in practice.

Main results

The main result of the research was a generic framework for computational data analysis. It has been implemented in the NDA (Neural Data Analysis) software package which has about one hundred different neural and statistical algorithms, including interactive graphics and data manipulation. Although the main intention of NDA is to support the use of self-organizing maps, it is a multipurpose package for all types of statistical computing and data visualization.

The NDA software has been designed for embedded intelligent software solutions. Therefore it has a platform independent core, written in Ansi-C, that compiles under different hardware and operating systems. Interfaces to other systems are built by sending and receiving ASCII stream commands to NDA and implementing six low level graphical routines (line, polygon, etc.) in the target system. Example user interfaces are written with Xview, Java, Exel and Visual Basic. Several products have been build that are using the NDA kernel as a computational engine.

(Figure is missing, sorry.)

Figure 1. Graphical representation of multivariate data with NDA software.

Project information

Participants, dates and volume

The research was performed at the University of Jyväskylä during 1995-1998. The project budget was about FIM 2.4 million, and the workload was about 120 man months. Industrial partners in the project were BEA Systems Oy, Suomen Gallup Infominer Oy, YIT Service Oy, Kymmene Logistics Oy, Synchron Tech Oy, Omni-Weight Control Oy, NNets Oy and Sonera Oyj.

Project leader

Prof. Pasi Koikkalainen
Department of Mathematical Information Technology
University of Jyväskylä
P.O. Box 35, FIN-40351 Jyväskylä, Finland
Tel: +358 14 60 2763
Fax: +358 14 60 2731
E-mail: pako@mit.jyu.fi

Publications

Koikkalainen, P. (1994) (Toim.), Neurolaskennan mahdollisuudet, TEKESin julkaisusarjan julkaisuja 43/94, Tekes, 152 pp. (In Finnish.)

Koikkalainen (1994), The Self-Organizing Template: natural way from pixels to representations, In proc. ICANN'94, Vol 2, Springer-Verlag, pp. 1137-1140.

Koikkalainen, P., (1994), Intelligent computing technology, Workshop on Machine Vision for advanced production, eds. Pietikäinen and Hakalahti, University of Oulu.

Koikkalainen, P., (1994), Progress with the Tree-Structured Self-Organizing Map, In proc. ECAI 94: 11th European Conference on Artificial Intelligence, ed. A. Cohn, John Wiley & Sons, Ltd., pp. 211-215.

Heikkonen, Koikkalainen (1994, to be published in 1996), Self-Organization and Autonomous Robots, Special issue on Neural Systems For Robotics in the Progress in Neural Network Series, Academic Press.

Koikkalainen, P. (1995, to be published in 1996), Tuottavaa tutkimusta yhteistyöllä?, PK-yritykset ja teknologiakeskukset avainasemassa, to appear in Tietojenkäsittelytiede 1/96. (In Finnish.)

Koikkalainen, P. (1995), Fast Deterministic Self-Organizing Maps, ICANN'95, International Conference on Artificial Neural Networks, Oct 9-13, 1995, Paris. Publisher CE2 & Cie, pp. 63-68.

Heikkonen, J., Koikkalainen, P. (1995), Computer vision system for analyzing air flows, in proc. EANN 95, International conference on engineering applications of neural networks.

Mononen, J., Häkkinen, E., Koikkalainen, P. (1995), Customer Analysis through the Self-Organizing Map, ICANN'95/Industrial conference, session 8 proceedings, Oct 9-13, 1995, Paris. Publisher CE2 & Cie.

Koikkalainen, P. (1995), Neurolaskenta tuo älyä elektroniikkaan - Piirit ja ohjelmistot hermottavat verkon, Prosessori, 4/95, pp. 34-39. (In Finnish.) Koikkalainen, P. (1995), Logistiikan strategiat selviksi neurolaskennalla, Automaatioväylä, 7-1995, pp. 9-10. (In Finnish.)

Koikkalainen, P., Varsta M. (1996); Robot path generation for surface processing applications via neural networks, In proc. SPIE Proceedings 2904, Intelligent Robots and Computer Vision XV. D. Casasent (ed.). pp. 228- 237.

Koikkalainen, P. (1996); Fault diagnostics of rotating machnies via self-organization, In proc. SPIE Proceedings 2904, Intelligent Robots and Computer Vision XV. D. Casasent (ed.). pp. 460- 468.

4.2 Measurement-based monitoring and modelling of complex systems

Abstract

In the modelling and control of industrial processes, it is usually assumed that a global, analytical system model can be defined. However, many processes are so complex that it is difficult or even impossible to model them analytically. In such cases, Artificial Neural Networks (ANNs) can be used. ANNs build models directly based on process measurements, and thus provide a means to analyze processes without an explicit physical process model. ANNs can also be used as "soft sensors" to estimate signal values or process variables that are difficult to obtain or can only be measured off-line. The use of the ANNS, however, requires that a large amount of high quality, stable, numerical data describing the process are available. Modern automation systems are capable of providing large amounts of data and, thus, efficient methods for data exploration and mining are required.

The Self-Organizing Map (SOM) is one of the most popular neural network models. The SOM algorithm is based on unsupervised learning, which means that the training is entirely data-driven and that little or no a priori information about the input data is required. The SOM can be used for clustering data without knowing the class memberships of the input data. The SOM can, thus, be used to automatically detect features inherent in the problem. This is a clear advantage when compared with the ANN methods based on supervised learning which require that the target values corresponding to the data vector are known. The SOM has successfully been used in various engineering applications covering areas such as pattern recognition, full-text and image analysis, financial data analysis, process monitoring and control and fault diagnosis [3].

In this project, new SOM based methods have been developed for the analysis of measurement data obtained from an industrial process environment. The goal was to monitor and model complex systems using adaptive and intelligent methods. In addition to process measurements, data from various data bases can be used in the analysis of industrial systems. Data mining and exploration as well as visualization methods based on the SOM algorithm have been investigated in the project.

Results

The scientific results achieved in the project have been tested in various industrial applications, especially in the forest industry and steel production. The main results of the project are SOM-based software packages for data analysis. The SOM Toolbox for MATLAB has been developed at the Laboratory of Computer and Information Science and the development work has partly been carried out in this project. The software is available in the public domain. In addition to this software, two more specific programs have been developed in the project for industrial partners: (1) ENTIRE - Dynamic, neural networks-based forest industry simulator for Jaakko Pöyry Consulting and (2) Computer-aided tools for online control of a steel production line for Rautaruukki Strip Products. The ENTIRE simulator was partly developed within a separate project financed by Tekes and coordinated by Jaakko Pöyry Consulting. The engineering results have been reported in five Master of Science theses and in one Licentiate thesis.

The Scientific results of the project include 11 international publications: (1) two international journal articles, (2) two book chapters, and (3) seven international conference papers listed below in section 4. One of the conference papers was an invited talk [7] and one was a keynote address [8].

Project information

Participants

Jaakko Pöyry Consulting Oy
UPM-Kymmene / Wisaforest Oy
Rautaruukki Research Center (Raahe) and Rautaruukki Strip Products (Hämeenlinna)
Leonia Bank (formerly Postipankki Oy)

Project dates

Pre-project: September 1, 1994 - February 28, 1995
Project: March 1, 1995 - February 28, 1998

Project volume

FIM 1.754.000

Project manager

Prof. Olli Simula
Helsinki University of Technology
P.O. Box 2200, FIN-02015 HUT
Telephone: +358-9-4513271; GSM: +358-500-746852
Fax: +358-9-4513277
E-mail: olli.simula@hut.fi

Publications

1. O. Simula, E. Alhoniemi, J. Hollmén, and J. Vesanto, "Monitoring and Modeling of Complex Processes Using Hierarchical Self-Organizing Map", Proc. IEEE 1996 International Symposium on Circuits and Systems (ISCAS'96), Supplement to Vol. 4, Atlanta, Georgia, USA, May, 12-15, 1996, pp. 73-76.

2. E. Alhoniemi, O. Simula, and J. Vesanto, "Monitoring and Modeling of Complex Processes Using the Self-Organizing Map", to be published in Proceedings 1996 International Conference on Neural Information Processing (ICONIP'96), Hong Kong, September, 24-27, 1996, pp. 1169-1174.

3. T. Kohonen, E. Oja, O. Simula, A. Visa, and J. Kangas, "Engineering Applications of the Self-Organizing Map", Proceedings of the IEEE, Vol. 84, No. 10, October 1996, pp. 1358-1384.

4. J. Hollmén and O. Simula, "Prediction Models and Sensitivity Analysis of Industrial Production Process Parameters by Using the Self-Organizing Maps", Proceedings of the 1996 IEEE Nordic Signal Processing Symposium, Espoo, Finland, September 24-27, 1996, pp. 79-82.

5. J. Vesanto, P. Vasara, R.-R. Helminen, and O. Simula, "Integrating Environmental, Technological and Financial Data in Forest Industry Analysis", Proceedings SNN'97 Europe's Best Neural Networks Practice, Amsterdam, The Netherlands, May 22, 1997, 4 pp.

6. J. Himberg and O. Simula, "Analyzing an Automatic Call Distribution System by the Self-Organizing Map”, Proceedings of the 1997 Finnish Signal Processing Symposium, FINSIG'97, Pori, Finland, May 22, 1997, pp. 153-157.

7. O. Simula, E. Alhoniemi, J. Hollmén, and J. Vesanto, Analysis of Complex Systems Using the Self-Organizing Map, (invited paper), to appear in the Proceedings of the 4th International Conference on Neural Information Processing (ICONIP'97), Dunedin, New Zealand, November 24-28, 1997, pp. 1313-1317.

8. O. Simula, P. Vasara, and J. Vesanto, Analysis of Industrial Systems Using the Self-Organizing Map (invited keynote paper), 1988 Second International Conference on Knowledge-Based Intelligent Engineering Systems (KES'98), Adelaide, Australia, April 21-23, 1998, pp. 61-68.

9. E. Alhoniemi, J. Hollmén, O. Simula, and J. Vesanto, "Process Monitoring and Modeling Using the Self-Organizing Map", accepted for publication in Journal of Integrated Computer-Aided Engineering, Special Issue on Neural Techniques for Industrial Applications, John Wiley & Sons, 1998.

10. O. Simula, P. Vasara, J. Vesanto, and R.-R. Helminen, The Self-Organizing Map in Industry Analysis, to be published in Intelligent Techniques in Industry, edited by L.C. Jain, CRC Press, 1998, 27 pp.

11. O. Simula, J. Vesanto, E. Alhoniemi, and J. Hollmén, Analysis and Modeling of Complex Systems Using the Self-Organizing Map, to be published in Neuro-Fuzzy Tool and Techniques, edited by N. Kasabov, Springer-Verlag Group (Physica-Verlag), 16 pp.

Figure 1. Data analysis and visualization using the SOM Toolbox: U-matrix and the component planes. The distribution of the parameter values are visualized using component planes which are obtained as a sliced version of the map. The cluster structure of the data is shown by the U-matrix.

4.3 Using external knowledge in neural network models

Abstract

One of the most important properties of neural networks is generality, as the same network can be trained to solve quite different tasks, depending on the training data. This is also one of the most prominent problems when practical real world problems are solved by neural networks, as existing domain knowledge is difficult to incorporate into the models.

The goal of this research project was to develop methods for adding prior knowledge to neural network modeling. The studied approach is based on training the knowledge on the network instead of hard-coding the knowledge in advance to the connections or weights. The knowledge is specified as target values or constraints for different order partial derivatives of the network. This approach can be viewed as a flexible regularization method that controls directly the characteristics of the resulting mapping.

A concrete goal of the project was to develop a neural network training and simulating system that supports modular network design and domain knowledge representation with fuzzy-like terms.

Results

The primary result of the project consisted of methods and algorithms for specifying background knowledge from the process as fuzzy-like rules or cost functions, and using such knowledge in addition to pure data to train the neural network models. Our research group has also developed a prototype neural modeling tool that supports the developed methods for including knowledge into the models, and also modular network design, where the networks are built from smaller blocks. The developed tool supports the following model construction principles:

1. Modular network design

Subprocesses of the whole process are encapsulated into modules that can be trained on the data, expert rules and simulation models. The modules are connected with adaptive layers and then the whole model can be trained together, so that the interconnection layers compensate the errors of the individual modules. This facilitates use of inaccurate simplified models for the subprocesses. To improve generalization, any of the modules can easily be specified as a committee of networks, so that the output of a module is averaged over an ensemble of independent member networks.

2. Training background knowledge on the network

The training data base consists of data samples and an additional set of rule data. The rules are of the form: if the system is in state F, then effect of input x_k to output o_i is , where the state F and target R_ik can be specified with uncertain, "fuzzy-like" terms. Thus the rules contain the target values and the membership functions that encode the uncertainty of the knowledge. A practical example of a rule would be: Adding 1 kg of chemical X increases the output Y by 1-3 units.

Figure 1. Overview of the prototype neural modeling tool Q-OPT 2, with main menu, variable control window and network display window. The prototype was developed with Taipale Engineering Ltd. as a basis for a commercial neural modeling application.

Figure 2. Graphical display of sensitivity analysis of the neural model.

Another application pursued in the project was related to real-time load measurement system of Omni Weight Control Ltd., which is based on fine measurement of strains in the load container. The results and software developed in the research project were used to solve the non-linear load estimation tasks in the product.

Project information

Participants

Lab. of Computational Engineering, Helsinki Univ. of Tech. (1996-1998)
Lab. of Information Processing, Lappeenranta Univ. of Tech.
Taipale Engineering Ltd.
Omni Weight Control Oy
Valmet Automation Oy
UPM-Kymmene Oy Kaukas
Enso Oy

Project dates

Project started 1.3.1995 and ended 28.2.1998

Project volume

Total volume of the project FIM 2.360.000
Tekes funding FIM 1.450.000
Industrial partners funding FIM 380.000
Research institutes’ own funding FIM 530.000

Project manager

Jouko Lampinen
Laboratory of Computational Engineering, Helsinki University of Technology
P.O. Box 9400, FIN-02015 ESPOO, FINLAND
Tel: +358 (0)9 451 4827, Mobile: 050 560 4827
Fax: +358 (0)9 451 4830
E-mail: Jouko.Lampinen@hut.fi

Publications

[1] Jouko Lampinen and Ossi Taipale. Optimization and simulation of quality properties in paper machine with neural networks. In Proc. ICNN’94, Int. Conf. on Neural Networks, pages 3812-3815, Piscataway, NJ, 1994. IEEE Service Center.

[2] J. Lampinen and A. Selonen. Multilayer perceptron training with inaccurate derivative information. In Proceedings of the IEEE International Conference on Neural Networks ICNN'95, volume 5, pages 2811-2815, Perth, WA, 1995.

[3] A. Selonen, J. Lampinen, and L. Ikonen. Using background knowledge in neural network learning. In D.P. Casasent, editor, Intelligent Robots and Computer Vision XV. Algorithms, Techniques, Active Vision, and Materials Handling, volume 2904, pages 239-249, 1996.

[4] J. Lampinen. Advances in neural network modeling. In L. Yliniemi and E. Juuso, editors, Proc. of TOOLMET’97, Tool Environments and Development Methods for Intelligent Systems, volume 1, pages 28-36, 1997.

[5] J. Lampinen and A. Selonen. Using background knowledge in multilayer perceptron learning. In M. Frydrych, J. Parkkinen, and A. Visa, editors, Proc. of The 10th Scandinavian Conference on Image Analysis, volume 2, pages 545-549, 1997.

[6] J. Lampinen. Modeling of non-stationary process by modular separation of stability and plasticity. In Proc. IJCNN’98, Anchorage, Alaska, May 1998.

jukka.iivarinen@hut.fi
http://www.cis.hut.fi/neuronet/Tekes/4.shtml
Tuesday, 28-Nov-2000 15:48:44 EET