Currently active researchers in the Laboratory of Computer and Information
Science and Adaptive Informatics Research Centre, Helsinki University of
Technology, Espoo, Finland:
Juha Karhunen, Antti Honkela, Alexander Ilin, Tapani Raiko, and Erkki Oja
Former members of the research group:
Harri Valpola,
Aapo Hyvärinen,
and Petteri Pajunen
In this project, we have considered extensions of basic linear independent
component analysis (ICA) and blind source separation (BSS) to nonlinear
mixture models.
In the following, we first briefly describe in time order the most important
results of the ICA and Bayes research groups of our laboratory on
nonlinear ICA and BSS. After that, we provide some useful general links
to nonlinear ICA and BSS.
More information can be found in the given references which provide links
to central electronic publications, useful web sites, and research
reports.
Our results on nonlinear ICA and BSS during the years 2000-2005 have been summarized in more detail in the research reports [15]-[19], which contain also descriptions of other research topics of our ICA and Bayes groups. Software on our Bayesian methods applicable to nonlinear BSS problems as well as more our publications related to nonlinear BSS and ICA are available on the home pages of our Bayes group [12].
The most important early results of our ICA group [13] deal with the existence
and uniqueness of the solutions of the nonlinear ICA problem. They have been
reported in the seminal theoretical paper [3], where A. Hyvärinen and
P. Pajunen have shown that the nonlinear ICA problem is highly
non-unique and ill-posed without additional regularization. More uniqueness
results can be found in the recent review paper [1].
Our other early results on nonlinear ICA and BSS include a method based on the self-organizing map and a maximum likelihood based approach. They have been reported in Chapter 17 of the book [14] and the maximum likelihood method also in [7]. However, these methods are applicable to small-scale problems only because their computational load explodes with the dimensionality of the problem, and their accuracy is rather limited.
During the last years, we have successfully applied variational Bayesian learning
(called in our earlier papers Bayesian ensemble learning) to nonlinear
BSS and factor analysis. This methodology can be used to the selection,
construction, and unsupervised (blind) learning of suitable latent variable
models for nonlinear BSS. Basic methods for nonlinear factor analysis
and ICA have been introduced in [4]. The simpler nonlinear factor analysis (NFA)
method in [4] finds a nonlinear PCA (principal component analysis) solution.
Its extension, the NIFA (nonlinear independent factor analysis) method then finds
the nonlinearly mixed source signals. The paper [6] extends the method to blind
identification of a nonlinear dynamic model, and our results on this line
of research have been reviewed in [5].
The variational Bayesian methods [4]-[6] can be applied to considerably larger scale problems than our early methods, but even their computational load grows quite large especially for the nonlinear dynamic model introduced in [6]. To alleviate this problem, we have developed variational Bayesian methods based on standardized building blocks [9]. In these methods, all the computations can be carried out locally, resulting in linear computational complexity. A basic paper on the application this approach to the nonlinear BSS problem is [8], and the building block approach is described in detail in the long journal paper [9].
A recent general review article on nonlinear blind source separation and
independent component analysis is [1]. Besides uniqueness of the nonlinear
ICA and BSS problems, it deals with the simpler problem of separating
post-nonlinear mixtures and the variational Bayesian approach to nonlinear
BSS. The paper also contains an extensive list of references. A recent book
on nonlinear blind source separation is [2]. It reviews the main methods,
including the MISEP method developed by the author L. Almeida.
A prominent European joint research effort on nonlinear BSS and ICA was
carried out under the BLISS (Blind Source Separation and Applications) project
funded by the European Union in 2000-2003. Our companions in this project
were INPG, Grenoble, France; Fraunhofer Institute, Berlin, Germany; and
INESC, Lissabon, Portugal. Final report (Deliverable D29) on the results
achieved in the subproject on nonlinear ICA and most important publications
can be found in the web site [10]. In the BLISS project, a lot of useful
software [11] for various nonlinear mixture models was also developed.
A wealth of information on ICA and BSS can be found in our book [14] which has become the standard reference in the field. Links to useful web sites on ICA and BSS can be found for example on the link page of the ICA book [14] and on the home pages of our ICA group [13].
[1] C. Jutten and J. Karhunen,
Advances in blind source separation (BSS) and independent component analysis
(ICA) for nonlinear mixtures.
International Journal of Neural Systems, Vol. 14, No. 5, 2004, pp.
267-292.
- Invited general review article on nonlinear blind source separation and
independent component analysis containing many references.
[2] L. Almeida,
Nonlinear Source Separation. Synthesis Lectures on Signal Processing,
Morgan&Claypool Publishers, 2005, 114 pages.
- New concise book which reviews the main nonlinear blind separation methods,
including the MISEP method developed by the author.
[3] A. Hyvärinen and P. Pajunen,
Nonlinear independent component
analysis: existence and uniqueness results.
Neural Networks , Vol. 12, No. 3, 1999, pp. 429-439.
- Seminal paper on the existence and uniqueness of the solutions of the
nonlinear ICA problem.
[4] H. Lappalainen and A. Honkela,
Bayesian nonlinear independent component analysis by multilayer perceptrons.
In M. Girolami (Ed.), Advances in Independent Component Analysis,
pp. 93-121, Springer-Verlag, 2000.
- Basic paper on our first variational Bayesian (ensemble) learning method
for nonlinear blind source separation.
[5] H. Valpola, E. Oja, A. Ilin, A. Honkela, and J. Karhunen,
Nonlinear blind source separation by
variational Bayesian learning. IEICE Transactions (Japan),
Vol. E86-A, No. 3, March 2003, pp. 532-541.
- Invited paper which concisely reviews our static and dynamic nonlinear
blind source separation methods based on variational Bayesian learning.
[6] H. Valpola and J. Karhunen,
An unsupervised ensemble learning method for nonlinear
dynamic state-space models. Neural Computation, Vol. 14, No. 11,
2002, pp. 2647-2692.
- Long and thorough paper which extends variational Bayesian learning to
blind estimation of a nonlinear dynamic model for the source signals.
[7] J. Karhunen, Nonlinear ICA, in
S. Roberts and R. Everson (Eds.) Independent Component Analysis: Principles
and Practice, Cambridge Univ. Press, 2001, Chapter 4, pp. 113-134.
- A short review which discusses somewhat in more detail a maximum likelihood
method and a variational Bayesian method. Chapter 17 of [11] is a longer
version of this book chapter.
[8] H. Valpola, T. Östman, and J. Karhunen,
Nonlinear independent factor analysis by hierarchical
models, in Proc. of the 4th Int. Symp. on Independent Component
Analysis and Blind Signal Separation (ICA2003), Nara, Japan, April 2003,
pp. 257-262.
- A basic paper on the application of the Bayes building blocks approach
to nonlinear blind source separation.
[9] T. Raiko, H. Valpola, M. Harva, and J. Karhunen,
Building blocks for variational Bayesian learning
of latent variable models, Conditionally accepted to J. of Machine
Learning Research subject to minor revisions, December 2005.
- A long journal paper where the Bayes building blocks approach is
introduced and applied to the construction and variational Bayesian learning
of several example structures.
[10] Final report and most important publications on nonlinear ICA and BSS developed in the European joint project BLISS (Blind Source Separation and Applications).
[11] Software for various nonlinear mixture models developed in the BLISS project.
[12] Home page of the Bayes group in the Adaptive Informatics Research Centre (formerly Neural Networks Research Centre) at Helsinki University of Technology, Finland. Publications and software related to nonlinear ICA and BSS.
[13] Home page of the ICA group in the Adaptive Informatics Research Centre (formerly Neural Networks Research Centre) at Helsinki University of Technology, Finland. Useful information on many aspects of ICA and BSS and links.
[14] Home page of the book A. Hyvärinen, J. Karhunen, and E. Oja, Independent Component Analysis, Wiley 2001. Chapter 17 deals with nonlinear ICA and BSS. The link page of the book provides links to useful websites containing a lot of information on ICA and BSS.
The following reports summarize the research results of the Bayes and ICA
groups in the Adaptive Informatics Research Centre (formerly Neural Networks
Research Centre) at the Laboratory of Computer and Information Science in
Helsinki University of Technology, Espoo, Finland, containing results
on nonlinear ICA and BSS:
[15] Bayes group, 2000-2001. Nonlinear factor analysis and ICA, Nonlinear dynamic state-space models, Application to detection of process state changes.
[16] ICA group, 2002-2003. Comparison studies on blind separation of post-nonlinear mixtures.
[17] Bayes group, 2002-2003. Nonlinear static and dynamic blind source separation, Application to detection of process state changes.
[18] Bayes group, 2004-2005. Nonlinear and non-negative blind source separation, Dynamic modelling using nonlinear state-space models.
[19] ICA group, 2004-2005. Nonlinear ICA and BSS.
You are at: CIS → Research → Algorithms → Nonlinear ICA
Page maintained by webmaster at hut.fi, last updated Friday, 27-Apr-2007 13:51:10 EEST