T-61.6040 Special Course in Computer and Information Science IV L:

Variable Selection for Regression

Nowadays, many machine learning problems involve the use of a large number of features. This might be the case, for example, in DNA and biomedical data analysis, in image processing, financial data mining, chemometrics, etc. In other cases, the number of features may be smaller, but of the same order of magnitude as the number of samples. In both cases, regression tasks are faced to the curse of dimensionality: overfitting easily appears, and in some cases the regression problem can become ill-posed (or not identifiable). The challenge is then to reduce the number of features, in order to improve the regression efficiency. Interpretability is often a major concern too, as a large number of features usually prevents any understanding of the underlying relationship.

Feature selection and dimension reduction includes two different ways of reducing the number of inputs of the regression model. First, inputs are selected among the original features; this is usually referred to as feature selection or input selection. Second, inputs can be built from the original features, by combining them in a linear or nonlinear way; this leads to dimension reduction. In this course, both feature selection and dimension reduction methods are covered.

Each student gives a presentation in the seminar. In addition, requirements include a project work and active participation in the lectures (one absence is allowed).

[1] J. Hao. Input selection using mutual information - applications to time series prediction. Master's thesis, Helsinki University of Technology, 2005.
[2] R. Kohavi and G. H. John. Wrappers for feature subset selection. Artifical Intelligence, 1997.
[3] I. Guyon, S. Gunn, M. Nikravesh and L. Zadeh. Feature Extraction, chapter Embedded Methods. Springer.
[4] A. Lendasse, G. Simon, v. Wertz and M. Verleysen. Fast bootstrap methodology for model selection. Neurocomputing, 2005.
[5] A. Lendasse. Testing neural models: how to use re-sampling techniques?
[6] B. Efron, T. Hastie, L. Johnstone and R. Tibshirani. Least angle regression. Annals of Statistics, 2004.
[7] A. Sorjamaa, N. Reyhani and A. Lendasse. Input and structure selection for k-nn approximator.
[8] A. Sorjamaa, A. Lendasse and M. Verleysen. Pruned lazy learning models for time series prediction.
[9] Chris Atkeson, A. Moore and S. Schaal. Locally weighted learning. AI Review, 11:11-73, April 1997.
[10] P. Leray and P. Gallinari. Feature selection with neural networks. Behaviormetrica, 1998.
[11] T. van Gestel, J. A. K. Suykens, B. de Moor and J. Vandewalle. Automatic relevance determination for least squares support vector machine classifiers. In M. Verleysen, editor, European Symposium on Artificial Neural Networks, 13-18, 2001.
[12] H. Vafaie. Robust feature selection algorithms. In Proc. 5th Intl. Conf. on Tools with Artificial Intelligence, 1993.
[13] C. R. Houck, J. A. Joines and M. G. Kay. A genetic algorithm for function optimization: a matlab implementation. Technical report, North Carolina State University, 1996.
[14] L. J. Herrera. Effective input variable selection for function approximation. In Lecture Notes on Computer Science.
[15] S. Perkins and J. Theiler. Online feature selection using grafting.
[16] S. Perkins, K. Lacker and J. Theiler. Grafting: Fast, incremental feature selection by gradient descent in function space. Journal of Machine Learning Research, 2003.
[17] Amir Navot, Lavi Shpigelman, Naftali Tishby and Eilon Vaadia. Nearest neighbor based feature selection for regression and its application to neural activity. NIPS 2005.
[18] S. Wold, L. Eriksson, J. Trygg and N. Kettaneh. The PLS method - partial least squares projection to latent structures - and its applications in industrial RDP (research, development and production).
[19] A. Lendasse, F. Corona, J. Hao, N. Reyhani and M. Verleysen. Determination of the Mahalanobis matrix using nonparametric noise estimations. ESANN 2006.