Updates

• Jan 23rd, 2012: Erratum for the challenge description added. The description incorrectly claimed the data consisted of envelopes of the signals. The description available in the web before the challenge was correct.
• Jan 5th, 2012: Citation details added.
• July 7th, 2011: Test labels and the challenge proceedings released.
• Jun 15th, 2011: Results added. The challenge proceedings will be added in early July.
• Jun 10th, 2011: Location added.
• Jun 2nd, 2011: Program of the workshop released.
• Mar 11th, 2011: Deadline extended to March 31st.
• Mar 1st, 2011: Detailed submission instructions added
• Feb 8th, 2011: New version of the data released. Please download the data again from the data description page
• Dec 10th, 2010: The data was released and the competition starts

Overview

The challenge combines two recent trends in neuroscience: Analysis of naturalistic stimulation and mind reading. The task in the challenge is to decode the stimulus identity based on magnetoencephalography (MEG) recording done during naturalistic stimulation. In more detail, the subject is viewing video stimuli of different kinds (football match, feature film, recording of natural scenery etc), and the goal is to classify unlabeled test examples into these categories based on the MEG signal alone.

The challenge aims at promoting awareness on feasibility of MEG for mind reading tasks, as well as encouraging application of recent advances in machine learning development to this difficult but possible modeling task. Demonstration of mind reading from MEG signals showcases the amount of information about perceptual processes captured by the imaging technology, whereas advanced modeling techniques are needed to reliably extract the information.

In this challenge the stimulus identity is inferred from very limited amount of signal measured during natural stimulation, lacking the usual characteristics of MEG analysis such as controlled experimental condition and averaging over multiple trials. The problem can be approached either as a single-trial MEG challenge, or as a generic classification task with high-dimensional signal representation and potentially large shift between the distributions of the training and test samples that were recorded in different sessions. To encourage participants with limited MEG expertise, the data is provided after standard preprocessing.

Data

The data consists of MEG recordings of a single subject, made during two separate measurement sessions (consecutive days). In each session the subject was watching visual stimuli consisting of five different movie categories. The stimuli were presented without audio.

The data contains measurements of 204 planar gradiometer channels at 200Hz rate, segmented into samples of one second length. The samples are given in random order, to enforce prediction based on the 1s window alone. The data is provided after standard preprocessing (removal of external inference, motion correction, low-pass filtering), and the raw signal measurements are complemented with outputs of five bandpass filters.

See full description of the data, including links to the actual data files. (Update, July 11): The same page also contains a link to the test labels released after the competition ended.

The data can be freely used for academic purposes after the challenge, given proper citation. See the Presentation section below for details.

Task

Design and implement a classifier that takes as an input the MEG signals of the test samples (one second of time) and produces as an output the predicted class label (the type of the video stimulus).

The test examples are all extracted from the measurements made during the second day, whereas the labeled training examples are extracted from the recordings of the first day. Consequently, one should expect that the data distribution may be different for the two sets. To help understand the nature of the difference, we also provide labels for a few examples extracted from the second day.

Eligibility

The challenge is open for everyone, but members belonging to the labs of the challenge organizers will not be eligible for the prizes.

No pre-registration is needed, but if you want to receive updates on the competition by email then please send a request to icann2011.meg@cis.hut.fi.

Submission

The submission deadline is March 31st, 2011 (20.00 CET) March 13th. The results are to be submitted by email to icann2011.meg@cis.hut.fi. We will acknowledge received submissions manually, and consequently there can be a bit of a delay before confirming the submission.

Each submission should include three files:

1. Brief description of the approach used for the solution as a pdf file
2. A Matlab file containing a 653-element column vector indicating the class labels for the test samples of day 2 (variable name class_test_day2)
3. An ascii text file containing the class labels, one per row

The files should be named report_ID.pdf, class_ID.mat and class_ID.txt, where ID is a short identifier string of your choice. In addition, the email accompanying the submission should include full contact information and affiliation of the participant(s). The text version of the labels is asked for backup. It will only be considered in case we cannot open the Matlab file due to possible incompatibilities with different Matlab versions. The text file can be created with the command "dlmwrite('class_ID.txt',class_test_day2)".

The report should follow the LNCS series format used for the main conference (see instructions). The report needs to include full author information and a description of how the problem was solved. The maximum length is four (4) pages, including all figures and references. Note that these reports will not be published, but instead will be used for evaluating the submissions together with the classification accuracy. Revised versions of selected submissions to be included on the web page will be asked separately at a later stage.

A single participant can send several submissions, but there must be a notable difference between the computational approach of each submission (and not just slightly different parameter values etc). The decision on whether the differences are sufficient will be made by the organizing committee.

Submissions can be revised once before the deadline if needed. Just send another email and clearly indicate that it will replace the earlier submission.

Presentation

Proceedings

The proceedings of the challenge summarize the output of the challenge. See also the erratum correcting one mistake in the data description. The contents of the proceedings, including links to individual articles are given below:

• A. Klami, P. Ramkumar, S. Virtanen, L. Parkkonen, R. Hari, S. Kaski: ICANN/PASCAL2 Challenge: MEG Mind Reading -- Overview and Results
• H. Huttunen, J-P. Kauppi, J. Tohka: Regularized logistic regression for mind reading with parallel validation
• R. Santana, C. Bielza, P. Larranaga: An ensemble of classifiers approach with multiple sources of information
• P. Jylänki, J. Riihimäki, A. Vehtari: Multi-class Gaussian process classification of single-trial MEG based on frequency specific latent features extracted with linear binary classifiers

The first article has a more comprehensive description of the challenge data, including information needed for further work using the now publicly available data. The remaining three articles describe the methods of the best three participants. Citation details for all four articles, as well as the whole proceedings, are given in bibtex format. When re-using the data of the challenge in future publications or commenting the challenge results you should directly site the overview article describing the details, not the whole proceedings.

Program

The challenge workshop is organized on June 14th, 2011. The full program for the workshop is:

• 13:00-13:15: Welcome (Organizers)
• 13:15-13:45: Brain decoding and the challenge data (P. Ramkumar)
• 13:45-14:15: Modeling challenges of brain decoding (A. Klami)
• 14:15-14:45: Results of the challenge (S. Virtanen)
• 14:45-15:00: Coffee break
• 15:00-16:00: Presentations of the three best teams:
• Jukka-Pekka Kauppi : Regularized logistic regression for mind reading with parallel validation
• Roberto Santana: An ensemble of classifiers approach with multiple sources of information
• Pasi Jylänki: Multi-class Gaussian process classification of single-trial MEG based on frequency specific latent features extracted with linear binary classifiers
• 16:00-17:00: Discussion

The workshop is held in the Computer Science building (hall T3 in the second floor) of Aalto University. See the workshop page of ICANN and map of the conference venue for more details.

Results

The challenge received 10 submissions and the teams were evaluated with the accuracy of predicting the categories of left-out stimuli. Below is a table summarizing the results. More comprehensive analysis of the submissions and results can be found in the challenge proceedings.

Rank	Team	Affiliation	Accuracy (%)
1.	Huttunen et al.	Tampere University of Technology	68.0
2.	Santana et al.	Universidad Politecnica de Madrid	63.2
3.	Jylänki et al.	Aalto University	62.8
4.	Tu & Sun (1)	East China Normal University	62.2
5.	Lievonen & Hyötyniemi	Helsinki Institute for Information Technology	56.5
6.	Tu & Sun (2)	East China Normal University	54.2
7.	Olivetti & Melchiori	University of Trento	53.9
8.	Van Gerven & Farquhar	Radboud University Nijmegen	47.2
9.	Grozea	Fraunhofer Institute FIRST	44.3
10.	Nicolaou	University of Cyprus	24.2

Organizers

A. Klami¹, P. Ramkumar², S. Virtanen¹, L. Parkkonen², R. Hari², and S. Kaski¹

Aalto University School of Science and Technology
¹Department of Information and Computer Science, Helsinki Institute for Information Technology HIIT
²Brain Research Unit, Low Temperature Laboratory

The challenge is sponsored by the PASCAL network of excellence.