Mind reading from MEG - PASCAL Challenge
- Jan 23rd, 2012: Erratum for the challenge description added. The description incorrectly claimed the data consisted of envelopes of the signals. The description available in the web before the challenge was correct.
- Jan 5th, 2012: Citation details added.
- July 7th, 2011: Test labels and the challenge proceedings released.
- Jun 15th, 2011: Results added. The challenge proceedings will be added in early July.
- Jun 10th, 2011: Location added.
- Jun 2nd, 2011: Program of the workshop released.
- Mar 11th, 2011: Deadline extended to March 31st.
- Mar 1st, 2011: Detailed submission instructions added
- Feb 8th, 2011: New version of the data released. Please download the data again from the data description page
- Dec 10th, 2010: The data was released and the competition starts
The challenge combines two recent trends in neuroscience: Analysis of naturalistic stimulation and mind reading. The task in the challenge is to decode the stimulus identity based on magnetoencephalography (MEG) recording done during naturalistic stimulation. In more detail, the subject is viewing video stimuli of different kinds (football match, feature film, recording of natural scenery etc), and the goal is to classify unlabeled test examples into these categories based on the MEG signal alone.
The challenge aims at promoting awareness on feasibility of MEG for mind reading tasks, as well as encouraging application of recent advances in machine learning development to this difficult but possible modeling task. Demonstration of mind reading from MEG signals showcases the amount of information about perceptual processes captured by the imaging technology, whereas advanced modeling techniques are needed to reliably extract the information.
In this challenge the stimulus identity is inferred from very limited amount of signal measured during natural stimulation, lacking the usual characteristics of MEG analysis such as controlled experimental condition and averaging over multiple trials. The problem can be approached either as a single-trial MEG challenge, or as a generic classification task with high-dimensional signal representation and potentially large shift between the distributions of the training and test samples that were recorded in different sessions. To encourage participants with limited MEG expertise, the data is provided after standard preprocessing.
The data consists of MEG recordings of a single subject, made during two separate measurement sessions (consecutive days). In each session the subject was watching visual stimuli consisting of five different movie categories. The stimuli were presented without audio.
The data contains measurements of 204 planar gradiometer channels at 200Hz rate, segmented into samples of one second length. The samples are given in random order, to enforce prediction based on the 1s window alone. The data is provided after standard preprocessing (removal of external inference, motion correction, low-pass filtering), and the raw signal measurements are complemented with outputs of five bandpass filters.
See full description of the data, including links to the actual data files. (Update, July 11): The same page also contains a link to the test labels released after the competition ended.
The data can be freely used for academic purposes after the challenge, given proper citation. See the Presentation section below for details.
Design and implement a classifier that takes as an input the MEG signals of the test samples (one second of time) and produces as an output the predicted class label (the type of the video stimulus).
The test examples are all extracted from the measurements made during the second day, whereas the labeled training examples are extracted from the recordings of the first day. Consequently, one should expect that the data distribution may be different for the two sets. To help understand the nature of the difference, we also provide labels for a few examples extracted from the second day.
The challenge is open for everyone, but members belonging to the labs of the challenge organizers will not be eligible for the prizes.
No pre-registration is needed, but if you want to receive updates on the competition by email then please send a request to firstname.lastname@example.org.
The submission deadline is March 31st, 2011 (20.00 CET)
March 13th. The results
are to be submitted by email to email@example.com. We will
acknowledge received submissions manually, and consequently there
can be a bit of a delay before confirming the submission.
Each submission should include three files:
- Brief description of the approach used for the solution as a pdf file
- A Matlab file containing a 653-element column vector indicating the class labels for the test samples of day 2 (variable name class_test_day2)
- An ascii text file containing the class labels, one per row
The files should be named report_ID.pdf, class_ID.mat and class_ID.txt, where ID is a short identifier string of your choice. In addition, the email accompanying the submission should include full contact information and affiliation of the participant(s). The text version of the labels is asked for backup. It will only be considered in case we cannot open the Matlab file due to possible incompatibilities with different Matlab versions. The text file can be created with the command "dlmwrite('class_ID.txt',class_test_day2)".
The report should follow the LNCS series format used for the main conference (see instructions). The report needs to include full author information and a description of how the problem was solved. The maximum length is four (4) pages, including all figures and references. Note that these reports will not be published, but instead will be used for evaluating the submissions together with the classification accuracy. Revised versions of selected submissions to be included on the web page will be asked separately at a later stage.
A single participant can send several submissions, but there must be a notable difference between the computational approach of each submission (and not just slightly different parameter values etc). The decision on whether the differences are sufficient will be made by the organizing committee.
Submissions can be revised once before the deadline if needed. Just send another email and clearly indicate that it will replace the earlier submission.
The proceedings of the challenge summarize the output of the challenge. See also the erratum correcting one mistake in the data description. The contents of the proceedings, including links to individual articles are given below:
- A. Klami, P. Ramkumar, S. Virtanen, L. Parkkonen, R. Hari, S. Kaski: ICANN/PASCAL2 Challenge: MEG Mind Reading -- Overview and Results
- H. Huttunen, J-P. Kauppi, J. Tohka: Regularized logistic regression for mind reading with parallel validation
- R. Santana, C. Bielza, P. Larranaga: An ensemble of classifiers approach with multiple sources of information
- P. Jylänki, J. Riihimäki, A. Vehtari: Multi-class Gaussian process classification of single-trial MEG based on frequency specific latent features extracted with linear binary classifiers
The first article has a more comprehensive description of the challenge data, including information needed for further work using the now publicly available data. The remaining three articles describe the methods of the best three participants. Citation details for all four articles, as well as the whole proceedings, are given in bibtex format. When re-using the data of the challenge in future publications or commenting the challenge results you should directly site the overview article describing the details, not the whole proceedings.
The challenge workshop is organized on June 14th, 2011. The full program for the workshop is:
- 13:00-13:15: Welcome (Organizers)
- 13:15-13:45: Brain decoding and the challenge data (P. Ramkumar)
- 13:45-14:15: Modeling challenges of brain decoding (A. Klami)
- 14:15-14:45: Results of the challenge (S. Virtanen)
- 14:45-15:00: Coffee break
- 15:00-16:00: Presentations of the three best teams:
- Jukka-Pekka Kauppi : Regularized logistic regression for mind reading with parallel validation
- Roberto Santana: An ensemble of classifiers approach with multiple sources of information
- Pasi Jylänki: Multi-class Gaussian process classification of single-trial MEG based on frequency specific latent features extracted with linear binary classifiers
- 16:00-17:00: Discussion
The challenge received 10 submissions and the teams were evaluated with the accuracy of predicting the categories of left-out stimuli. Below is a table summarizing the results. More comprehensive analysis of the submissions and results can be found in the challenge proceedings.
|1.||Huttunen et al.||Tampere University of Technology||68.0|
|2.||Santana et al.||Universidad Politecnica de Madrid||63.2|
|3.||Jylänki et al.||Aalto University||62.8|
|4.||Tu & Sun (1)||East China Normal University||62.2|
|5.||Lievonen & Hyötyniemi||Helsinki Institute for Information Technology||56.5|
|6.||Tu & Sun (2)||East China Normal University||54.2|
|7.||Olivetti & Melchiori||University of Trento||53.9|
|8.||Van Gerven & Farquhar||Radboud University Nijmegen||47.2|
|9.||Grozea||Fraunhofer Institute FIRST||44.3|
|10.||Nicolaou||University of Cyprus||24.2|
A. Klami1, P. Ramkumar2, S. Virtanen1, L. Parkkonen2, R. Hari2, and S. Kaski1
Aalto University School of Science and Technology
1Department of Information and Computer Science, Helsinki Institute for Information Technology HIIT
2Brain Research Unit, Low Temperature Laboratory
The challenge is sponsored by the PASCAL network of excellence.