Active Learning Challenge

clopinet.com/al

A new challenge on active learning just started. This challenge addresses machine learning problems in which labeling data is expensive, but large amounts of unlabeled data are available at low cost. For instance, images are available in abundance on the Internet, but few come with annotations to classify them according to their contents (car, animal, building, landscape, etc.). Human operators may be hired to provide annotations (labels), but usually for a fee.

Such problems might be tackled from different angles: learning from unlabeled data or active learning. In the former case, the algorithms must satisfy themselves with the limited amount of labeled data and capitalize on the unlabeled data with semi-supervised learning methods. In the latter case, the algorithms may place a limited number of queries to get labels. The goal in that case is to optimize the queries to label data and the problem is referred to as active learning.

In this challenge, the participants are alloted an intial budget of virtual cash and are allowed to purchase labels until they run out of cash. The goal is to reach as fact as possible the best prediction performance. Six datasets from various domains (chemo-informatics, handwriting recognition, text processing, ecology, marketing, and embryology) are made available for algorithm development during 2 months, with immediate on-line performance feed-back (you may also download the labels if you want). The final evaluation will be performed on 6 fresh datasets from the same domains.

NO PRIOR EXPERIENCE OF ACTIVE LEARNING NEEDED!
We give sample code, tutorials, and you have lots of time to develop your techniques.

Two workshops in Italy and Spain, two publications opportunities in IEEE proceedings and JMLR W&CP.

Schedule:

• Dec. 1, 2009 Start of the development period. Development datasets made available.
• Jan. 31, 2010 WCCI 2010 papers due. End of the development period.
• Feb. 1, 2010 Begin final testing. Final datasets made available.
• Feb. 28, 2010 End of the challenge. Submissions closed.
• Feb. 15, 2010 All teams must turn in fact sheets (used as abstracts for AISTATS workshop). Reviewers and the participants are given access to the provisional ranking and the fact sheets. Start of post-challenge verifications.
• Mar. 15, 2010 End of the post challenge verifications. Release of the official ranking. Notification of paper acceptance. May 2, 2010 Camera ready copies due (for IEEE proceedings and JMLR W&CP papers).
• May 16, 2010 Workshop at AISTATS, Sardinia Italy.
• July 19-23, 2010 Workshop at WCCI 2010, Barcelona, Spain.