CLEF-ER 2013: Named Entity Recognition

Produce annotations in a multilingual document where there are explicit references to entities, specified through a Concept Unique Identifier.

CLEF-ERCLEF-ER 2013: Named Entity Recognition in parallel multilingual biomedical corpora (aiming for terminology translation)

Multilingual identification of mentions of named entities (attribution of CUIs) in corpora, where each corpus is either in English, French, German, Spanish, and Dutch. Currently patent texts, titles of Medline abstracts and EMEA documents serve as parallel corpora. The mention annotations will be evaluated as part of the challenge and, in addition, the mention annotations will be analysed to contribute to an augmented biomedical terminological resource.

Participants produce mention annotations in a multilingual document where the annotations make explicit reference to entities, which are specified through a Concept Unique Identifier (CUI, from the UMLS). Challenge participants should at least contribute one non-English annotated corpus and have to register at the submission site .

A number of resources are available too:
(1) a terminological resource (TR) containing English and non-English concepts in combination with their CUIs,
(2) a selection of corpora in English, where the entity mentions have been annotated automatically with their CUIs, and
(3) a selection of corpora in different languages other than English (i.e., de, fr, sp, nl) that have to be annotated with entity mentions and their CUIs.

Note: Each non-English corpus has a parallel English corpus. Not all languages (de, fr, sp, nl) are available for each corpus. Parallel corpora offer the opportunity to use machine-translation or bi-lingual term identification methods.

The results will be presented on the CLEF-ER workshop, which is part of the CLEF 2013 Conference, 23-26 September in Valencia, Spain. The challenge is sponsored by the EU project "Mantra". For more details on the task please visit or send an email to


  • Feb 5, 2013: corpora available
  • Apr 30, 2013: submission system for runs closes
  • May 31, 2013: preliminary evaluation results released
  • Sep 23, 2013: CLEF 2013 conference starts (26.09.2013: CLEF 2013 conference ends)