Shared tasks
Due to scientific community's high interest in the SMT field, there are several shared tasks (evaluation campaigns), proposed to the research grouped all over the world to evaluate translation quality of their translation systems on an unseen test set. Evaluation is normally based on the automatic or/and human criteria, robusting evaluation results. A parallel corpus as training data and additional resources are provided. There are several evaluation campaigns, the most widely known being:
- The NIST MT evaluation. Annual evaluation, organized by the National Institute of Standards and Technology. The evaluation is a part of an ongoing series of evaluations of human language translation technology. This evaluation campaign is characterized by large amount of training material, mainly belonging to the news domain.
- IWSLT (The International Workshop on Spoken Language Translation) evaluation. Annual evaluation organized by the Consortium for Speech Translation Advanced Research (C-STAR), focusing on the different aspects of the SMT (IWSLT 2004 evaluation - automatic evaluation metrics for speech-to-speech translation, IWSLT 2005 - automatic translation of the ASR output, IWSLT 2006 - spontaneous speech translation) and mainly supplied with small amount of in-domain training data [iwslt06].
- ACL-WMT evaluation. Evaluation organized in the framework of the annual ACL workshop and facing, among other challenges, the problem of automatic translating towards languages other than English [koehn-monz].