Number of the records: 1
A Pilot Eye-Tracking Study of WMT-Style Ranking Evaluation
- 1.0486100 - PSÚ 2018 SI eng C - Conference Paper (international conference)
Bojar, O. - Děchtěrenko, Filip - Zelenina, M.
A Pilot Eye-Tracking Study of WMT-Style Ranking Evaluation.
Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem: Proceedings. Portorož: LREC 2016 Workshop, 2016 - (Rehm, G.), s. 20-26. ISBN N.
[LREC 2016 Workshop: Translation Evaluation: From Fragmented Tools and Data Sets to an Integrated Ecosystem. Portorož (SI), 24.05.2016-24.05.2016]
R&D Projects: GA MŠMT(CZ) LM2010013
Grant - others:AV ČR(CZ) StrategieAV21/14
Program: StrategieAV
Institutional support: RVO:68081740
Keywords : MT Evaluation * WMT * Eye Tracking
OECD category: Cognitive sciences
http://www.cracking-the-language-barrier.eu/wp-content/uploads/LREC-2016-MT-Eval-Workshop-Proceedings.pdf
The shared translation task of the Workshop of Statistical Machine Translation (WMT) is one of the key annual events of the field. Participating machine translation systems in WMT translation task are manually evaluated by relatively ranking five candidate translations of a given sentence. This style of evaluation has been used since 2007 with some discussion on interpreting the collected judgements but virtually no insight into what the annotators are actually doing. The scoring task is relatively cognitively demanding and many scoring strategies are possible, influencing the reliability of the final judgements. In this paper, we describe our first steps towards explaining the scoring task: we run the scoring under an eye-tracker and monitor what the annotators do. At the current stage, our results are more of a proof-of-concept, testing the feasibility of eye tracking for the analysis of such a complex MT evaluation setup.
Permanent Link: http://hdl.handle.net/11104/0281491
File Download Size Commentary Version Access 0486100 C Děchtěrenko LREC-2016-MT-Eval-Workshop-Proceedings.pdf 0 230.9 KB Author´s preprint require
Number of the records: 1