Dear all, all participants of the metrics task are expected to spend 8 hours of annotations (about 16 HITs) for each submission they make.
In the end, we *will* use Appraise and we are searching for people able to evaluate translations into Chinese, Finnish, German, Latvian, and Turkish. Please consider asking your colleagues if you don't speak the languages yourself. To join the manual evaluation, please sign up to the following Google Sheet: http://tinyurl.com/wmt17-appraise-signup (You can do this now, it is not related to your metrics paper submission.) Please create one line per research group and provide: - the number of submissions your group is making - a contact person for your group - the number of invite tokens, i.e. a rough upper bound on the number of people who will work on the evaluation The contact person will receive the invite tokens from Christian Federmann and should distribute them to the evaluators. Please note that invite tokens are one-time use only. The total number of HITs annotated by each group will be listed most likely at this URL: http://wmt17.appraise.cf/group-status/ Thanks, Ondrej. ----- Original Message ----- > From: "Ondrej Bojar" <bo...@ufal.mff.cuni.cz> > To: "wmt-tasks" <wmt-ta...@googlegroups.com>, "moses-support" > <moses-support@mit.edu>, "nmt-developers" > <nmt-develop...@googlegroups.com> > Cc: "Yvette Graham" <graham.yve...@gmail.com>, "Amir Kamran" > <kam...@ufal.mff.cuni.cz> > Sent: Thursday, 15 June, 2017 10:31:10 > Subject: Metrics Task Last Call and a Small Deadline Extension > Dear all, > > This is the last call for participation in the WMT17 Metrics Task and we are > happy to offer you at least a small deadline extension: > > * Papers: Saturday June 17 (anywhere on Earth) > * Metrics themselves: Wednesday June 21 AoE, yes, later; you can add improved > results to the camera-ready version > > Looking forward to your submissions. > > Please remember that in order to be included in the metrics evaluation, you > should complete the requested number of human evaluation hits (depending on > numbers of submissions your team will make). > > This year the metrics evaluation will run via the DFKI-DA system (as opposed > to > Appraise). Instructions will be sent tomorrow. Languages that will be > evaluated > are Chinese, Finnish, German, Latvian, and Turkish, as the other languages for > the metrics task are covered by crowd-sourcing. Please consider also reaching > out to your foreign colleagues if you don't speak these languages. Any > problems, please let us know. > > Deadline for manual evaluation: already Friday June 23. > > Your metrics task organizers, > Ondrej Bojar > Yvette Graham > Amir Kamran > Raphael Rubino > > -- > Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) > http://www.cuni.cz/~obo -- Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) http://www.cuni.cz/~obo _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
