Hi, can Apertium somehow be used for this task? Intuitively, the
analysis (part of speech tags) would be useful. Especially if the tags
could be remembered for the previous sentence.
Yours, Per Tunedal
----- Original message ----- From: joerg <tiede...@gmail.com> To:
"moses-support" <moses-supp...@mit.edu> Subject: [Moses-support] DiscoMT
2015 Shared Task on Pronoun Translation (at EMNLP 2015) Date: Fri, 27
Feb 2015 13:43:50 +0100
===============================================
DiscoMT 2015 Shared Task on Pronoun Translation
===============================================
Website: https://www.idiap.ch/workshop/DiscoMT/shared-task In connection
with EMNLP 2015 (http://emnlp2015.emnlp.org)
We are happy to announce a new exciting task for people interested in
(discourse-aware) machine translation, anaphora resolution and machine
learning in general. The EMNLP 2015 Workshop on Discourse in Machine
Translation features two shared tasks:
Task 1: Pronoun-Focused Machine Translation Task 2: Cross-Lingual
Pronoun Prediction
Task 1 requires machine translation (from English to French) and focuses
on the evaluation of translated pronouns. We provide training data and a
baseline SMT model to get started.
Task 2 is a straightforward classification task in which one has to
predict the correct translation of a given pronoun in English (it or
they) into French (ce, elle, elles, il, ils, ça, cela, on, OTHER). We
provide training and development data and a simple baseline system using
an N-gram language model.
More details of the two tasks are attached below and can be found at our
website: https://www.idiap.ch/workshop/DiscoMT/shared-task
Important Dates:
4 May, 2015 Release of the MT test set (task 1) 10 May, 2015 Submission
of translations (task 1) 11 May, 2015 Release of the classification
test set (task 2) 18 May, 2015 Submissions of classification results
(task 2) 28 May, 2015 System paper submission deadline Sep., 2015
Workshop in Lisbon
Mailing list: https://groups.google.com/d/forum/discomt2015
Downloads:
https://www.dropbox.com/sh/c8qnpag5z29jyh6/AAAAqk1TE9-UvcgEnfccdRwxa?dl=0
Download alternative 1: http://opus.lingfil.uu.se/DiscoMT2015/ Download
alternative 2: http://stp.lingfil.uu.se/~joerg/DiscoMT2015/
-------------------------------------------------------------------------
Acknowledgements: Funding for the manual evaluation of the
pronoun-focused translation task is generously provided by the European
Association for Machine Translation (EAMT)
-------------------------------------------------------------------------
==========================
Detailed Task Description:
==========================
* Overview
The DiscoMT 2015 shared task will consist of two subtasks, relevant to
both the MT and discourse communities: pronoun-focused translation, a
practical MT task, and cross-lingual pronoun prediction, a
classification task that requires no specific MT expertise and is
interesting as a machine learning task in its own right. For groups
wishing to participate in both tasks, one possibility is to convert a
system for the classification task into an MT feature model using
existing software such as the Docent decoder (Hardmeier et al., ACL
2013). Both tasks use the English–French language pair, which has a
sufficiently high baseline performance to produce basically intelligible
output, as well as interesting differences in their pronoun systems.
* Task 1: Pronoun-Focused Translation Task
In the pronoun-focused translation task, you are given a collection of
English input documents, which you are asked to translate into French.
This task is the same as for other MT shared tasks such as that of WMT.
The difference is in the way the translations are evaluated. Instead of
checking the overall translation quality, we specifically look at how
the English subject pronouns it and they were translated. The principal
evaluation will be carried out manually and will focus specifically on
the correctness of pronoun translation. Thanks to a grant from the EAMT,
the manual evaluation will be run by the organisers and participants
don't have to contribute evaluations. Automatic reference-based metrics
are available for development purposes.
The texts in the test corpus will consist of transcripts of TED talks.
The training data contains an in-domain corpus of TED talks as well as
some additional data from Europarl and news texts. To make the
participating systems as comparable as possible, we ask you to constrain
the training data of your system to the resources listed below as far as
you can, but this is not a strict requirement and we do accept
submissions using additional resources. If your system uses any
resources other than those of the official data release, please be
specific about what was included in the system description paper. For
the same reason, we also suggest that you use the tokeniser provided by
us unless you have a good reason to do otherwise.
The test set will be supplied in the XML source format of the 2009 NIST
MT evaluation, which is described on the last page of this document. See
the development set included in the data release for an example. Your
translation should be submitted in the XML translation format of the
2009 NIST MT evaluation. We also need you to submit, in a separate file,
word alignments linking occurrences of the pronouns it and they
(case-insensitive) to the corresponding words generated by your MT
system. The format of the word alignments should be the same as that of
the alignments included in the cross-lingual pronoun prediction data
(see below). Word alignments can be obtained, for instance, by running
the Moses SMT decoder with the -print-alignment-info option or by
parsing the segment-level comments added to the output by the Docent
decoder. You may submit alignments for the complete sentence if it's
easier for you, but only links for it and they will be used. If your MT
system cannot output word alignments, please contact the shared task
organisers to discuss how to proceed. We'll try to find a solution. More
details on how to submit will be added to this page later.
The test set will be released on 4 May 2015, and your translations are
due on 10 May 2015. Note that we will ensure that each document in the
test set contains an adequate number of challenging pronouns, so the
corpus-level distribution of the pronouns in the test set may differ
from that of the training corpus. However, each document will be a
complete TED talk with a naturally occurring ensemble of pronouns.
* Task 2: Cross-Lingual Pronoun Prediction
In the cross-lingual pronoun prediction task, you are given an English
document with a human-generated French translation and a set of word
alignments between the two languages. In the French translation, the
words aligned to the English third-person subject pronouns it and they
are substituted by placeholders. Your task is to predict, for each
placeholder, the word that should go there from a small, closed set of
classes, using any information you can extract from the documents. The
following classes exist:
ce The French pronoun ce (sometimes with elided vowel as c') as in the
expression c'est 'it is' elle feminine singular subject pronoun elles
feminine plural subject pronoun il masculine singular subject pronoun
ils masculine plural subject pronoun ça demonstrative pronoun (including
the misspelling ca and the rare elided form ç') cela demonstrative
pronoun on indefinite pronoun OTHER some other word, or nothing at all,
should be inserted
This task will be evaluated automatically by matching the predictions
against the words found in the reference translation by computing the
overall accuracy and precision, recall and F-score for each class. The
primary score for the evaluation is the macro-averaged F-score over all
classes. Compared to accuracy, the macro-averaged F-score favours
systems that consistently perform well on all classes and penalises
systems that maximise the performance on frequent classes while
sacrificing infrequent ones.
The data supplied for the classification task consists of parallel
English-French text with word alignments. In the French text, a subset
of the words aligned to English occurrences of it and they have been
replaced by placeholders of the form REPLACE_xx, where xx is the index
of the English word the placeholder is aligned to. Your task is to
predict one of the classes listed above for each occurrence of a
placeholder.
The training and development data is supplied in a file format with five
tab-separated columns:
1. the class label
2. the word actually removed from the text (may be different from the
class label for class OTHER and in some edge cases)
3. the English source segment
4. the French target segment with pronoun placeholders
5. the word alignment (a space-separated list of alignments of the form
SRC-TGT, where SRC and TGT are zero-based word indices in the source
and target segment, respectively)
A single segment may contain more than one placeholder. In that case,
columns 1 and 2 contain multiple space-separated entries in the order of
placeholder occurrence. A document segmentation of the data is provided
in separate files for each corpus. These files contain one line per
segment, but the precise format varies depending on the type of document
markup available for the different corpora. In the development and test
data, the files have a single column containing the ID of the document
the segment is part of.
Here's an example line from one of the training data files:
elles Elles They arrive first . REPLACE_0 arrivent en premier . 0-0
1-1 2-3 3-4
The test set will be supplied in the same format, but with columns 1 and
2 (elles and Elles) empty, so each line starts with two tab characters.
Your submission should have the same format as column 1 above, so a
correct solution would contain the class label elles in this case. Each
line should contain as many space-separated class labels as there are
REPLACE tags in the corresponding segment. For each segment not
containing any REPLACE tags, an empty line should be emitted. Additional
tab-separated columns may be present in the submission, but will be
ignored. Note in particular that you are not required to predict the
second column. The submitted files should be encoded in UTF-8 (like the
data we provide).
The test set will be the same as for the pronoun-focused translation
task. The complete test data for the classification task, including
reference translations and word alignments, will be released on 11 May
2015, after the completion of the translation task. Your submission is
due on 18 May 2015. Details on how to submit will be added to our
website later.
Note: If you create a classifier for this task, but haven't got an MT
system of your own, you might consider using your classifier as a
feature function in the document-level SMT decoder Docent to
create a submission for the pronoun translation task.
* Discussion Group
If you are interested in participating in the shared task, we recommend
that you sign up to our discussion group to make sure you don't miss any
important information. Feel free to ask any questions you may have about
the shared task!
https://groups.google.com/d/forum/discomt2015
* Training Data and Tools
All training and development data for both subtasks can be downloaded
from the following location:
https://www.dropbox.com/sh/c8qnpag5z29jyh6/AAAAqk1TE9-UvcgEnfccdRwxa?dl=0
Download alternative 1: http://opus.lingfil.uu.se/DiscoMT2015/ Download
alternative 2: http://stp.lingfil.uu.se/~joerg/DiscoMT2015/
The dropbox folder contains many files, see the list below. To create a
system for the pronoun classification task, you should start with the
classification training data. For the pronoun-focused translation task,
we provide both the original training data, preprocessed data sets
including full word alignments and a complete pre-trained phrase-based
SMT system. To minimise preprocessing differences among the submitted
system we suggest (but do not require) that you start from the most
processed version of the data that is usable for the type of system that
you plan to build.
Look at the README file for more information about the individual files
we provide: http://stp.lingfil.uu.se/~joerg/DiscoMT2015/README
* Classification Baseline
We have a baseline model for the classification task that looks only at
the language model scores (using KenLM, and the language model that is
used needs to be in KenLM's binary format (which is the case for the
"corpus.5.trie.kenlm" included in the "baseline-all" tarball).
Results with default options on TEDdev (same data as tst2010):
ce : P = 110/ 129 = 85.27% R = 110/ 148 = 74.32% F1 = 79.42% cela : P =
4/ 15 = 26.67% R = 4/ 10 = 40.00% F1 = 32.00% elle : P = 6/ 13 = 46.15%
R = 6/ 30 = 20.00% F1 = 27.91% elles : P = 4/ 12 = 33.33% R = 4/ 16 =
25.00% F1 = 28.57% il : P = 35/ 137 = 25.55% R = 35/ 55 = 63.64% F1 =
36.46% ils : P = 86/ 94 = 91.49% R = 86/ 139 = 61.87% F1 = 73.82% on : P
= 3/ 10 = 30.00% R = 3/ 10 = 30.00% F1 = 30.00% ça : P = 16/ 22 = 72.73%
R = 16/ 61 = 26.23% F1 = 38.55% OTHER : P = 225/ 315 = 71.43% R = 225/
278 = 80.94% F1 = 75.89%
or a macro-averaged fine-grained F1 of 46.96%
Results with "--null-penalty -2.0"
ce : P = 121/ 145 = 83.45% R = 121/ 148 = 81.76% F1 = 82.59% cela : P =
4/ 21 = 19.05% R = 4/ 10 = 40.00% F1 = 25.81% elle : P = 7/ 15 = 46.67%
R = 7/ 30 = 23.33% F1 = 31.11% elles : P = 5/ 14 = 35.71% R = 5/ 16 =
31.25% F1 = 33.33% il : P = 36/ 143 = 25.17% R = 36/ 55 = 65.45% F1 =
36.36% ils : P = 99/ 109 = 90.83% R = 99/ 139 = 71.22% F1 = 79.84% on :
P = 3/ 13 = 23.08% R = 3/ 10 = 30.00% F1 = 26.09% ça : P = 19/ 32 =
59.38% R = 19/ 61 = 31.15% F1 = 40.86% OTHER : P = 211/ 255 = 82.75% R =
211/ 278 = 75.90% F1 = 79.17%
or a fine-grained F1 score of 48.35%
_________________________________________________
Moses-support mailing list moses-supp...@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------------------------------------------------------
Dive into the World of Parallel Programming The Go Parallel Website, sponsored
by Intel and developed in partnership with Slashdot Media, is your hub for all
things parallel software development, from weekly thought leadership blogs to
news, videos, case studies, tutorials and more. Take a look and join the
conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Apertium-stuff mailing list
Apertium-stuff@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/apertium-stuff
