note also that NIST changed to IBM BLEU recently which has a different treatment of multiple references.
(mteval 13 uses IBM BLEU if i recall) generally the BLEU scores will be a little lower than before, but MERT performance should be more robust Miles On 17 October 2010 09:57, liu chang <liuchangj...@gmail.com> wrote: > On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei > <s.bakhsh...@yahoo.com> wrote: >> >> Hello, >> >> I have some question about mteval-v11b.pl >> >> 1) It can not use multi-reference with mteval what is a equivalent tool for >> this aim? >> 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >> increase while adding more reference sets !! How it is may? >> 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they >> do not always agree ! sometimes mteval and sometimes the other gives better >> scores. Is there any problem? >> 4) and at the end, isn't there any better tool with the property of >> multi-reference? > > Hi Somayeh, > > BLEU has defined treatment for multiple references from the very > beginning (see the original Papineni et al 2002 paper for details). > Any implementation of BLEU that does not support multiple references > should be considered defective. > > Personally I've always used mteval-v13a from > http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem > dealing with multiple references at all. All you need to do is to > provide the multiple references as multiple doc sections in your > reference set: > > <doc docid="document" sysid="r1"> > <seg>...</seg> > ... > </doc> > <doc docid="document" sysid="r2"> > ... > > Disclaimer: The above definitely works for v13a but I'm not > specifically familiar with v11b. > > Cheers, > Liu Chang > National University of Singapore > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
