On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei <s.bakhsh...@yahoo.com> wrote: > > Hello, > > I have some question about mteval-v11b.pl > > 1) It can not use multi-reference with mteval what is a equivalent tool for > this aim? > 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to > increase while adding more reference sets !! How it is may? > 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they > do not always agree ! sometimes mteval and sometimes the other gives better > scores. Is there any problem? > 4) and at the end, isn't there any better tool with the property of > multi-reference?
Hi Somayeh, BLEU has defined treatment for multiple references from the very beginning (see the original Papineni et al 2002 paper for details). Any implementation of BLEU that does not support multiple references should be considered defective. Personally I've always used mteval-v13a from http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem dealing with multiple references at all. All you need to do is to provide the multiple references as multiple doc sections in your reference set: <doc docid="document" sysid="r1"> <seg>...</seg> ... </doc> <doc docid="document" sysid="r2"> ... Disclaimer: The above definitely works for v13a but I'm not specifically familiar with v11b. Cheers, Liu Chang National University of Singapore _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
