Re: [Moses-support] Moses-support Digest, Vol 48, Issue 28
Hello, Thank s, mteval-v11b.pl is specified in the step-by-step manual on the moses sit ! Isn't it better to update the site?! -- Best Regards, S.Bakhshaei --- On Sun, 10/17/10, moses-support-requ...@mit.edu wrote: From: moses-support-requ...@mit.edu Subject: Moses-support Digest, Vol 48, Issue 28 To: moses-support@mit.edu Date: Sunday, October 17, 2010, 7:40 PM Send Moses-support mailing list submissions to moses-support@mit.edu To subscribe or unsubscribe via the World Wide Web, visit http://mailman.mit.edu/mailman/listinfo/moses-support or, via email, send a message with subject or body 'help' to moses-support-requ...@mit.edu You can reach the person managing the list at moses-support-ow...@mit.edu When replying, please edit your Subject line so it is more specific than "Re: Contents of Moses-support digest..." Today's Topics: 1. mteval-v11b (Somayeh Bakhshaei) 2. Re: mteval-v11b (liu chang) 3. Re: mteval-v11b (Miles Osborne) 4. Re: mteval-v11b (Ondrej Bojar) -- Message: 1 Date: Sun, 17 Oct 2010 00:41:24 -0700 (PDT) From: Somayeh Bakhshaei Subject: [Moses-support] mteval-v11b To: moses-support@mit.edu Message-ID: <695244.86557...@web111717.mail.gq1.yahoo.com> Content-Type: text/plain; charset="us-ascii" Hello, I have some question about mteval-v11b.pl 1) It can not use multi-reference with mteval what is a equivalent tool for this aim? 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to increase while adding more reference sets !! How it is may? 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they do not always agree ! sometimes mteval and sometimes the other gives better scores. Is there any problem? 4) and at the end, isn't there any better tool with the property of multi-reference? -- Best Regards, S.Bakhshaei -- next part -- An HTML attachment was scrubbed... URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20101017/34ad975b/attachment-0001.htm -- Message: 2 Date: Sun, 17 Oct 2010 16:57:55 +0800 From: liu chang Subject: Re: [Moses-support] mteval-v11b To: Somayeh Bakhshaei Cc: moses-support@mit.edu Message-ID: Content-Type: text/plain; charset=ISO-8859-1 On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei wrote: > > Hello, > > I have some question about mteval-v11b.pl > > 1) It can not use multi-reference with mteval what is a equivalent tool for > this aim? > 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to > increase while adding more reference sets !! How it is may? > 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they > do not always agree ! sometimes mteval and sometimes the other gives better > scores. Is there any problem? > 4) and at the end, isn't there any better tool with the property of > multi-reference? Hi Somayeh, BLEU has defined treatment for multiple references from the very beginning (see the original Papineni et al 2002 paper for details). Any implementation of BLEU that does not support multiple references should be considered defective. Personally I've always used mteval-v13a from http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem dealing with multiple references at all. All you need to do is to provide the multiple references as multiple doc sections in your reference set: ... ... ... Disclaimer: The above definitely works for v13a but I'm not specifically familiar with v11b. Cheers, Liu Chang National University of Singapore -- Message: 3 Date: Sun, 17 Oct 2010 10:03:23 +0100 From: Miles Osborne Subject: Re: [Moses-support] mteval-v11b To: liu chang Cc: moses-support@mit.edu, Somayeh Bakhshaei Message-ID: Content-Type: text/plain; charset=ISO-8859-1 note also that NIST changed to IBM BLEU recently which has a different treatment of multiple references. (mteval 13 uses IBM BLEU if i recall) generally the BLEU scores will be a little lower than before, but MERT performance should be more robust Miles On 17 October 2010 09:57, liu chang wrote: > On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei > wrote: >> >> Hello, >> >> I have some question about mteval-v11b.pl >> >> 1) It can not use multi-reference with mteval what is a equivalent tool for >> this aim? >> 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >> increase while adding more reference sets !! How it is may? >> 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they >> do not always agree ! sometimes mteval and sometimes the other gives better >> scores. Is there any
Re: [Moses-support] mteval-v11b
Dear Somayeh, note also that the absolute scores differ heavily based on tokenization (I've seen difference of up to 10 points absolute). mteval-11b does tokenization on its own (possibly tokenizing tokenized input even further), multi-bleu from moses trusts your tokenization. Another difference can come from the definition of "reference length" with multiple translations. Some usethe shortest ref. length, the original paper by Papineni says 'closest' but does not specify *which*! (If the hypothesis is 10 words and two references are 8 and 12 words, which of the two has the closest length?) Implementations differ on this and they even sometimes depend on the *order* of multi references loaded! The main message: never trust the numbers. Compare only BLEU scores you calculated yourself using a fixed tokenization tool and a fixed BLEU implementation. Cheers, O. "Somayeh Bakhshaei" wrote: >Hello, > >I have some question about mteval-v11b.pl > >1) It can not use multi-reference with mteval what is a equivalent tool for >this aim? >2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >increase while adding more reference sets !! How it is may? >3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they do >not always agree ! sometimes mteval and sometimes the other gives better >scores. Is there any problem? >4) and at the end, isn't there any better tool with the property of >multi-reference? > >-- > >Best Regards, > >S.Bakhshaei > > > ___ >Moses-support mailing list >Moses-support@mit.edu >http://mailman.mit.edu/mailman/listinfo/moses-support -- Ondrej Bojar http://www.cuni.cz/~obo ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] mteval-v11b
note also that NIST changed to IBM BLEU recently which has a different treatment of multiple references. (mteval 13 uses IBM BLEU if i recall) generally the BLEU scores will be a little lower than before, but MERT performance should be more robust Miles On 17 October 2010 09:57, liu chang wrote: > On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei > wrote: >> >> Hello, >> >> I have some question about mteval-v11b.pl >> >> 1) It can not use multi-reference with mteval what is a equivalent tool for >> this aim? >> 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to >> increase while adding more reference sets !! How it is may? >> 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they >> do not always agree ! sometimes mteval and sometimes the other gives better >> scores. Is there any problem? >> 4) and at the end, isn't there any better tool with the property of >> multi-reference? > > Hi Somayeh, > > BLEU has defined treatment for multiple references from the very > beginning (see the original Papineni et al 2002 paper for details). > Any implementation of BLEU that does not support multiple references > should be considered defective. > > Personally I've always used mteval-v13a from > http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem > dealing with multiple references at all. All you need to do is to > provide the multiple references as multiple doc sections in your > reference set: > > > ... > ... > > > ... > > Disclaimer: The above definitely works for v13a but I'm not > specifically familiar with v11b. > > Cheers, > Liu Chang > National University of Singapore > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] mteval-v11b
On Sun, Oct 17, 2010 at 3:41 PM, Somayeh Bakhshaei wrote: > > Hello, > > I have some question about mteval-v11b.pl > > 1) It can not use multi-reference with mteval what is a equivalent tool for > this aim? > 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to > increase while adding more reference sets !! How it is may? > 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they > do not always agree ! sometimes mteval and sometimes the other gives better > scores. Is there any problem? > 4) and at the end, isn't there any better tool with the property of > multi-reference? Hi Somayeh, BLEU has defined treatment for multiple references from the very beginning (see the original Papineni et al 2002 paper for details). Any implementation of BLEU that does not support multiple references should be considered defective. Personally I've always used mteval-v13a from http://www.itl.nist.gov/iad/mig/tests/mt/2009/ which has no problem dealing with multiple references at all. All you need to do is to provide the multiple references as multiple doc sections in your reference set: ... ... ... Disclaimer: The above definitely works for v13a but I'm not specifically familiar with v11b. Cheers, Liu Chang National University of Singapore ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] mteval-v11b
Hello, I have some question about mteval-v11b.pl 1) It can not use multi-reference with mteval what is a equivalent tool for this aim? 2) I tried multi-bleu.perl, but the scores reduced ! while we expect to increase while adding more reference sets !! How it is may? 3) I test mteval-v11b.pl and multi-bleu.perl in equivalent situations, they do not always agree ! sometimes mteval and sometimes the other gives better scores. Is there any problem? 4) and at the end, isn't there any better tool with the property of multi-reference? -- Best Regards, S.Bakhshaei ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
