yes, you can easily get a 1BP drop between multiple runs. if you want to do experiments and report BLEU scores then people really need to do multiple runs and report on averages, along with variances. i think from no-one i'm going to start penalising papers i get to review if people don't do something about this
(and i do a lot of reviewing ...) Miles 2010/1/11 李贤华 <08lixian...@gmail.com>: > hi, > > Thanks for you quick response. > But, will this cause a drop of BLEU, like, 0.5 point? > I thinks that's too much... > > I have run my baseline experiments three times, and got three different > results. > The results for test set are: 0.2798, 0.2741, 0.2790. > The first is run on server1 previously, > the second and the third are run recently, > while the second is run on server2, and the third is run on server1. > > Now I don't know what is my baseline. > > > > > Regards, > > Lee Xianhua > > ________________________________ > 2010-01-11 > > > ________________________________ > 发件人： Miles Osborne > 发送时间： 2010-01-11 16:12:38 > 收件人： 李贤华 > 抄送： moses-support > 主题： Re: [Moses-support] different servers + different time - > differentresult? > Giza++ and MERT both can produce different results, even when using > the same code, corpora etc. This is because multiple solutions exist > and each time you run Moses, you find one of these (different) optima. > Miles > 2010/1/11 李贤华 <08lixian...@gmail.com>: >> Hi all, >> >> I ran some experiments with moses like, half a year ago. >> And recently I ran them for a second time. >> The time I got the reuslts, I got confused. >> Beacause they're so different from those I got previously. >> >> The softwares I used was not changed, the same version. >> The corpus is of course the same. I just copied them. >> And I used the same script the run the experiments, just changed some >> directory. >> It seems I ran the same experiments on two different servers at different >> time, and got different results. >> >> I checked alignment results, aligned.grow-diag-and-final, >> and there're a lot of differences. >> I also checked moses.ini, and the parameters are greatly different. >> >> So, has anybody ever come into this situation? I'm really confused... >> >> >> >> Regards, >> >> Lee Xianhua >> >> ________________________________ >> 2010-01-11 >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
