Hi Xianhua > that to reduce the server burden? So the question is like, will the > improvement we get while run deterministic mert still be observed while > run undeterministic mert?
It depends how large the improvement is. If you run (underterministic) mert a few times on your baseline then you will see how much variation there is, and that will tell you (roughly) how big your improvement needs to be before you can be confident about it, cheers - Barry On Tuesday 01 November 2011 09:06:47 Li Xianhua wrote: > Hi Barry, > > Thanks for your suggestion. I am doing Japanese-Chinese machine > translation, solving the reordering problem. I want to make sure whether > my method improves the results. Our server is not strong enough and > several people run experiments on it. So I am wondering, can I first run > deterministic mert to make sure the method is effective, then run > undeterministic mert several times and average them to get the results, so > that to reduce the server burden? So the question is like, will the > improvement we get while run deterministic mert still be observed while > run undeterministic mert? > > ---------------------------------------------------- > Best wishes! > Xianhua Li > Information Technology Laboratory > Fujitsu Research & Development Center Co.,LTD. > 13F Tower A, Ocean International Center, > No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025 > E-mail:lixian...@cn.fujitsu.com > > > -----邮件原件----- > 发件人: Barry Haddow [mailto:bhad...@staffmail.ed.ac.uk] > 发送时间: 2011年11月1日 16:51 > 收件人: moses-support@mit.edu > 抄送: Li Xianhua > 主题: Re: [Moses-support] mert-moses.pl script > > Hi Xianhua > > Way 1 uses no random restarts at all, so each iteration starts where the > last one left off, which is why the results are always the same. The other > methods both use random restarts (which make them different from Way 1) > and they each use fixed seeds (so they are deterministic) but the seeds > are different (so they are different from each other). > > My guess is that Ways 2 & 3 will give better results as they explore a > larger part of the parameter space. However the best method is to follow > Clark et al. and run mert several times and average. This advice also > applies to most other methods of tuning MT systems, in particular online > algorithms where the order of the data may matter. > > cheers - Barry > > On Tuesday 01 November 2011 01:42:26 Li Xianhua wrote: > > Hi all, > > > > About the moses mert problem Neda mentioned, I tried three ways to > > make MERT deterministic, but their results between them are different. > > Way 1: edit mert-moses.pl line 105: "my $_RANDOM-RESTARTS=20", > > change 20 to 0 > > Way 2: switch to the mert-moses.pl call: --mertargs=" -r $seed " as > > Patrik said > > Way 3: activate flag "--predictable-seed", as Nicola and Barry said > > > > I ran totally 9 experiments, 3 for each way with the same corpus and > > parameters, and compared the results. the results in the same way are > > the same, but the results with the same corpus but different ways are > > different. So my questions are: > > Are the three ways right to make MERT deterministic? Which is best? > > Why are they different? > > > > ---------------------------------------------------- > > Best wishes! > > Xianhua Li > > Information Technology Laboratory > > Fujitsu Research & Development Center Co.,LTD. > > 13F Tower A, Ocean International Center, > > No.56 Dong Si Huan Zhong Rd, Chaoyang District, Beijing, China ,100025 > > E-mail:lixian...@cn.fujitsu.com > > > > > > > > _______________________________________________ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- > Barry Haddow > University of Edinburgh > +44 (0) 131 651 3173 > > -- > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > -- Barry Haddow University of Edinburgh +44 (0) 131 651 3173 -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
