Hi Neda If you're interested in determining whether translation method A is better than method B, or not, then you shouldn't use predictable-seed. You should run mert several times and take the average score. Because if you run with predictable seeds then all you are verifying is whether A beats B using the rather arbitrary seed settings from deterministic mert, nothing more.
I'd suggest that you read the paper I linked - it explains these points and more, cheers - Barry On Monday 31 Oct 2011 08:35:01 Nicola Bertoldi wrote: > Hi Neda > > There is also a parameter of mert-moses.pl "--predictable-seed" > (see the help) which makes MERT deterministic. > > In general, MERT procedure relies on an initial seed seed_t at each > iteration t for the creation of the (20 by default) random starting > points for the optimization; not that seed_i differs from seed_j (if > i differs from j) > > If you activate this flag "--predictable-seed", seed_i still differs from > seed_j BUT the sequence seed_1, seed_2, ... seed_N is always the same; > and hence, the random starting points considered for the optimization are > always the same, so that the final results of the MERT is always the > same. > > best regards > Nicola > > On Oct 29, 2011, at 6:37 PM, Patrik Lambert wrote: > > Hi Neda, > > > > this happens because the seed used in the MERT optimizer depends by > > default on the moment on which you launch it. > > If you need deterministic MERT runs, you can set the seed by adding this > > switch to the mert-moses.pl call: > > > > --mertargs=" -r $seed " > > > > Patrik > > > > Le 29/10/2011 18:11, moses-support-requ...@mit.edu a écrit : > >> Message: 3 > >> Date: Sat, 29 Oct 2011 17:05:11 +0100 > >> From: Barry Haddow<bhad...@staffmail.ed.ac.uk> > >> Subject: Re: [Moses-support] mert-moses.pl script > >> To: moses-support@mit.edu > >> Message-ID:<201110291705.11699.bhad...@staffmail.ed.ac.uk> > >> Content-Type: Text/Plain; charset="utf-8" > >> > >> Hi Neda > >> > >> Yes, this is quite normal. The best plan is to do several runs and take > >> the average bleu. See this paper for a discussion > >> > >> http://www.cs.cmu.edu/~jhclark/pubs/significance.pdf > >> > >> cheers - Barry > >> > >> On Saturday 29 Oct 2011 10:09:29 Neda NoorMohammadi wrote: > >>> hello, > >>> > >>> In my experiment this is happening: > >>> > >>> I am running mert-moses.pl script on a same develop set and configure > >>> file in a same condition but by each execution I gain different weights > >>> which leads to different Bleu (about 1%) (note that these executions > >>> are independent of each other and are initialized to predefine weights > >>> of moses.ini). > >>> > >>> The used features in configure file are: LM: 3gram, Reordering: > >>> msd-bidirectional-fe, Alignment: grow-diag-final-and, phrase scores. > >>> > >>> Is it normal? What is your suggestion? > >>> In this situation what is my baseline? The best one or the worth? > >>> > >>> Neda Noormohammadi > >> > >> ------------------------------ > >> > >> _______________________________________________ > >> Moses-support mailing list > >> Moses-support@mit.edu > >> http://mailman.mit.edu/mailman/listinfo/moses-support > >> > >> > >> End of Moses-support Digest, Vol 60, Issue 46 > >> ********************************************* > > > > _______________________________________________ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
