Well, it seems as if there is at least some interest in highlighting the issue. I suggest detailed discussion take place off the Moses support list, so if you are interested in discussing what steps we could take (running some experiments and writing a paper seems like the likely strategy, but what experiments, what venue, etc...), email me directly. (You don't have to commit to working on the paper to take part.)
Suzy On 26/03/11 12:08 AM, Barry Haddow wrote: > This might be what Miles is referring too > http://www.statmt.org/wmt09/pdf/WMT-0939.pdf > > There was some progress towards getting this into moses > http://lium3.univ-lemans.fr/mtmarathon2010/ProjectFinalPresentation/MERT/StabilizingMert.pdf > > On Friday 25 March 2011 13:02, Miles Osborne wrote: >> There is work published on making mert more stable (on the train so can't >> easily dig it up) >> >> Miles >> >> sent using Android >> >> On 25 Mar 2011 12:49, "Lane Schwartz"<dowob...@gmail.com> wrote: >> >> We know that there is nondeterminism during optimization, yet virtually all >> papers report results based on a single MERT run. We know that results can >> very widely based on language pair and data sets, but a large majority of >> papers report results on a single language pair, and often for a single >> data set. >> >> While these issues are widely known at the informal level, I think that >> Suzy's point is well taken. I think there would be value in published >> studies showing just how wide the gap due to nondeterminism can be expected >> to be. It may be that such studies already exist, and I'm just not aware of >> them. Does anyone know of any? >> >> Cheers, >> Lane >> >> On Fri, Mar 25, 2011 at 7:03 AM, Barry Haddow<bhad...@inf.ed.ac.uk> wrote: >>> Hi >>> >>> This is an is... > -- Suzy Howlett http://www.showlett.id.au/ _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
