Please note that in order for the baseline to be meaningful it has to also use no LM. So, naturally the scores are lower than those of baselines you are referring to.
Regarding expectations. Are you seriously suggesting that we would expect the translation model to be incapable of finding higher scoring translations when not filtering out less likely phrase pairs? How high exactly would that rank on your desirable qualities of a TM list? James ________________________________________ From: amittai axelrod <amit...@umiacs.umd.edu> Sent: Wednesday, June 17, 2015 8:20 PM To: Read, James C; Hieu Hoang; Kenneth Heafield; moses-support@mit.edu Cc: Arnold, Doug Subject: Re: [Moses-support] Major bug found in Moses hi -- you might not be aware, but your emails sound almost belligerently confrontational. i can see how you would be frustrated, but starting a conversation with "i have found a major bug" and then repeatedly saying that "clearly" everything is broken -- that may not be the best way to convince the few hundred people on the mailing list of the soundness of your approach. also, your argument could be easily mis-interpreted as "this behavior is unexpected to me, ergo this is unexpected behavior", and that will unfortunately bias the listener against you, as that is the preferred argument structure of conspiracy theorists. at any rate, "the system" is designed to take a large number of phrase pairs and model scores cobble them together into a translation. it does do that. it appears that you have identified a different way of doing that cobbling-together, one that uses much fewer models -- so far so good! however, from reading your paper, it seems that your baseline is completely unoptimized, so performance gains against it may not show up in the real world. as specific examples, Table 1 in your paper shows that your baseline French-English system score is 11.36, Spanish-English is 7.16, and German-English is 6.70 BLEU. if you compare those baselines against published results in those languages from the previous few years, you will see that those scores are well off the mark. your position will be helped by showing results against a stronger, yet still basic, baseline. what happens if you compare your approach against a vanilla use of the Moses pipeline [this includes tuning]? cheers, ~amittai On 6/17/15 12:45, Read, James C wrote: > Doesn't look like the LM is contributing all that much then does it? > > James > > ________________________________________ > From: moses-support-boun...@mit.edu <moses-support-boun...@mit.edu> on behalf > of Hieu Hoang <hieuho...@gmail.com> > Sent: Wednesday, June 17, 2015 7:35 PM > To: Kenneth Heafield; moses-support@mit.edu > Subject: Re: [Moses-support] Major bug found in Moses > > On 17/06/2015 20:13, Kenneth Heafield wrote: >> I'll bite. >> >> The moses.ini files ship with bogus feature weights. One is required to >> tune the system to discover good weights for their system. You did not >> tune. The results of an untuned system are meaningless. >> >> So for example if the feature weights are all zeros, then the scores are >> all zero. The system will arbitrarily pick some awful translation from >> a large space of translations. >> >> The filter looks at one feature p(target | source). So now you've >> constrained the awful untuned model to a slightly better region of the >> search space. >> >> In other words, all you've done is a poor approximation to manually >> setting the weight to 1.0 on p(target | source) and the rest to 0. >> >> The problem isn't that you are running without a language model (though >> we generally do not care what happens without one). The problem is that >> you did not tune the feature weights. >> >> Moreover, as Marcin is pointing out, I wouldn't necessarily expect >> tuning to work without an LM. > Tuning does work without a LM. The results aren't half bad. fr-en > europarl (pb): > with LM: 22.84 > retuned without LM: 18.33 >> >> On 06/17/15 11:56, Read, James C wrote: >>> Actually the approximation I expect to be: >>> >>> p(e|f)=p(f|e) >>> >>> Why would you expect this to give poor results if the TM is well trained? >>> Surely the results of my filtering experiments provve otherwise. >>> >>> James >>> >>> ________________________________________ >>> From: moses-support-boun...@mit.edu <moses-support-boun...@mit.edu> on >>> behalf of Rico Sennrich <rico.sennr...@gmx.ch> >>> Sent: Wednesday, June 17, 2015 5:32 PM >>> To: moses-support@mit.edu >>> Subject: Re: [Moses-support] Major bug found in Moses >>> >>> Read, James C <jcread@...> writes: >>> >>>> I have been unable to find a logical explanation for this behaviour other >>> than to conclude that there must be some kind of bug in Moses which causes a >>> TM only run of Moses to perform poorly in finding the most likely >>> translations according to the TM when >>>> there are less likely phrase pairs included in the race. >>> I may have overlooked something, but you seem to have removed the language >>> model from your config, and used default weights. your default model will >>> thus (roughly) implement the following model: >>> >>> p(e|f) = p(e|f)*p(f|e) >>> >>> which is obviously wrong, and will give you poor results. This is not a bug >>> in the code, but a poor choice of models and weights. Standard steps in SMT >>> (like tuning the model weights on a development set, and including a >>> language model) will give you the desired results. >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > -- > Hieu Hoang > Researcher > New York University, Abu Dhabi > http://www.hoang.co.uk/hieu > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
