Hi James, Well, it's pretty straightforward: The decoder's job is to find the hypothesis with the maximum model score. That's why everybody builds models which assign high model score to high-quality translations. Unfortunately, you missed this last point in your own work.
Cheers, Matthias On Fri, 2015-06-19 at 14:15 +0000, Read, James C wrote: > I'm gonna try once more. This is what he said: > > "the decoder's job is NOT to find the high quality translation" > > The next time I have a panel of potential investors in front of me > I'm gonna pass that line by them and see how it goes down. I stress > the words HIGH QUALITY TRANSLATION. > > Please promise me that the next time you put in a bid for funding you > will guarantee your prospective funders that under no circumstances > will you attempt to design a system which searches for HIGH QUALITY > TRANSLATION. > > James > > ________________________________________ > From: Matthias Huck <mh...@inf.ed.ac.uk> > Sent: Friday, June 19, 2015 5:08 PM > To: Read, James C > Cc: Hieu Hoang; moses-support@mit.edu; Arnold, Doug > Subject: Re: [Moses-support] Major bug found in Moses > > Hi James, > > Yes, he just said that. > > The decoder's job is to find the hypothesis with the maximum model > score. That's one reason why your work is flawed. You did not care at > all whether your model score correlates with BLEU or not. > > Cheers, > Matthias > > > On Fri, 2015-06-19 at 13:24 +0000, Read, James C wrote: > > I quote: > > > > > > "the decoder's job is NOT to find the high quality translation" > > > > > > > > Did you REALLY just say that? > > > > > > James > > > > > > > > > > ______________________________________________________________________ > > From: Hieu Hoang <hieuho...@gmail.com> > > Sent: Wednesday, June 17, 2015 9:00 PM > > To: Read, James C > > Cc: Kenneth Heafield; moses-support@mit.edu; Arnold, Doug > > Subject: Re: [Moses-support] Major bug found in Moses > > > > the decoder's job is NOT to find the high quality translation (as > > measured by bleu). It's job is to find translations with high model > > score. > > > > > > you need the tuning to make sure high quality translation correlates > > with high model score. If you don't tune, it's pot luck what quality > > you get. > > > > > > You should tune with the features you use > > > > > > Hieu Hoang > > Researcher > > > > New York University, Abu Dhabi > > > > http://www.hoang.co.uk/hieu > > > > > > On 17 June 2015 at 21:52, Read, James C <jcr...@essex.ac.uk> wrote: > > The analogy doesn't seem to be helping me understand just how > > exactly it is a desirable quality of a TM to > > > > a) completely break down if no LM is used (thank you for > > showing that such is not always the case) > > b) be dependent on a tuning step to help it find the higher > > scoring translations > > > > What you seem to be essentially saying is that the TM cannot > > find the higher scoring translations because I didn't pretune > > the system to do so. And I am supposed to accept that such is > > a desirable quality of a system whose very job is to find the > > higher scoring translations. > > > > Further, I am still unclear which features you prequire a > > system to be tuned on. At the very least it seems that I have > > discovered the selection process that tuning seems to be > > making up for in some unspecified and altogether opaque way. > > > > James > > > > > > ________________________________________ > > From: Hieu Hoang <hieuho...@gmail.com> > > Sent: Wednesday, June 17, 2015 8:34 PM > > To: Read, James C; Kenneth Heafield; moses-support@mit.edu > > Cc: Arnold, Doug > > Subject: Re: [Moses-support] Major bug found in Moses > > > > 4 BLEU is nothing to sniff at :) I was answering Ken's tangent > > aspersion > > that LM are needed for tuning. > > > > I have some sympathy for you. You're looking at ways to > > improve > > translation by reducing the search space. I've bashed my head > > against > > this wall for a while as well without much success. > > > > However, as everyone is telling you, you haven't understood > > the role of > > tuning. Without tuning, you're pointing your lab rat to some > > random part > > of the search space, instead of away from the furry animal > > with whiskers > > and towards the yellow cheesy thing > > > > On 17/06/2015 20:45, Read, James C wrote: > > > Doesn't look like the LM is contributing all that much then > > does it? > > > > > > James > > > > > > ________________________________________ > > > From: moses-support-boun...@mit.edu > > <moses-support-boun...@mit.edu> on behalf of Hieu Hoang > > <hieuho...@gmail.com> > > > Sent: Wednesday, June 17, 2015 7:35 PM > > > To: Kenneth Heafield; moses-support@mit.edu > > > Subject: Re: [Moses-support] Major bug found in Moses > > > > > > On 17/06/2015 20:13, Kenneth Heafield wrote: > > >> I'll bite. > > >> > > >> The moses.ini files ship with bogus feature weights. One > > is required to > > >> tune the system to discover good weights for their system. > > You did not > > >> tune. The results of an untuned system are meaningless. > > >> > > >> So for example if the feature weights are all zeros, then > > the scores are > > >> all zero. The system will arbitrarily pick some awful > > translation from > > >> a large space of translations. > > >> > > >> The filter looks at one feature p(target | source). So now > > you've > > >> constrained the awful untuned model to a slightly better > > region of the > > >> search space. > > >> > > >> In other words, all you've done is a poor approximation to > > manually > > >> setting the weight to 1.0 on p(target | source) and the > > rest to 0. > > >> > > >> The problem isn't that you are running without a language > > model (though > > >> we generally do not care what happens without one). The > > problem is that > > >> you did not tune the feature weights. > > >> > > >> Moreover, as Marcin is pointing out, I wouldn't necessarily > > expect > > >> tuning to work without an LM. > > > Tuning does work without a LM. The results aren't half bad. > > fr-en > > > europarl (pb): > > > with LM: 22.84 > > > retuned without LM: 18.33 > > >> On 06/17/15 11:56, Read, James C wrote: > > >>> Actually the approximation I expect to be: > > >>> > > >>> p(e|f)=p(f|e) > > >>> > > >>> Why would you expect this to give poor results if the TM > > is well trained? Surely the results of my filtering > > experiments provve otherwise. > > >>> > > >>> James > > >>> > > >>> ________________________________________ > > >>> From: moses-support-boun...@mit.edu > > <moses-support-boun...@mit.edu> on behalf of Rico Sennrich > > <rico.sennr...@gmx.ch> > > >>> Sent: Wednesday, June 17, 2015 5:32 PM > > >>> To: moses-support@mit.edu > > >>> Subject: Re: [Moses-support] Major bug found in Moses > > >>> > > >>> Read, James C <jcread@...> writes: > > >>> > > >>>> I have been unable to find a logical explanation for this > > behaviour other > > >>> than to conclude that there must be some kind of bug in > > Moses which causes a > > >>> TM only run of Moses to perform poorly in finding the most > > likely > > >>> translations according to the TM when > > >>>> there are less likely phrase pairs included in the > > race. > > >>> I may have overlooked something, but you seem to have > > removed the language > > >>> model from your config, and used default weights. your > > default model will > > >>> thus (roughly) implement the following model: > > >>> > > >>> p(e|f) = p(e|f)*p(f|e) > > >>> > > >>> which is obviously wrong, and will give you poor results. > > This is not a bug > > >>> in the code, but a poor choice of models and weights. > > Standard steps in SMT > > >>> (like tuning the model weights on a development set, and > > including a > > >>> language model) will give you the desired results. > > >>> > > >>> _______________________________________________ > > >>> Moses-support mailing list > > >>> Moses-support@mit.edu > > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > > >>> > > >>> _______________________________________________ > > >>> Moses-support mailing list > > >>> Moses-support@mit.edu > > >>> http://mailman.mit.edu/mailman/listinfo/moses-support > > >> _______________________________________________ > > >> Moses-support mailing list > > >> Moses-support@mit.edu > > >> http://mailman.mit.edu/mailman/listinfo/moses-support > > >> > > > -- > > > Hieu Hoang > > > Researcher > > > New York University, Abu Dhabi > > > http://www.hoang.co.uk/hieu > > > > > > _______________________________________________ > > > Moses-support mailing list > > > Moses-support@mit.edu > > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > . > > > > > > > -- > > Hieu Hoang > > Researcher > > New York University, Abu Dhabi > > http://www.hoang.co.uk/hieu > > > > > > > > > > _______________________________________________ > > Moses-support mailing list > > Moses-support@mit.edu > > http://mailman.mit.edu/mailman/listinfo/moses-support > > > > -- > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
