Hi Ventzi The discussion has moved on from the 'beam search' issue, which is not 'the root of the problem'. This issue is (afaik) a fairly minor issue which is highly unlikely to change anyone's bleu score, but can cause minor irritations when checking results. It's not that straightforward to fix as it's related to srilm, which we can't change, but using kenlm seems to help.
The wider issue - which stimulated the discussion about papers - is that results can be influenced by the use of stochastic algorithms like mert, as well as choice of data set, language pair, choice of metric etc, and many other arbitrary features of the experimental setup. Results can also be influenced by unexpected sources of non-determinism, such as the pointer issues above, and out-and-out bugs. These influences are often not acknowledged in MT papers. best regards - Barry On Friday 25 March 2011 13:18, Венцислав Жечев (Ventsislav Zhechev) wrote: > Here is what I think about this whole issue. > > Wouldn’t it be much more beneficial to the whole community to fix the basic > beam search issue in Moses that seems to be the root of the problem, rather > than write papers investigating it? As far as I understand, the issue most > likely arises from the fact that the beam width limit might cut off some > hypotheses from a group with equal probability, while leaving some for > further processing. The fix would include checking the beam border and > including all hypotheses with the least permissible probability, regardless > of the actual beam width. This might require some redesign of the data > structures that hold the hypotheses (I’m writing this without actually > inspecting the source code), but this cannot be an untenable task. Let’s > say this takes 8–16 man-hours to code and another 8–16 to test. Any > volunteers in academia? > > > Cheers, > > Ventzi > > ––––––– > Dr. Ventsislav Zhechev > Computational Linguist > > CMS & Language Technologies > Localisation Services > Autodesk Development Sàrl > Neuchâtel, Switzerland > > http://VentsislavZhechev.eu > tel: +41 32 723 9122 > fax: +41 32 723 9399 > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
