I am surt that I can not find the unknown word in the phrase table. But the unknown word is in the dev set. And during the MERT, the unknown word appears in the translation of dev set, even though adding the '-unknown-word' switch. I am rather fuzzy.
2013/6/11 Hieu Hoang <[email protected]> > strange. I'm pretty sure > -drop-unknown > works. > > how exactly did you search your training set or phrase table? eg. > zcat phrase-table.gz | grep "^word " > are you sure your phrase table and terminal is in the same encoding? > > if you make your phrase table available for download, i can double check > > > On 11 June 2013 05:23, Li Xiang <[email protected]> wrote: > >> Yes. I can't find the word that appears in the output in the training set. >> >> Xiang Li >> >> 在 2013年6月9日，18:05，Hieu Hoang <[email protected]> 写道： >> >> looks ok. Are you sure your phrase-table doesn't have the translation? >> >> input: day BLAHBLAH >> $ ../moses_chart -f moses.ini -i 1 >> Tag BLAHBLAH >> $ ../moses_chart -f moses.ini -i 1 -drop-unknown >> Tag >> >> >> On 9 June 2013 06:28, Li Xiang <[email protected]> wrote: >> >>> Hi, >>> >>> I add the option '-drop-unknown' when decoding based on hierarchy model. >>> But I can also find some words which is in the source language. I don't >>> know whether these words are unknown words. Maybe the option >>> '-drop-unknown' does not work for moses_chart. >>> >>> -- >>> Xiang Li >>> >>> _______________________________________________ >>> Moses-support mailing list >>> [email protected] >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> >> >> -- >> Hieu Hoang >> Research Associate >> University of Edinburgh >> http://www.hoang.co.uk/hieu >> >> > > > -- > Hieu Hoang > Research Associate > University of Edinburgh > http://www.hoang.co.uk/hieu > > > _______________________________________________ > Moses-support mailing list > [email protected] > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- Xiang Li
_______________________________________________ Moses-support mailing list [email protected] http://mailman.mit.edu/mailman/listinfo/moses-support
