How much RAM do you have? Have you tried Marcin's suggestion regarding the trie?
To filter you need to 1) have a test set 2) match that test set to the phrase table to produce a target vocabulary and 3) provide that target vocabulary to the filter. See https://github.com/kpu/kenlm/blob/master/lm/filter/phrase_table_vocab_main.cc which should work if you don't have factors . Kenneth On 04/25/2015 04:08 PM, liling tan wrote: > Dear Moses devs/users, > > @Ken, MERT with 100 iterations might be an overkill but MERT with 20 > also wouldn't finished in a day on the 71GB binarzied LM. It was at the > 5th iteration when i killed it. > > @Marcin, I'll try to retune with the trie LM and see how far it goes. > > *Does the build_binary come with thread options?* > * > * > *Also, would the KenLM filter help? How does the filter work? * > > I've tried on the 71GB LM: > > ~/moses/bin/filter union lm.en.binary lm.filter.binary > > > After waiting for 15 mins, it sort of looks like there nothing going on > and looks stuck. > More details > on: > http://stackoverflow.com/questions/29869607/how-to-tune-a-machine-translation-model-with-huge-language-model > > Regards, > Liling > > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
