Re: [Moses-support] Filtering Binarized LM

2016-04-06 Thread Kenneth Heafield
Probing format models can't be filtered because they only retain hashes of ngrams. Trie format models can be filtered and dumped, but only with the very hacky and undocumented dump_trie program in the bounded-noquant branch. Hasn't been a priority to make it release quality; volunteers? Kenneth

[Moses-support] Filtering Binarized LM

2016-04-06 Thread liling tan
Dear Moses devs/users, The filter tool in KenLM is able to filter a LM based on a dev set ( https://kheafield.com/code/kenlm/filter/) but it only allows raw|arpa file. Is there another tool that filters binarized LMs? Given a binarized LM, is there a way to "debinarize" the LM? Thanks in