Dear Hieu, Thanks for the info on the KenLM,
Regards, Liling On Tue, Aug 11, 2015 at 5:57 PM, Hieu Hoang <hieuho...@gmail.com> wrote: > > > On 10/08/2015 17:22, liling tan wrote: > > Dear Moses devs/users, > > @Marcin @Ken , Thanks for the tips on the -S for build_binary, RAM > estimation and the probing vs trie explanations. > > Just to do a check, currently, is there an option for lmplz to output > binarized directly without going through ARPA? If there is, is there also a > binary to arpa dumping mechanism? > > as far as i know, neither of these options are available in the current > version of kenlm > > > Regards, > LIling > > > > > On Fri, Aug 7, 2015 at 9:31 PM, liling tan <alvati...@gmail.com> wrote: > >> Dear Moses dev/users, >> >> On a related note, without multi-threads, can anyone give a gauge of how >> much RAM is required to binarized a 80GB (compressed .gz) 6gram arpa file? >> The no. of ngrams are: >> >> \data\ >> ngram 1=7503209 >> ngram 2=131003943 >> ngram 3=671005861 >> ngram 4=1510529519 >> ngram 5=2165163610 >> ngram 6=2477533666 >> >> >> Also, how long would it take (single-threadedly) on a 2.4Ghz core with >> 128GB RAM? Is there a way to mathematically estimate the time taken and RAM >> required to binarize a language model? >> >> Also, is binarized and quantized LM from KenLM lossy? If so how lossy? >> The KenLM paper states "To conserve memory at the expense of accuracy, >> values may be quantized using q bits per probability and r bits per >> backoff". Can someone help point us to papers that quanitfy how lossy it >> gets in terms of MT experiments or word perplexity task? >> >> Thanks in advance for the pointers! >> >> Regards, >> Liling >> >> On Fri, Aug 7, 2015 at 8:56 PM, liling tan < <alvati...@gmail.com> >> alvati...@gmail.com> wrote: >> >>> Dear Moses dev/users, >>> >>> Is there multithread option for KenLM's build_binary? >>> >>> Regards, >>> Liling >>> >> >> > > > _______________________________________________ > Moses-support mailing > listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support > > > -- > Hieu Hoang > Researcher > New York University, Abu Dhabihttp://www.hoang.co.uk/hieu > >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
