I have found the answer on the kenlm web page and it seems to be working: Full or lazy loading
KenLM supports lazy loading via mmap. This allows you to further reduce memory usage, especially with trie which has good memory locality. In Moses, this is controlled by the language model number in moses.ini. Using language model number 8 will load the full model into memory (MAP_POPULATE on Linux and read() on other OSes). Language model number 9 will lazily load the model using mmap. I recommend fully loading if you have the RAM for it; it actually takes less time to load the full model and use it because the disk does not have to seek during decoding. Lazy loading works best with local disk and is not recommended for networked filesystems. On Thu, Mar 20, 2014 at 2:32 PM, Alexandra Birch <lexi.bi...@gmail.com>wrote: > Hi there, > > I want to run 4 MT servers at the same time on a machine with limited > memory. Kenlm seems to reserve the amount of memory which the language > model would have taken if it had been loaded into memory. So I don't have > enough memory to run all these servers and the machine grinds to a halt if > I try. Is there any flag I could use which would limit the amount of memory > reserved? > > Lexi >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
