Thanks for the quick reply. I will try the load option. Quick question: You said that the memory will not be shared across processes with that option. Does that mean that it will load the LM for each thread? That would mean a lot in my setup.
By the way, I also did the cat >/dev/null thing but I didn’t have the impression that this changed a lot. Does it really help and how much would you usually gain? Thanks again! Jörg > On 10 Apr 2016, at 12:55, Kenneth Heafield <mo...@kheafield.com> wrote: > > Hi, > > I'm assuming you have enough RAM to fit everything. The kernel seems > to preferentially evict mmapped pages as memory usage approaches full > (it doesn't have to be full). To work around this, use > > load=read > > in your moses.ini line for the models. REMOVE any "lazyken" argument > which is deprecated and might override the load= argument. > > The effect of load=read is to malloc (ok, anonymous mmap which is how > malloc is implemented anyway) at a 1 GB aligned address (to optimize for > huge pages) and read() the file into that memory. It will no longer > share across processes, but memory will have the same swapiness as the > rest of the Moses process. > > Lazy loading will only make things worse here. > > Kenneth > > On 04/10/2016 07:29 AM, Jorg Tiedemann wrote: >> Hi, >> >> I have a large language model from the common crawl data set and it >> takes forever to load when running moses. >> My model is a trigram kenlm binarized with quantization, trie structures >> and pointer compression (-a 22 -q 8 -b 8). >> The model is about 140GB and it takes hours to load (I’m still waiting). >> I run on a machine with 256GB RAM ... >> >> I also tried lazy loading without success. Is this normal or do I do >> something wrong? >> Thanks for your help! >> >> Jörg >> >> >> >> >> >> >> >> >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
