Hi,
I'm assuming you have enough RAM to fit everything. The kernel seems
to preferentially evict mmapped pages as memory usage approaches full
(it doesn't have to be full). To work around this, use
load=read
in your moses.ini line for the models. REMOVE any "lazyken" argument
which is deprecated and might override the load= argument.
The effect of load=read is to malloc (ok, anonymous mmap which is how
malloc is implemented anyway) at a 1 GB aligned address (to optimize for
huge pages) and read() the file into that memory. It will no longer
share across processes, but memory will have the same swapiness as the
rest of the Moses process.
Lazy loading will only make things worse here.
Kenneth
On 04/10/2016 07:29 AM, Jorg Tiedemann wrote:
> Hi,
>
> I have a large language model from the common crawl data set and it
> takes forever to load when running moses.
> My model is a trigram kenlm binarized with quantization, trie structures
> and pointer compression (-a 22 -q 8 -b 8).
> The model is about 140GB and it takes hours to load (I’m still waiting).
> I run on a machine with 256GB RAM ...
>
> I also tried lazy loading without success. Is this normal or do I do
> something wrong?
> Thanks for your help!
>
> Jörg
>
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
