Re: [Moses-support] unsolved) Moses eating up too much memory
In the moses.ini file, change PhraseDictionaryMemory to PhraseDictionaryBinary This will let the decoder know that you want to use the binary phrase-table On 29 October 2013 06:10, Andrew Shin wrote: > I apologize since my previous question on the same topic didn't specify > enough. > > I'm following the steps in http://www.statmt.org/moses/?n=Moses.Baseline > exactly > the same way, > and when I try to run Moses with the following command: > ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with > -output-hypo-score option) > it starts eating up RAM until there's no space and starts taking hard disk > space as swap memory, > while it says its reading from phrase table. > It went up to 17GB of swap memory and pretty much stopped. > > I binarised the phrase table and reordering table, and the following are > the sizes of the binarised files > -rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx > -rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45 > phrase-table.binphr.srctree.wa > -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45 > phrase-table.binphr.srcvoc > -rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45 > phrase-table.binphr.tgtdata.wa > -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45 > phrase-table.binphr.tgtvoc > -rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00 > reordering-table.binlexr.idx > -rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00 > reordering-table.binlexr.srctree > -rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00 > reordering-table.binlexr.tgtdata > -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00 > reordering-table.binlexr.voc0 > -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00 > reordering-table.binlexr.voc1 > > I also attached moses.ini which I tried to run. > I'm very sure that this is not the way it should be, but can't figure out > what is wrong since I followed the tutorial above..(and it's worked before) > Please let me know if there is any additional information I should provide. > > I sincerely thank you very much for your time. > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
Re: [Moses-support] unsolved) Moses eating up too much memory
Hi Andrew In your moses.ini file I see the following line: > 0.00820395 on dev /Users/ipadmin/Desktop/blcorpus/tuning.true.src This is extremely low, and suggests something went wrong in tuning ... although this should not cause a memory leak. I also noticed that you are decoding files of 700,000 sentences. The problem with this is that Moses will load the whole file into memory before it starts decoding. Again, this should not take up 21G of RAM, but it would be worth trying decoding a small set to see if you still have the memory problems, cheers - Barry On 29/10/13 06:10, Andrew Shin wrote: > I apologize since my previous question on the same topic didn't > specify enough. > > I'm following the steps in > http://www.statmt.org/moses/?n=Moses.Baseline exactly the same way, > and when I try to run Moses with the following command: > ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with > -output-hypo-score option) > it starts eating up RAM until there's no space and starts taking hard > disk space as swap memory, > while it says its reading from phrase table. > It went up to 17GB of swap memory and pretty much stopped. > > I binarised the phrase table and reordering table, and the following > are the sizes of the binarised files > -rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx > -rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45 > phrase-table.binphr.srctree.wa > -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45 > phrase-table.binphr.srcvoc > -rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45 > phrase-table.binphr.tgtdata.wa > -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45 > phrase-table.binphr.tgtvoc > -rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00 > reordering-table.binlexr.idx > -rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00 > reordering-table.binlexr.srctree > -rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00 > reordering-table.binlexr.tgtdata > -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00 > reordering-table.binlexr.voc0 > -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00 > reordering-table.binlexr.voc1 > > I also attached moses.ini which I tried to run. > I'm very sure that this is not the way it should be, but can't figure > out what is wrong since I followed the tutorial above..(and it's > worked before) Please let me know if there is any additional > information I should provide. > > I sincerely thank you very much for your time. > > > > ___ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
[Moses-support] unsolved) Moses eating up too much memory
I apologize since my previous question on the same topic didn't specify enough.I'm following the steps in http://www.statmt.org/moses/?n=Moses.Baseline exactly the same way,and when I try to run Moses with the following command:~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with -output-hypo-score option)it starts eating up RAM until there's no space and starts taking hard disk space as swap memory,while it says its reading from phrase table.It went up to 17GB of swap memory and pretty much stopped.I binarised the phrase table and reordering table, and the following are the sizes of the binarised files-rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx-rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45 phrase-table.binphr.srctree.wa-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45 phrase-table.binphr.srcvoc-rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45 phrase-table.binphr.tgtdata.wa-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45 phrase-table.binphr.tgtvoc-rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00 reordering-table.binlexr.idx-rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00 reordering-table.binlexr.srctree-rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00 reordering-table.binlexr.tgtdata-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00 reordering-table.binlexr.voc0-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00 reordering-table.binlexr.voc1I also attached moses.ini which I tried to run.I'm very sure that this is not the way it should be, but can't figure out what is wrong since I followed the tutorial above..(and it's worked before) Please let me know if there is any additional information I should provide.I sincerely thank you very much for your time. moses.ini Description: Binary data ___ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
