naturally, the parallel data could be down-sampled (eg use 1/2 of it). you probably won't see a significant degradation in translation quality and the whole training process will use less RAM and will be quicker.
Miles On 18 April 2011 15:05, Tom Hoar <tah...@precisiontranslationtools.com> wrote: > Your report of 100% physical usage, growing swap usage and low CPU load > is normal when working with limited RAM machines. With only 4 Gb Ram and > the new (larger) EuroParl v6 corpus, you could train for 3 or 4 days > depending on how you setup your swap partition. Even then, it's possible > you will run out of RAM before it's finished. Upgrading to 8 Gb ram is a > move in the right direction. > > Once it's finished training, you'll want to use the binarized the > tables and language model, which MMM's train-1.11 script creates. > > Tom > > > On Mon, 18 Apr 2011 14:52:10 +0100, Philipp Koehn <pko...@inf.ed.ac.uk> > wrote: >> Hi, >> >> I am not familiar with the MMM setup, but one of the causes >> of memory use may be the translation table. You should use >> the on-disk translation table. >> >> -phi >> >> On Mon, Apr 18, 2011 at 2:47 PM, David Wilkinson >> <davidzw...@hotmail.com> wrote: >>> I have set up an Ubuntu 10.04 system with the moses-for-mere-mortals >>> scripts. The default corpus trained in about 6-7 hours on my system >>> (Athlon >>> x3 3.2Ghz, 4Gb Ram). I am now trying to train the system with the >>> Europarl >>> German-English parallel corpus (about 45m words in each language), >>> again >>> using the default moses-for-mere-mortals settings. The system has >>> been >>> running for 24 hrs and is currently using all the physical memory >>> and about >>> 1.2Gb of swap. None of the cores are being used more than 10%, so >>> like this >>> it will take a very long time to finish. If I double the ram to 8gb, >>> will >>> this be sufficient? >>> Many Thanks >>> David >>> _______________________________________________ >>> Moses-support mailing list >>> Moses-support@mit.edu >>> http://mailman.mit.edu/mailman/listinfo/moses-support >>> >>> >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
