Hello Barry, hello Tom, thank you for your answers. I think I have a better idea about different approaches to MOSES efficiency issues now.
Best regards, Sandra -----Ursprüngliche Nachricht----- Von: Barry Haddow [mailto:bhad...@inf.ed.ac.uk] Gesendet: Montag, 31. Januar 2011 10:52 An: moses-support@mit.edu Cc: Noubours, Sandra; Tom Hoar Betreff: Re: [Moses-support] running moses on a cluster with sge Hi Sandra The short answer is that it really depends how big your models are. Running on a cluster helps speed up tuning because most of the time in tuning is spent decoding, which can be easily parallelised by splitting up the file into chunks. So each of the individual machines should be capable of loading your models and running a decoder. The problem with using a cluster (as opposed to multicore) is that each machine has to have its own ram, and if you want to load large models then you need a lot of ram. Whereas with multicore, each thread can access the same model. Sure, binarising saves a lot on ram usage, but it slows you down and puts a lot of load on the filesystem which can cause problems on clusters. Our group's machines are a mixture of 8 and 16 core Xeon 2.67GHz, with 36-72G ram, no sge. We also have access to the university cluster, but since the most ram you can get is 16G and sge hold jobs don't work at the moment we don't really use it for moses any more, hope that helps - regards - Barry On Monday 31 January 2011 07:42, Noubours, Sandra wrote: > Hello, > > > > thanks for the tips! When talking about using a Sun Grid Engine I was > referring tuning. Making use of a cluster is supposed to speed up the > tuning process (see http://www.statmt.org/moses/?n=Moses.FAQ#ntoc10). In > this context I wondered what hardware exactly is needed for such a cluster. > > > > Sandra > > > > > > > > Von: Tom Hoar [mailto:tah...@precisiontranslationtools.com] > Gesendet: Freitag, 28. Januar 2011 09:01 > An: Noubours, Sandra > Cc: moses-support@mit.edu > Betreff: Re: [Moses-support] running moses on a cluster with sge > > > > Sandra, > > What kind of capacity do you need to support? I just finished translating > 21,000 pages, over 1/2 million phrases, in 22 hours on an old Intel > Core2Quad, 2.4 Ghz with 4 GB RAM and a 4-disk RAID-0. Moses was configured > with binarized phrase/reordering tables and kenlm binarized language model. > The advances in Moses supporting efficient binarized tables/models are > great! > > We're planning tests for a 2-socket host with two Intel Xeon 5680 6-core > 3.33 Ghz CPU's, 48 GB RAM and 4 1-TB disks as RAID0. With 12 cores > (totaling 24 simultaneous threads according to Intel specs), we're > expecting to boot capacity to well over 15 million phrases per day on one > host. > > What's the advantage of running Moses on a grid or cluster? > > Tom > > > > On Fri, 28 Jan 2011 08:40:22 +0100, "Noubours, Sandra" > <sandra.noubo...@fkie.fraunhofer.de> wrote: > > Hello, > > > > I would like to run Moses on a cluster. I am yet inexperienced in using > Sun Grid as well as clusters in common. Could you give me any instructions > or tips for implementing a Linux-Cluster with Sun Grid Engine for running > Moses? > > a) What kind of cluster would you recommend, i.e. how many > machines, > how many cpus, what memory, etc.? > > b) When tuning is performed with the multicore option it does not > use > more than one cpu. Does the tuning step use more than one cpu when run on a > cluster? > > c) Can Sun Grid implement a cluster virtually on one computer, so > that jobs are spread locally to different cpus of one computer? > > > > Thank you and best regards! > > > > Sandra -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
