Dear Ondrej, I have checked with Hieu when I met him in February, seems that the SGE submission in MERT is using the -sync mode, which makes submission difficult (user still in submission states until the all jobs end). In short, the modification runs in a "no-sync" mode.
In terms of efficiency, as for the reasons you have mentioned, the combined wallclock time of N machines (N times actual program runtime) may be longer than the single-threaded execution. But as in a lot of shared computing environments, a single-threaded job execution for 20+ hours is not favourable (sometimes disallowed). So by using the parallel mode, runtime of single jobs is shortened. In my experience, using parallel mode shortens the tuning time from 30hours (single-threaded) to 3 hours (20 threads on different machines). We are having infiniband access among nodes and its a bit more sophisticated than NFS mounting though. best raymond On 16 April 2014 13:48, Ondrej Bojar <bo...@ufal.mff.cuni.cz> wrote: > Dear Raymond, > > The existing scripts always allowed running MERT in parallel jobs on SGE, > one just had to use generic/moses-parallel as the "moses executable". > > Is there some other functionality that your modifications now bring? > > Btw, in my experience, parallelization into SGE jobs can be even less > efficient than single-job-multi-threaded execution. It is hard to exactly > describe the circumstances, but in general if your models are big and > loaded from NFS, and you run many experiments at the same time, the > slowdown of the network multiplied by the many SGE jobs makes the > parallelization much more wasteful and sometimes slower (in wallclock time). > > Cheers, Ondrej. > > On April 16, 2014 1:07:37 PM CEST, "Raymond W. M. Ng" < > wm...@sheffield.ac.uk> wrote: > >Hi Moses support, > > > >Not sure I am making this enquiry in the right mailing list.... > >I have some modified scripts for parallel MERT tuning which can run on > >SGE. > >Now I would like to share this. It is based on an old version of moses > >(around April 2012), what is the best way for sharing? > > > >Best > >raymond > > > > > >------------------------------------------------------------------------ > > > >_______________________________________________ > >Moses-support mailing list > >Moses-support@mit.edu > >http://mailman.mit.edu/mailman/listinfo/moses-support > > -- > Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) > http://www.cuni.cz/~obo > >
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
