Raymond, I would be very interested to see what you have. I agree that there are definitely environments where short jobs are required or preferred.
Thanks, Lane On Wed, Apr 16, 2014 at 11:21 AM, Raymond W. M. Ng <wm...@sheffield.ac.uk>wrote: > Dear Ondrej, > > I have checked with Hieu when I met him in February, seems that the SGE > submission in MERT is using the -sync mode, which makes submission > difficult (user still in submission states until the all jobs end). In > short, the modification runs in a "no-sync" mode. > > In terms of efficiency, as for the reasons you have mentioned, the > combined wallclock time of N machines (N times actual program runtime) may > be longer than the single-threaded execution. But as in a lot of shared > computing environments, a single-threaded job execution for 20+ hours is > not favourable (sometimes disallowed). So by using the parallel mode, > runtime of single jobs is shortened. In my experience, using parallel mode > shortens the tuning time from 30hours (single-threaded) to 3 hours (20 > threads on different machines). We are having infiniband access among nodes > and its a bit more sophisticated than NFS mounting though. > > best > raymond > > > On 16 April 2014 13:48, Ondrej Bojar <bo...@ufal.mff.cuni.cz> wrote: > >> Dear Raymond, >> >> The existing scripts always allowed running MERT in parallel jobs on SGE, >> one just had to use generic/moses-parallel as the "moses executable". >> >> Is there some other functionality that your modifications now bring? >> >> Btw, in my experience, parallelization into SGE jobs can be even less >> efficient than single-job-multi-threaded execution. It is hard to exactly >> describe the circumstances, but in general if your models are big and >> loaded from NFS, and you run many experiments at the same time, the >> slowdown of the network multiplied by the many SGE jobs makes the >> parallelization much more wasteful and sometimes slower (in wallclock time). >> >> Cheers, Ondrej. >> >> On April 16, 2014 1:07:37 PM CEST, "Raymond W. M. Ng" < >> wm...@sheffield.ac.uk> wrote: >> >Hi Moses support, >> > >> >Not sure I am making this enquiry in the right mailing list.... >> >I have some modified scripts for parallel MERT tuning which can run on >> >SGE. >> >Now I would like to share this. It is based on an old version of moses >> >(around April 2012), what is the best way for sharing? >> > >> >Best >> >raymond >> > >> > >> >------------------------------------------------------------------------ >> > >> >_______________________________________________ >> >Moses-support mailing list >> >Moses-support@mit.edu >> >http://mailman.mit.edu/mailman/listinfo/moses-support >> >> -- >> Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) >> http://www.cuni.cz/~obo >> >> > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- When a place gets crowded enough to require ID's, social collapse is not far away. It is time to go elsewhere. The best thing about space travel is that it made it possible to go elsewhere. -- R.A. Heinlein, "Time Enough For Love"
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
