hi raymond you're welcome to create a branch on the Moses github repository and add your code there. It's unlikely anyone will look at it or use it, but at least it won't get lost.
Maybe in future, you or someone else might want to merge it with the master branch where it will get much more exposure On 16 April 2014 16:21, Raymond W. M. Ng <wm...@sheffield.ac.uk> wrote: > Dear Ondrej, > > I have checked with Hieu when I met him in February, seems that the SGE > submission in MERT is using the -sync mode, which makes submission > difficult (user still in submission states until the all jobs end). In > short, the modification runs in a "no-sync" mode. > > In terms of efficiency, as for the reasons you have mentioned, the > combined wallclock time of N machines (N times actual program runtime) may > be longer than the single-threaded execution. But as in a lot of shared > computing environments, a single-threaded job execution for 20+ hours is > not favourable (sometimes disallowed). So by using the parallel mode, > runtime of single jobs is shortened. In my experience, using parallel mode > shortens the tuning time from 30hours (single-threaded) to 3 hours (20 > threads on different machines). We are having infiniband access among nodes > and its a bit more sophisticated than NFS mounting though. > > best > raymond > > > On 16 April 2014 13:48, Ondrej Bojar <bo...@ufal.mff.cuni.cz> wrote: > >> Dear Raymond, >> >> The existing scripts always allowed running MERT in parallel jobs on SGE, >> one just had to use generic/moses-parallel as the "moses executable". >> >> Is there some other functionality that your modifications now bring? >> >> Btw, in my experience, parallelization into SGE jobs can be even less >> efficient than single-job-multi-threaded execution. It is hard to exactly >> describe the circumstances, but in general if your models are big and >> loaded from NFS, and you run many experiments at the same time, the >> slowdown of the network multiplied by the many SGE jobs makes the >> parallelization much more wasteful and sometimes slower (in wallclock time). >> >> Cheers, Ondrej. >> >> On April 16, 2014 1:07:37 PM CEST, "Raymond W. M. Ng" < >> wm...@sheffield.ac.uk> wrote: >> >Hi Moses support, >> > >> >Not sure I am making this enquiry in the right mailing list.... >> >I have some modified scripts for parallel MERT tuning which can run on >> >SGE. >> >Now I would like to share this. It is based on an old version of moses >> >(around April 2012), what is the best way for sharing? >> > >> >Best >> >raymond >> > >> > >> >------------------------------------------------------------------------ >> > >> >_______________________________________________ >> >Moses-support mailing list >> >Moses-support@mit.edu >> >http://mailman.mit.edu/mailman/listinfo/moses-support >> >> -- >> Ondrej Bojar (mailto:o...@cuni.cz / bo...@ufal.mff.cuni.cz) >> http://www.cuni.cz/~obo >> >> > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support > > -- Hieu Hoang Research Associate University of Edinburgh http://www.hoang.co.uk/hieu
_______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
