Hi Szilárd, Sorry I was bit quick to say its working with pinoffset. I just submitted four same jobs (2 gpus, 4 nprocs) on the same node with -pin on and different -pinoffset to 0, 5, 10, 15 (numbers should be fine as there are 40 cores on node). Still I don't get same performance (all variably less than 50%) as expected from a single independent job. Now am wondering if its still related to overlap of cores as pin on should lock the cores for the same job.
-J On Wed, Sep 13, 2017 at 7:33 PM, gromacs query <gromacsqu...@gmail.com> wrote: > Hi Szilárd, > > Thanks, option 3 was in my mind but I need to figure out now how :) > Manually fixing pinoffset as of now seems working with some quick test. > I think option 1 would require to ask the admin but I can try option 3 > myself. As there are other users from different places who may not bother > using option 3. I think I would need to ask the admin to force option 1 but > before that I will try option 3. > > JIom > > On Wed, Sep 13, 2017 at 7:10 PM, Szilárd Páll <pall.szil...@gmail.com> > wrote: > >> J, >> >> You have a few options: >> >> * Use SLURM to assign not only the set of GPUs, but also the correct >> set of CPU cores to each mdrun process. If you do so, mdrun will >> respect the affinity mask it will inherit and your two mdrun jobs >> should be running on the right set of cores. This has the drawback >> that (AFAIK) SLURM/aprun (or srun) will not allow you to bind each >> application thread to a core/hardware thread (which is what mdrun >> does), only a process to a group of cores/hw threads which can >> sometimes lead to performance loss. (You might be able to compensate >> using some OpenMP library environment variables, though.) >> >> * Run multiple jobs with mdrun "-multi"/"-multidir" (either two per >> node or mulitple across nodes) and benefit from the rank/thread to >> core/hw thread assignment that's supported also across multiple >> simulations part of a multi-run; e.g.: >> mpirun -np 4 gmx mdrun -multi 4 -ntomp N -multidir my_input_dir{1,2,3,4} >> will launch 4 ranks and start 4 simulations in each of the four >> directories passed. >> >> * Write a wrapper script around gmx mdrun which will be what you >> launch with SLURM; you can then inspect the node and decide what >> pinoffset value to pass to your mdrun launch command. >> >> >> I hope one of these will deliver the desired results :) >> >> Cheers, >> -- >> Szilárd >> >> >> On Wed, Sep 13, 2017 at 7:47 PM, gromacs query <gromacsqu...@gmail.com> >> wrote: >> > Hi Szilárd, >> > >> > Thanks for your reply. This is useful but now am thinking because the >> slurm >> > launches job in an automated way it is not really in my control to >> choose >> > the node. So following things can happen; say for two mdrun jobs I set >> > -pinoffset 0 and -pinoffset 4; >> > >> > - if they are running on the same node this is good >> > - if jobs run on different nodes (partially occupied or free) whether >> these >> > chosen pinoffsets will make sense or not as I don't know what pinoffset >> I >> > would need to set >> > - if I have to submit many jobs together and slurm chooses >> different/same >> > node itself then I think it is difficult to define pinoffset. >> > >> > - >> > J >> > >> > On Wed, Sep 13, 2017 at 6:14 PM, Szilárd Páll <pall.szil...@gmail.com> >> > wrote: >> > >> >> My guess is that the two jobs are using the same cores -- either all >> >> cores/threads or only half of them, but the same set. >> >> >> >> You should use -pinoffset; see: >> >> >> >> - Docs and example: >> >> http://manual.gromacs.org/documentation/2016/user-guide/ >> >> mdrun-performance.html >> >> >> >> - More explanation on the thread pinning behavior on the old website: >> >> http://www.gromacs.org/Documentation/Acceleration_ >> >> and_parallelization#Pinning_threads_to_physical_cores >> >> >> >> Cheers, >> >> -- >> >> Szilárd >> >> >> >> >> >> On Wed, Sep 13, 2017 at 6:35 PM, gromacs query <gromacsqu...@gmail.com >> > >> >> wrote: >> >> > Sorry forgot to add; we thought the two jobs are using same GPU ids >> but >> >> > cuda visible devices show both jobs are using different ids (0,1 and >> 2,3) >> >> > >> >> > - >> >> > J >> >> > >> >> > On Wed, Sep 13, 2017 at 5:33 PM, gromacs query < >> gromacsqu...@gmail.com> >> >> > wrote: >> >> > >> >> >> Hi All, >> >> >> >> >> >> I have some issues with gromacs performance. There are many nodes >> and >> >> each >> >> >> node has number of gpus and the batch process is controlled by >> slurm. >> >> >> Although I get good performance with some settings of number of >> gpus and >> >> >> nprocs but when I submit same job twice on the same node then the >> >> >> performance is reduced drastically. e.g >> >> >> >> >> >> For 2 GPUs I get 300 ns per day when there is no other job running >> on >> >> the >> >> >> node. When I submit same job twice on the same node & at the same >> time, >> >> I >> >> >> get only 17 ns/day for both the jobs. I am using this: >> >> >> >> >> >> mpirun -np 4 gmx_mpi mdrun -deffnm test -ntomp 2 -maxh 0.12 >> >> >> >> >> >> Any suggestions highly appreciated. >> >> >> >> >> >> Thanks >> >> >> >> >> >> Jiom >> >> >> >> >> > -- >> >> > Gromacs Users mailing list >> >> > >> >> > * Please search the archive at http://www.gromacs.org/ >> >> Support/Mailing_Lists/GMX-Users_List before posting! >> >> > >> >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> > >> >> > * For (un)subscribe requests visit >> >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users >> or >> >> send a mail to gmx-users-requ...@gromacs.org. >> >> -- >> >> Gromacs Users mailing list >> >> >> >> * Please search the archive at http://www.gromacs.org/ >> >> Support/Mailing_Lists/GMX-Users_List before posting! >> >> >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> >> >> * For (un)subscribe requests visit >> >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> >> send a mail to gmx-users-requ...@gromacs.org. >> > -- >> > Gromacs Users mailing list >> > >> > * Please search the archive at http://www.gromacs.org/Support >> /Mailing_Lists/GMX-Users_List before posting! >> > >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> > >> > * For (un)subscribe requests visit >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> send a mail to gmx-users-requ...@gromacs.org. >> -- >> Gromacs Users mailing list >> >> * Please search the archive at http://www.gromacs.org/Support >> /Mailing_Lists/GMX-Users_List before posting! >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> * For (un)subscribe requests visit >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> send a mail to gmx-users-requ...@gromacs.org. >> > > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.