After much experimentation I managed to run mpirun -np 48 gmx_mpi mdrun -ntomp 1 -v -deffnm memb_prod1 -multidir 1 2 3 4 -replex 1000
on a single node at 27 ns/day. This scaled up for 73 replicas on my 190 000 atom system ( using the same logic -np num_sims*12) on our gadi cluster in australia. I will soon see if i can get away with fewer replicas. Thanks for your help. Miro On Sun, Mar 29, 2020 at 9:04 PM Benson Muite <benson_mu...@emailplus.org> wrote: > > > On Sun, Mar 29, 2020, at 4:55 AM, Miro Astore wrote: > > Hi everybody. I've been experimenting with REMD for my system running > > on 48 cores with 4 gpus (I will need to scale up to 73 replicas > > because this is a complicated system with many DOF I'm open to being > > told this is all a silly idea). > > > > My run configuration is > > mpirun -np 4 --map-by numa gmx_mpi mdrun -cpi memb_prod1.cpt -ntomp 11 > > -v -deffnm memb_prod1 -multidir 1 2 3 4 -replex 1000 > > > > the best I can squeeze out of this is 9ns/day. In a non-replica > > simulation I can hit 50ns/day with a single GPU and 12 cores. > > What happens for a small number of replicas? > > > > > Looking at my accounting, for a single replica 52% of time is being > > spent on the "Force" category with 92% of my Mflops going into NxN > > Ewald Elec. + LJ [F] > > > > I'm wondering what I could do to reduce this bottle neck if anything. > > Do you have access to more hardware? There area number of HPC centers in > Australia. > > > > > Thank you. > > -- > > Miro A. Astore (he/him) > > PhD Candidate | Computational Biophysics > > Office 434 A28 School of Physics > > University of Sydney > > -- > > Gromacs Users mailing list > > > > * Please search the archive at > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > send a mail to gmx-users-requ...@gromacs.org. > > > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a > mail to gmx-users-requ...@gromacs.org. -- Miro A. Astore (he/him) PhD Candidate | Computational Biophysics Office 434 A28 School of Physics University of Sydney -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.