Chris, I can suggest two possible tweaks, these won't do miracles, but may give you a bit better performance.
* icc is better than gcc at optimizing the naiive free energy kernel code. I observed in the past up to 1.5x faster free energy kernel performance with icc 15 vs gcc 4.9 or so. * The default nstlist heuristic/suggestion assumes fast non-bondeds. In your case, depending on how long is the list buffer with the nstlist you use (25?), you may be able to gain a bit of performance by shifting load back to the search with a smaller nstlist, e.g. 10. These two combined could give in best case 2x, I guess. What I find slightly unusual in the logs you posted on redmine is the 3-4x slowdown in PME and constraints (I'd expect ~2x in PME and less in constraints), but it could be that this too is simply because most interaction in the system are perturbed which trigger non-optimized code-paths. Cheers, -- Szilárd On Wed, Mar 30, 2016 at 3:05 PM, Mark Abraham <mark.j.abra...@gmail.com> wrote: > Hi, > > Yeah, unfortunately Michael's pretty much right - the free-energy kernel is > currently that cousin nobody talks about (or to). It's essentially > unchanged since GROMACS 4.0 days, except that the Verlet scheme has some > kludge so that it can call the same kernel that the group scheme used to > call. But it has none of the optimizations and also some simplifying > pessimizations. The result is highly unsuitable for Chris's use case, but > not horrible for the more normal case of perturbing a small part of a > system. For now, I can only suggest trying a run with more than one OpenMP > thread per rank, but there's nothing in the log file snippets that fills me > with any hope that it would be noticeably faster. > > We have a pile of infrastructure built and in the works that will lead to > being able to offer similarly optimized free-energy kernels, but they won't > see the light of day until next year, I'm afraid. A set of sample .tprs at > Redmine 742 would be most welcome, however - it's very good for us to know > when/that we're optimizing a workflow someone actually wants to run, but > currently has reason to avoid. > > Mark > > On Wed, Mar 30, 2016 at 8:05 AM Michael Shirts <mrshi...@gmail.com> wrote: > > > Hi, Chris: I'm pretty sure that it's because the nonbonded free > > energies are much slower than the standard free energies. You state: > > > > > I took a look at gmxlib/nonbonded/nb_free_energy.c in v.5.1.2, but I > was > > unable to find a function called "gmx_waste_time_here()" and beyond that > I > > was out of my depth. > > > > But it's much more the fact that the non-free energy nonbondeds are > > SUPER optimized. > > > > I don't see a particularly viable way around this for now. The only > > thing I can think of splitting the neighborlists into two force calls, > > and scaling the forces and energies coming out of those. That would > > be a huge pain. > > > > On Tue, Mar 29, 2016 at 9:50 PM, Christopher Neale > > <chris.ne...@alum.utoronto.ca> wrote: > > > Dear Users: > > > > > > I am trying to do some Hamiltonian replica exchange (H-REMD) in gromacs > > 5.1.2 and am running up against really large slowdowns when decoupling a > > large number of atoms. I am decoupling 5360 atoms out of the 15520 atoms > in > > my system. The goal is not to get a PMF, but to enhance sampling using > the > > REST approach to partially decouple lipids in a bilayer. This approach > > enhances lipid relaxation times ( > > http://pubs.acs.org/doi/pdf/10.1021/ct500305u ) though the authors of > > that paper modified the gromacs code to do their own H-REMD in order to > > avoid the really slow speed they also got when decoupling lots of atoms > via > > the free energy code. > > > > > > I have already posted details here > http://redmine.gromacs.org/issues/742 > > , which includes .mdp options and some timing output. I compare the > timing > > output to a standard temperature REMD (T-REMD) run. For my usage, the > > slowdown is about 12x for H-REMD vs. T-REMD. > > > > > > I am motivated to find a solution within gromacs because the > alternative > > is to use gromacs 4.6.7 with plumed (or with the aforementioned modified > > code, which is also gromacs v4). Normally that would be a viable option, > > but I am using the charmm force field and the charmm TIP3P water and I > > would rather not give up the speed boost that I see in gromacs v5.1.2, > > which allows the use of the verlet cutoff scheme and has been tested and > > shown to give the correct reproduction of charmm forces (vs. the forces > one > > would get using the charmm simulation software). > > > > > > I took a look at gmxlib/nonbonded/nb_free_energy.c in v.5.1.2, but I > was > > unable to find a function called "gmx_waste_time_here()" and beyond that > I > > was out of my depth. > > > > > > Thank you for any pointers, > > > Chris. > > > > > > > > > -- > > > Gromacs Users mailing list > > > > > > * Please search the archive at > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > posting! > > > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > > > * For (un)subscribe requests visit > > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > send a mail to gmx-users-requ...@gromacs.org. > > -- > > Gromacs Users mailing list > > > > * Please search the archive at > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > send a mail to gmx-users-requ...@gromacs.org. > > > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.