Hi Szila I tried more more OpenMP threads, (-ntomp 4), however, the performance dropped down drastically so that an NVT simulation which just took 3 hours to be finished in "-ntomp 1", now takes more thank 16 hours!
Cheers, Alex On Mon, Aug 15, 2016 at 6:34 PM, Szilárd Páll <pall.szil...@gmail.com> wrote: > Hi, > > Although I don't know what exactly is the system you are simulating, > one thing is clear: you're pushing the parallelization limit with > - 200 atoms/core > - likely "concentrated" free energy interactions. > The former that alone will make the run very sensitive to load > imbalance and the latter makes imbalance even worse as the very > expensive free energy interactions likely all fall in a few domains > (unless your 8 perturbed atoms are scattered). > > There is not much you can do except the what I previously suggested > (trying more OpenMP threads e.g. 2-4 or simply use less cores). If you > have the possibility, using some hardware with fewer and faster cores > (and perhaps a GPU) will also be much more suitable than this 128-core > AMD node. > > Cheers, > -- > Szilárd > > > On Mon, Aug 15, 2016 at 4:01 PM, Alexander Alexander > <alexanderwie...@gmail.com> wrote: > > Hi Szilárd, > > > > Thanks for your response, please find below a link containing required > > files.log files. > > > > https://drive.google.com/file/d/0B_CbyhnbKqQDc2FaeWxITWxqdDg > /view?usp=sharing > > > > Thanks, > > Cheers, > > Alex > > > > On Mon, Aug 15, 2016 at 2:52 PM, Szilárd Páll <pall.szil...@gmail.com> > > wrote: > > > >> Hi, > >> > >> Please post full logs; what you cut out of the file will often miss > >> information needed to diagnose your issues. > >> > >> At first sight it seems that you simply have an imbalanced system. Not > >> sure about the source of the imbalance and without knowing more about > >> your system/setup and how is it decomposed what I can suggest is to: > >> try other decomposition schemes or simply less decomposition (use more > >> threads or less cores). > >> > >> Additionally you also have a pretty bad PP-PME load balance, but > >> that's likely going to get better if you get you PP performance > >> better. > >> > >> Cheers, > >> -- > >> Szilárd > >> > >> > >> On Sun, Aug 14, 2016 at 3:23 PM, Alexander Alexander > >> <alexanderwie...@gmail.com> wrote: > >> > Dear gromacs user, > >> > > >> > My free energy calculation works well, however, I am loosing around > 56.5 > >> % > >> > of the available CPU time as stated in my log file which is really > >> > considerable. The problem is due to the load imbalance and domain > >> > decomposition, but I have no idea to improve it, below is the very > end of > >> > my log file and I would be so appreciated if you could help avoid > this. > >> > > >> > > >> > D O M A I N D E C O M P O S I T I O N S T A T I S T I C S > >> > > >> > av. #atoms communicated per step for force: 2 x 115357.4 > >> > av. #atoms communicated per step for LINCS: 2 x 2389.1 > >> > > >> > Average load imbalance: 285.9 % > >> > Part of the total run time spent waiting due to load imbalance: 56.5 > % > >> > Steps where the load balancing was limited by -rdd, -rcon and/or > -dds: > >> X 2 > >> > % Y 2 % Z 2 % > >> > Average PME mesh/force load: 0.384 > >> > Part of the total run time spent waiting due to PP/PME imbalance: > 14.5 % > >> > > >> > NOTE: 56.5 % of the available CPU time was lost due to load imbalance > >> > in the domain decomposition. > >> > > >> > NOTE: 14.5 % performance was lost because the PME ranks > >> > had less work to do than the PP ranks. > >> > You might want to decrease the number of PME ranks > >> > or decrease the cut-off and the grid spacing. > >> > > >> > > >> > R E A L C Y C L E A N D T I M E A C C O U N T I N G > >> > > >> > On 96 MPI ranks doing PP, and > >> > on 32 MPI ranks doing PME > >> > > >> > Computing: Num Num Call Wall time > Giga-Cycles > >> > Ranks Threads Count (s) total sum > % > >> > ------------------------------------------------------------ > >> ----------------- > >> > Domain decomp. 96 1 175000 242.339 53508.472 > >> 0.5 > >> > DD comm. load 96 1 174903 9.076 2003.907 > >> 0.0 > >> > DD comm. bounds 96 1 174901 27.054 5973.491 > >> 0.1 > >> > Send X to PME 96 1 7000001 44.342 9790.652 > >> 0.1 > >> > Neighbor search 96 1 175001 251.994 55640.264 > >> 0.6 > >> > Comm. coord. 96 1 6825000 1521.009 335838.747 > >> 3.4 > >> > Force 96 1 7000001 7001.990 1546039.264 > >> 15.5 > >> > Wait + Comm. F 96 1 7000001 10761.296 2376093.759 > >> 23.8 > >> > PME mesh * 32 1 7000001 11796.344 868210.788 > >> 8.7 > >> > PME wait for PP * 22135.752 1629191.096 > >> 16.3 > >> > Wait + Recv. PME F 96 1 7000001 393.117 86800.265 > >> 0.9 > >> > NB X/F buffer ops. 96 1 20650001 132.713 29302.991 > >> 0.3 > >> > COM pull force 96 1 7000001 165.613 36567.368 > >> 0.4 > >> > Write traj. 96 1 7037 55.020 12148.457 > >> 0.1 > >> > Update 96 1 14000002 140.972 31126.607 > >> 0.3 > >> > Constraints 96 1 14000002 12871.236 2841968.551 > >> 28.4 > >> > Comm. energies 96 1 350001 261.976 57844.219 > >> 0.6 > >> > Rest 52.349 11558.715 > >> 0.1 > >> > ------------------------------------------------------------ > >> ----------------- > >> > Total 33932.096 9989607.639 > >> 100.0 > >> > ------------------------------------------------------------ > >> ----------------- > >> > (*) Note that with separate PME ranks, the walltime column actually > sums > >> to > >> > twice the total reported, but the cycle count total and % are > >> correct. > >> > ------------------------------------------------------------ > >> ----------------- > >> > Breakdown of PME mesh computation > >> > ------------------------------------------------------------ > >> ----------------- > >> > PME redist. X/F 32 1 21000003 2334.608 171827.143 > >> 1.7 > >> > PME spread/gather 32 1 28000004 3640.870 267967.972 > >> 2.7 > >> > PME 3D-FFT 32 1 28000004 1587.105 116810.882 > >> 1.2 > >> > PME 3D-FFT Comm. 32 1 56000008 4066.097 299264.666 > >> 3.0 > >> > PME solve Elec 32 1 14000002 148.284 10913.728 > >> 0.1 > >> > ------------------------------------------------------------ > >> ----------------- > >> > > >> > Core t (s) Wall t (s) (%) > >> > Time: 4341204.790 33932.096 12793.8 > >> > 9h25:32 > >> > (ns/day) (hour/ns) > >> > Performance: 35.648 0.673 > >> > Finished mdrun on rank 0 Sat Aug 13 23:45:45 2016 > >> > > >> > Thanks, > >> > Regards, > >> > Alex > >> > -- > >> > Gromacs Users mailing list > >> > > >> > * Please search the archive at http://www.gromacs.org/ > >> Support/Mailing_Lists/GMX-Users_List before posting! > >> > > >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >> > > >> > * For (un)subscribe requests visit > >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > >> send a mail to gmx-users-requ...@gromacs.org. > >> -- > >> Gromacs Users mailing list > >> > >> * Please search the archive at http://www.gromacs.org/ > >> Support/Mailing_Lists/GMX-Users_List before posting! > >> > >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >> > >> * For (un)subscribe requests visit > >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > >> send a mail to gmx-users-requ...@gromacs.org. > > -- > > Gromacs Users mailing list > > > > * Please search the archive at http://www.gromacs.org/Support > /Mailing_Lists/GMX-Users_List before posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- > Gromacs Users mailing list > > * Please search the archive at http://www.gromacs.org/Support > /Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.