Re: [gmx-users] why it is so slow in Blue gene?
On 25/04/2012 3:24 PM, Albert wrote: hello: it is blue gene P. And the gromacs is single precision in the cluster. Getting Loaded...And the administrator told me that I have to use the multiples of 32 in the bg_size parameter. The number specified in -np should be 4 times bg_size. Yes, but your system is too small to make use of 128 processors. Also, get rid of -launch and -nt from your command line, since they do nothing. It is even slower than my own workstation with 16 core. here is the log file I get: No, that's the stdout file. Look at the end of the .log file. -log Reading file npt_01.tpr, VERSION 4.5.5 (single precision) Loaded with Money Will use 112 particle-particle and 16 PME only nodes This is guaranteed to lead to woeful performance with your .mdp settings, but you will have to look towards the beginning of the .log file to find out why mdrun selected this. Odds are good that your system size is so small that the minimum particle-particle cell size (constrained by rcoulomb) doesn't give mdrun any good options that use all the processors. You'd likely get better raw performance with twice the number of atoms or half the number of processors. Mark This is a guess, check the performance at the end of the log file Making 3D domain decomposition 4 x 4 x 7 starting mdrun 'GRowing Old MAkes el Chrono Sweat' 50 steps,500.0 ps. step 0 vol 0.64! imb F 16% pme/F 0.22 step 100, will finish Wed Apr 25 18:28:06 2012 vol 0.65! imb F 17% pme/F 0.21 step 200, will finish Wed Apr 25 18:09:54 2012 vol 0.67! imb F 18% pme/F 0.21 step 300, will finish Wed Apr 25 18:03:12 2012 vol 0.69! imb F 18% pme/F 0.21 step 400, will finish Wed Apr 25 17:58:25 2012 vol 0.67! imb F 19% pme/F 0.21 step 500, will finish Wed Apr 25 17:55:26 2012 vol 0.68! imb F 19% pme/F 0.22 step 600, will finish Wed Apr 25 17:53:31 2012 vol 0.68! imb F 19% pme/F 0.22 step 700, will finish Wed Apr 25 17:51:57 2012 vol 0.68! imb F 19% pme/F 0.22 step 800, will finish Wed Apr 25 17:50:32 2012 vol 0.68! imb F 20% pme/F 0.22 step 900, will finish Wed Apr 25 17:49:14 2012 vol 0.67! imb F 21% pme/F 0.22 step 1000, will finish Wed Apr 25 17:48:13 2012 vol 0.68! imb F 20% pme/F 0.22 step 1100, will finish Wed Apr 25 17:47:28 2012 vol 0.67! imb F 21% pme/F 0.22 step 1200, will finish Wed Apr 25 17:46:50 2012 vol 0.67! imb F 21% pme/F 0.22 step 1300, will finish Wed Apr 25 17:46:15 2012 On 04/24/2012 06:01 PM, Hannes Loeffler wrote: On Tue, 24 Apr 2012 15:42:15 +0200 Albertmailmd2...@gmail.com wrote: hello: I am running a 60,000 atom system with 128 core in a blue gene cluster. and it is only 1ns/day here is the script I used for You don't give any information what exact system that is (L/P/Q?), if you run single or double precision and what force field you are using. But for a similar sized system using a united atom force field in single precision we find about 4 ns/day on a BlueGene/P (see our benchmarking reports on http://www.stfc.ac.uk/CSE/randd/cbg/Benchmark/25241.aspx). I would expect a run with the CHARMM 27 force field in double precision to be roughly 3 times slower. We found scaling to 128 cores to be reasonably good. Also, check our report for problems when compiling with higher optimisation. Hannes. -- gmx-users mailing listgmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
[gmx-users] why it is so slow in Blue gene?
hello: I am running a 60,000 atom system with 128 core in a blue gene cluster. and it is only 1ns/day here is the script I used for submitting jobs: # @ job_name = gmx_test # @ class = kdm-large # @ error = gmx_test.err # @ output = gmx_test.out # @ wall_clock_limit = 00:20:00 # @ job_type = bluegene # @ bg_size = 32 # @ queue mpirun -exe /opt/gromacs/4.5.5/bin/mdrun_mpi_bg -args -nosum -dlb yes -v -s npt _01.tpr -o npt_01.trr -cpo npt_01.cpt -g npt_01.log -launch -nt -mode VN -np 128 here is my npt.mdp title= NPT-01 cpp = /usr/bin/cpp include = define = -DPOSRES -DPOSRES_POPE_HEAD integrator = md dt = 0.001 nsteps = 500 nstxout = 10 nstvout = 10 nstlog = 10 nstenergy= 5 nstxtcout= 5 xtc_grps = energygrps = Protein SOL ION nstcalcenergy= 10 nstlist = 10 nstcomm = 10 comm_mode= Linear comm-grps= Protein_POPEWater_and_ions ns_type = grid rlist= 1.2 rlistlong = 1.4 vdwtype = Switch rvdw = 1.2 rvdw_switch = 0.8 coulombtype = pme rcoulomb = 1.2 rcoulomb_switch = 0.0 fourierspacing = 0.15 pme_order = 6 DispCorr = no tcoupl = V-rescale ;nose-hoover nhchainlength= 1 tc-grps = Protein_POPEWater_and_ions tau_t= 0.1 0.1 ref_t= 310 310 Pcoupl = berendsen;parrinello-rahman Pcoupltype = semiisotropic tau_p= 1.0 compressibility = 4.5e-5 4.5e-5 ref_p= 1.0 1.0 pbc = xyz refcoord_scaling = com gen_vel = no optimize_fft = no constraints = hbonds constraint_algorithm = Lincs Does anybody have any advices? thank you very much -- gmx-users mailing listgmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
Re: [gmx-users] why it is so slow in Blue gene?
On 4/24/12 9:42 AM, Albert wrote: hello: I am running a 60,000 atom system with 128 core in a blue gene cluster. and it is only 1ns/day here is the script I used for submitting jobs: # @ job_name = gmx_test # @ class = kdm-large # @ error = gmx_test.err # @ output = gmx_test.out # @ wall_clock_limit = 00:20:00 # @ job_type = bluegene # @ bg_size = 32 # @ queue mpirun -exe /opt/gromacs/4.5.5/bin/mdrun_mpi_bg -args -nosum -dlb yes -v -s npt _01.tpr -o npt_01.trr -cpo npt_01.cpt -g npt_01.log -launch -nt -mode VN -np 128 here is my npt.mdp title = NPT-01 cpp = /usr/bin/cpp include = define = -DPOSRES -DPOSRES_POPE_HEAD integrator = md dt = 0.001 nsteps = 500 nstxout = 10 nstvout = 10 nstlog = 10 nstenergy = 5 nstxtcout = 5 xtc_grps = energygrps = Protein SOL ION nstcalcenergy = 10 nstlist = 10 nstcomm = 10 comm_mode = Linear comm-grps = Protein_POPE Water_and_ions ns_type = grid rlist = 1.2 rlistlong = 1.4 vdwtype = Switch rvdw = 1.2 rvdw_switch = 0.8 coulombtype = pme rcoulomb = 1.2 rcoulomb_switch = 0.0 fourierspacing = 0.15 pme_order = 6 DispCorr = no tcoupl = V-rescale ;nose-hoover nhchainlength = 1 tc-grps = Protein_POPE Water_and_ions tau_t = 0.1 0.1 ref_t = 310 310 Pcoupl = berendsen ;parrinello-rahman Pcoupltype = semiisotropic tau_p = 1.0 compressibility = 4.5e-5 4.5e-5 ref_p = 1.0 1.0 pbc = xyz refcoord_scaling = com gen_vel = no optimize_fft = no constraints = hbonds constraint_algorithm = Lincs Does anybody have any advices? The end of the log file will print information about where performance may have been lost. For 60,000 atoms I would think that 128 cores is too many; you're sacrificing performance to communication overhead. A good ballpark is 1000 atoms/core. A few quick benchmark calculations should give you a better idea on the setup for optimal performance. -Justin -- Justin A. Lemkul Ph.D. Candidate ICTAS Doctoral Scholar MILES-IGERT Trainee Department of Biochemistry Virginia Tech Blacksburg, VA jalemkul[at]vt.edu | (540) 231-9080 http://www.bevanlab.biochem.vt.edu/Pages/Personal/justin -- gmx-users mailing listgmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
Re: [gmx-users] why it is so slow in Blue gene?
On Tue, 24 Apr 2012 15:42:15 +0200 Albert mailmd2...@gmail.com wrote: hello: I am running a 60,000 atom system with 128 core in a blue gene cluster. and it is only 1ns/day here is the script I used for You don't give any information what exact system that is (L/P/Q?), if you run single or double precision and what force field you are using. But for a similar sized system using a united atom force field in single precision we find about 4 ns/day on a BlueGene/P (see our benchmarking reports on http://www.stfc.ac.uk/CSE/randd/cbg/Benchmark/25241.aspx). I would expect a run with the CHARMM 27 force field in double precision to be roughly 3 times slower. We found scaling to 128 cores to be reasonably good. Also, check our report for problems when compiling with higher optimisation. Hannes. -- Scanned by iCritical. -- gmx-users mailing listgmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
