Hi Szilard, Thank you very much for your suggestions.
>Actually, I was jumping to conclusions too early, as you mentioned AMD >"cluster", I assumed you must have 12-16-core Opteron CPUs. If you >have an 8-core (desktop?) AMD CPU, than you may not need to run more >than one rank per GPU. Yes, we do have independent clusters of AMD, AMD opteron, Intel Corei7. All nodes of three clusters are installed with (at least) 1 GPU card. I have run the same test on these three clusters. Let's focus on a basic scaling issue: One GPU v.s Two GPUs within the same node of 8-core AMD cpu. Using 1 GPU, we can have a performance of ~32 ns/day. Using two GPU, we gain not much more ( ~38.5 ns/day ). It is about ~20% more performance. However, this is not really true because in some tests, I also saw only 2-5% more, which really surprised me. As you can see, this test was made on the same node regardless of networking. Can the performance be improved say 50% more when 2 GPUs are used on a general task ? If yes, how ? >Indeed, as Richard pointed out, I was asking for *full* logs, these >summaries can't tell much, the table above the summary entitled "R E A >L C Y C L E A N D T I M E A C C O U N T I N G" as well as >other reported information across the log file is what I need to make >an assessment of your simulations' performance. Please see below. >>However, in your case I suspect that the >>bottleneck is multi-threaded scaling on the AMD CPUs and you should >>probably decrease the number of threads per MPI rank and share GPUs >>between 2-4 ranks. After I test all three clusters, I found it may NOT be an issue of AMD cpus. Intel cpus has the SAME scaling issue. However, I am curious as to how you justify the setup of 2-4 ranks sharing GPUs ? Can you please explain it a bit more ? >You could try running >mpirun -np 4 mdrun -ntomp 2 -gpu_id 0011 >but I suspect this won't help because your scaling issue Your guess is correct but why is that ? it is worse. The more nodes are involved in a task, the performance is worse. >> in my >>experience even reaction field runs don't scale across nodes with 10G >>ethernet if you have more than 4-6 ranks per node trying to >>communicate (let alone with PME). What dose it mean " let alone with PME" ? how to do so ? by mdrun ? I do know " mdrun -npme to specify PME process. Thank you. Dwey ### One GPU #### R E A L C Y C L E A N D T I M E A C C O U N T I N G Computing: Nodes Th. Count Wall t (s) G-Cycles % ----------------------------------------------------------------------------- Neighbor search 1 8 100001 431.817 13863.390 1.6 Launch GPU ops. 1 8 5000001 472.906 15182.556 1.7 Force 1 8 5000001 1328.611 42654.785 4.9 PME mesh 1 8 5000001 11561.327 371174.090 42.8 Wait GPU local 1 8 5000001 6888.008 221138.111 25.5 NB X/F buffer ops. 1 8 9900001 1216.499 39055.455 4.5 Write traj. 1 8 1030 12.741 409.039 0.0 Update 1 8 5000001 1696.358 54461.226 6.3 Constraints 1 8 5000001 1969.726 63237.647 7.3 Rest 1 1458.820 46835.133 5.4 ----------------------------------------------------------------------------- Total 1 27036.812 868011.431 100.0 ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- PME spread/gather 1 8 10000002 6975.086 223933.739 25.8 PME 3D-FFT 1 8 10000002 3928.259 126115.976 14.5 PME solve 1 8 5000001 636.488 20434.327 2.4 ----------------------------------------------------------------------------- GPU timings ----------------------------------------------------------------------------- Computing: Count Wall t (s) ms/step % ----------------------------------------------------------------------------- Pair list H2D 100001 43.435 0.434 0.2 X / q H2D 5000001 567.168 0.113 2.8 Nonbonded F kernel 4000000 14174.316 3.544 70.8 Nonbonded F+ene k. 900000 4314.438 4.794 21.5 Nonbonded F+ene+prune k. 100001 572.370 5.724 2.9 F D2H 5000001 358.120 0.072 1.8 ----------------------------------------------------------------------------- Total 20029.846 4.006 100.0 ----------------------------------------------------------------------------- Force evaluation time GPU/CPU: 4.006 ms/2.578 ms = 1.554 For optimal performance this ratio should be close to 1! NOTE: The GPU has >20% more load than the CPU. This imbalance causes performance loss, consider using a shorter cut-off and a finer PME grid. Core t (s) Wall t (s) (%) Time: 216205.510 27036.812 799.7 7h30:36 (ns/day) (hour/ns) Performance: 31.956 0.751 ### Two GPUs ##### R E A L C Y C L E A N D T I M E A C C O U N T I N G Computing: Nodes Th. Count Wall t (s) G-Cycles % ----------------------------------------------------------------------------- Domain decomp. 2 4 100000 339.490 10900.191 1.5 DD comm. load 2 4 49989 0.262 8.410 0.0 Neighbor search 2 4 100001 481.583 15462.464 2.2 Launch GPU ops. 2 4 10000002 579.283 18599.358 2.6 Comm. coord. 2 4 4900000 523.096 16795.351 2.3 Force 2 4 5000001 1545.584 49624.951 6.9 Wait + Comm. F 2 4 5000001 821.740 26384.083 3.7 PME mesh 2 4 5000001 11097.880 356326.030 49.5 Wait GPU nonlocal 2 4 5000001 1001.868 32167.550 4.5 Wait GPU local 2 4 5000001 8.613 276.533 0.0 NB X/F buffer ops. 2 4 19800002 1061.238 34073.781 4.7 Write traj. 2 4 1025 5.681 182.419 0.0 Update 2 4 5000001 1692.233 54333.503 7.6 Constraints 2 4 5000001 2316.145 74365.788 10.3 Comm. energies 2 4 1000001 15.802 507.373 0.1 Rest 2 908.383 29165.963 4.1 ----------------------------------------------------------------------------- Total 2 22398.880 719173.747 100.0 ----------------------------------------------------------------------------- ----------------------------------------------------------------------------- PME redist. X/F 2 4 10000002 1519.288 48780.654 6.8 PME spread/gather 2 4 10000002 5398.693 173338.936 24.1 PME 3D-FFT 2 4 10000002 2798.482 89852.482 12.5 PME 3D-FFT Comm. 2 4 10000002 947.033 30406.937 4.2 PME solve 2 4 5000001 420.667 13506.611 1.9 ----------------------------------------------------------------------------- Core t (s) Wall t (s) (%) Time: 178961.450 22398.880 799.0 6h13:18 (ns/day) (hour/ns) Performance: 38.573 0.622 -- View this message in context: http://gromacs.5086.x6.nabble.com/mdrun-on-8-core-AMD-GTX-TITAN-was-Re-gmx-users-Re-Gromacs-4-6-on-two-Titans-GPUs-tp5012330p5012391.html Sent from the GROMACS Users Forum mailing list archive at Nabble.com. -- gmx-users mailing list gmx-users@gromacs.org http://lists.gromacs.org/mailman/listinfo/gmx-users * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/Search before posting! * Please don't post (un)subscribe requests to the list. Use the www interface or send it to gmx-users-requ...@gromacs.org. * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists