Can you post a log, I have a hard time believing this was a driver causing it.
BTW, GROMACS uses the CPU heavily. If your rcoulomb gets tuned by >10% or so, you'll see much more gain by using a faster GPU (or you can crank up the CPU clock if you feel like adventurous and have a decent fan). -- Szilárd On Thu, Jul 28, 2016 at 8:04 PM, Vito Spadavecchio <spadavecc...@gmail.com> wrote: > Just as an update: I upgraded my cuda7.5 installation to cuda8, (keeping > the 367.27 driver), and am now getting ~150ns/day on the same system. This > is still ~1/2 of what I would expect given the benchmarks I've seen for the > 1080gtx, but I will take a ~10x increase in throughput for now! > > On Thu, Jul 28, 2016 at 9:10 AM, Szilárd Páll <pall.szil...@gmail.com> > wrote: > >> Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm >> increasing the GPU load by about 3x (see the table on line 550 of the >> log file you posted). >> I find it somewhat suspicious that your CPU seems so slow, but given >> that you have a GTX 1080, that may just be normal. >> -- >> Szilárd >> >> >> On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio >> <spadavecc...@gmail.com> wrote: >> > Just wanted to ping this and see if anyone had any ideas :) >> > >> > On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio < >> spadavecc...@gmail.com> >> > wrote: >> > >> >> My apologies! I meant to include the logfile as well. You can view it >> >> here: >> >> >> >> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0 >> >> >> >> Thanks! >> >> >> >> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalem...@vt.edu> wrote: >> >> >> >>> >> >>> >> >>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote: >> >>> >> >>>> Szilard >> >>>> >> >>>> You are correct; I misspoke. It appears that the GPU is doing >> >>>> *something*, >> >>>> but I it would seem it is severely under performing (near the point >> where >> >>>> its basically doing nothing). >> >>>> >> >>>> the entire log, not just parts, in particular we want to see the >> >>>> >> >>>>> header, perf table >> >>>>> >> >>>>> >> >>>> xyz@Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu >> >>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-: >> >>>> >> >>>> GROMACS is written by: >> >>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par >> >>>> Bjelkmar >> >>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian >> >>>> Fritsch >> >>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent >> >>>> Hindriksen >> >>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten >> >>>> Kutzner >> >>>> >> >>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter >> >>>> Meulenhoff >> >>>> Erik Marklund Teemu Murtola Szilard Pall Sander >> Pronk >> >>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons >> >>>> Sijbers >> >>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten >> Wolf >> >>>> and the project leaders: >> >>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel >> >>>> >> >>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands. >> >>>> Copyright (c) 2001-2015, The GROMACS development team at >> >>>> Uppsala University, Stockholm University and >> >>>> the Royal Institute of Technology, Sweden. >> >>>> check out http://www.gromacs.org for more information. >> >>>> >> >>>> GROMACS is free software; you can redistribute it and/or modify it >> >>>> under the terms of the GNU Lesser General Public License >> >>>> as published by the Free Software Foundation; either version 2.1 >> >>>> of the License, or (at your option) any later version. >> >>>> >> >>>> GROMACS: gmx mdrun, VERSION 5.1.2 >> >>>> Executable: /usr/local/gromacs/bin/gmx >> >>>> Data prefix: /usr/local/gromacs >> >>>> Command line: >> >>>> gmx mdrun -deffnm md_0_2 -nb gpu >> >>>> >> >>>> >> >>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1# >> >>>> >> >>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible >> GPU >> >>>> Hardware detected: >> >>>> CPU info: >> >>>> Vendor: GenuineIntel >> >>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz >> >>>> SIMD instructions most likely to fit this hardware: AVX2_256 >> >>>> SIMD instructions selected at GROMACS compile time: AVX2_256 >> >>>> GPU info: >> >>>> Number of GPUs detected: 1 >> >>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: >> >>>> compatible >> >>>> >> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) >> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >> >>>> >> >>>> Using 1 MPI thread >> >>>> Using 8 OpenMP threads >> >>>> >> >>>> 1 compatible GPU is present, with ID 0 >> >>>> 1 GPU auto-selected for this run. >> >>>> Mapping of GPU ID to the 1 PP rank in this node: 0 >> >>>> >> >>>> >> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# >> >>>> >> >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# >> >>>> >> >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# >> >>>> starting mdrun 'Protein in water' >> >>>> 500000 steps, 1000.0 ps. >> >>>> >> >>>> >> >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution >> >>>> >> >>>> >> >>>> heres a snapshot: >> >>>> >> >>>> # gpu pwr temp sm mem enc dec mclk pclk >> >>>> >> >>>>> # Idx W C % % % % MHz MHz >> >>>>> 0 117 51 56 3 0 0 4513 1847 >> >>>>> 0 115 52 54 3 0 0 4513 1847 >> >>>>> 0 117 52 47 3 0 0 4513 1847 >> >>>>> 0 101 52 52 3 0 0 4513 1848 >> >>>>> 0 119 53 54 3 0 0 4513 1847 >> >>>>> 0 115 53 54 3 0 0 4513 1848 >> >>>>> 0 106 54 54 3 0 0 4513 1848 >> >>>>> 0 118 54 54 3 0 0 4513 1849 >> >>>>> 0 94 54 52 3 0 0 4513 1848 >> >>>>> 0 110 55 52 3 0 0 4513 1849 >> >>>>> 0 116 55 53 3 0 0 4513 1854 >> >>>>> 0 126 55 51 3 0 0 4513 1854 >> >>>>> 0 119 56 51 3 0 0 4513 1855 >> >>>>> 0 116 56 52 3 0 0 4513 1855 >> >>>>> 0 116 56 53 3 0 0 4513 1855 >> >>>>> 0 120 57 53 3 0 0 4513 1855 >> >>>>> 0 113 57 54 3 0 0 4513 1855 >> >>>>> >> >>>>> >> >>>> and here's the tail of the performance in .log >> >>>> >> >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) >> >>>> >> >>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >> >>>>> Using 1 MPI thread >> >>>>> Using 8 OpenMP threads >> >>>>> 1 compatible GPU is present, with ID 0 >> >>>>> 1 GPU auto-selected for this run. >> >>>>> Mapping of GPU ID to the 1 PP rank in this node: 0 >> >>>>> >> >>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# >> >>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# >> >>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# >> >>>>> starting mdrun 'Protein in water' >> >>>>> 500000 steps, 1000.0 ps. >> >>>>> Writing final coordinates. >> >>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1# >> >>>>> Core t (s) Wall t (s) (%) >> >>>>> Time: 25285.344 3409.337 741.6 >> >>>>> 56:49 >> >>>>> (ns/day) (hour/ns) >> >>>>> Performance: 25.342 0.947 >> >>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain >> Beefheart) >> >>>>> >> >>>>> >> >>>>> As we can see, there is *something* running on the GPU (process id: >> >>>> 30784), >> >>>> but its never more than 135MiB of memory being used and only uses a >> very >> >>>> small portion of the gpus power. >> >>>> >> >>>> Any thoughts are greatly appreciated! >> >>>> >> >>>> >> >>> You cut out the part of the .log file that shows how mdrun actually >> spent >> >>> its time. This is perhaps the most valuable information. This is why >> >>> Szilard asked for you to share a complete .log file (upload it >> somewhere, >> >>> post a URL) so we can see all relevant details. >> >>> >> >>> -Justin >> >>> >> >>> >> >>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szil...@gmail.com >> > >> >>>> wrote: >> >>>> >> >>>> Hi, >> >>>>> >> >>>>> If mdrun claims it's using the GPU and you see no errors (and results >> >>>>> look reasonable), the GPU is likely being used. That nvidia-smi is >> not >> >>>>> showing it is admittedly strange. However, this truncated log output >> >>>>> does not confirm much, e.g. it does not show the performance table >> and >> >>>>> the final performance. >> >>>>> >> >>>>> Could you please run again and look at (and share if approapriate): >> >>>>> - the entire log, not just parts, in particular we want to see the >> >>>>> header, perf table >> >>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution >> >>>>> (this monitors the device continuously) >> >>>>> >> >>>>> Cheers, >> >>>>> >> >>>>> -- >> >>>>> Szilárd >> >>>>> >> >>>>> >> >>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio >> >>>>> <spadavecc...@gmail.com> wrote: >> >>>>> >> >>>>>> Hello >> >>>>>> >> >>>>>> After building GROMACS with GPU support and running some simple MD, >> I >> >>>>>> am >> >>>>>> only getting ~20ns/day (which is what I would expect to get out of >> my >> >>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which >> >>>>>> >> >>>>> should >> >>>>> >> >>>>>> even on moderately large systems, be getting ~100ns/day) >> >>>>>> >> >>>>>> I've tried launching a mdrun with with the following command: >> >>>>>> >> >>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> With the following in my md.mdp file: >> >>>>>> >> >>>>>> ; Neighborsearching >> >>>>>> >> >>>>>> cutoff-scheme = Verlet >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> The top of the log file for the mdrun notes: >> >>>>>> >> >>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible >> GPU >> >>>>>> >> >>>>>>> Hardware detected: >> >>>>>>> CPU info: >> >>>>>>> Vendor: GenuineIntel >> >>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz >> >>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256 >> >>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256 >> >>>>>>> GPU info: >> >>>>>>> Number of GPUs detected: 1 >> >>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: >> >>>>>>> compatible >> >>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision) >> >>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >> >>>>>>> Using 1 MPI thread >> >>>>>>> Using 8 OpenMP threads >> >>>>>>> 1 GPU user-selected for this run. >> >>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0 >> >>>>>>> >> >>>>>>> >> >>>>>> The output of *nvidia-smi *on my machine is: >> >>>>>> >> >>>>>> hu Jul 14 11:25:04 2016 >> >>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> +-----------------------------------------------------------------------------+ >> >>>>> >> >>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27 >> >>>>>>> | >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> |-------------------------------+----------------------+----------------------+ >> >>>>> >> >>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile >> >>>>>>> >> >>>>>> Uncorr. >> >>>>> >> >>>>>> ECC | >> >>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util >> >>>>>>> >> >>>>>> Compute >> >>>>> >> >>>>>> M. | >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> |===============================+======================+======================| >> >>>>> >> >>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On | >> >>>>>>> N/A | >> >>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3% >> >>>>>>> Default | >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> +-------------------------------+----------------------+----------------------+ >> >>>>> >> >>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> +-----------------------------------------------------------------------------+ >> >>>>> >> >>>>>> | Processes: >> GPU >> >>>>>>> Memory | >> >>>>>>> | GPU PID Type Process name >> >>>>>>> Usage >> >>>>>>> | >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> |=============================================================================| >> >>>>> >> >>>>>> | 0 1035 G /usr/lib/xorg/Xorg >> >>>>>>> 424MiB | >> >>>>>>> | 0 2132 G compiz >> >>>>>>> 103MiB | >> >>>>>>> | 0 5296 G ...s-passed-by-fd >> --v8-snapshot-passed-by-fd >> >>>>>>> 59MiB | >> >>>>>>> | 0 14079 G ...s-passed-by-fd >> --v8-snapshot-passed-by-fd >> >>>>>>> 31MiB | >> >>>>>>> | 0 14234 G ...ing<BrotliEncoding >> --disable-features=Doc >> >>>>>>> 108MiB | >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>> >> +-----------------------------------------------------------------------------+ >> >>>>> >> >>>>>> >> >>>>>> >> >>>>>> and output of *nvcc --version* is: >> >>>>>> >> >>>>>> nvcc: NVIDIA (R) Cuda compiler driver >> >>>>>> >> >>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation >> >>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015 >> >>>>>>> Cuda compilation tools, release 7.5, V7.5.17 >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> Does anyone have any ideas what the problem might be? >> >>>>>> >> >>>>>> Thanks! >> >>>>>> -- >> >>>>>> Gromacs Users mailing list >> >>>>>> >> >>>>>> * Please search the archive at >> >>>>>> >> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> >>>>> posting! >> >>>>> >> >>>>>> >> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >>>>>> >> >>>>>> * For (un)subscribe requests visit >> >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users >> or >> >>>>>> >> >>>>> send a mail to gmx-users-requ...@gromacs.org. >> >>>>> -- >> >>>>> Gromacs Users mailing list >> >>>>> >> >>>>> * Please search the archive at >> >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> >>>>> posting! >> >>>>> >> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >>>>> >> >>>>> * For (un)subscribe requests visit >> >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users >> or >> >>>>> send a mail to gmx-users-requ...@gromacs.org. >> >>>>> >> >>>>> >> >>> -- >> >>> ================================================== >> >>> >> >>> Justin A. Lemkul, Ph.D. >> >>> Ruth L. Kirschstein NRSA Postdoctoral Fellow >> >>> >> >>> Department of Pharmaceutical Sciences >> >>> School of Pharmacy >> >>> Health Sciences Facility II, Room 629 >> >>> University of Maryland, Baltimore >> >>> 20 Penn St. >> >>> Baltimore, MD 21201 >> >>> >> >>> jalem...@outerbanks.umaryland.edu | (410) 706-7441 >> >>> http://mackerell.umaryland.edu/~jalemkul >> >>> >> >>> ================================================== >> >>> >> >>> -- >> >>> Gromacs Users mailing list >> >>> >> >>> * Please search the archive at >> >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> >>> posting! >> >>> >> >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >>> >> >>> * For (un)subscribe requests visit >> >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> >>> send a mail to gmx-users-requ...@gromacs.org. >> >>> >> >> >> >> >> > -- >> > Gromacs Users mailing list >> > >> > * Please search the archive at >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> posting! >> > >> > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> > >> > * For (un)subscribe requests visit >> > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> send a mail to gmx-users-requ...@gromacs.org. >> -- >> Gromacs Users mailing list >> >> * Please search the archive at >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> posting! >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> * For (un)subscribe requests visit >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> send a mail to gmx-users-requ...@gromacs.org. >> > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a > mail to gmx-users-requ...@gromacs.org. -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.
