Just as an update: I upgraded my cuda7.5 installation to cuda8, (keeping the 367.27 driver), and am now getting ~150ns/day on the same system. This is still ~1/2 of what I would expect given the benchmarks I've seen for the 1080gtx, but I will take a ~10x increase in throughput for now!
On Thu, Jul 28, 2016 at 9:10 AM, Szilárd Páll <pall.szil...@gmail.com> wrote: > Note that the CPU-GPU load balancing scaled the cut-off to 1.5 nm > increasing the GPU load by about 3x (see the table on line 550 of the > log file you posted). > I find it somewhat suspicious that your CPU seems so slow, but given > that you have a GTX 1080, that may just be normal. > -- > Szilárd > > > On Sun, Jul 17, 2016 at 10:57 PM, Vito Spadavecchio > <spadavecc...@gmail.com> wrote: > > Just wanted to ping this and see if anyone had any ideas :) > > > > On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio < > spadavecc...@gmail.com> > > wrote: > > > >> My apologies! I meant to include the logfile as well. You can view it > >> here: > >> > >> https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0 > >> > >> Thanks! > >> > >> On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalem...@vt.edu> wrote: > >> > >>> > >>> > >>> On 7/14/16 10:35 PM, Vito Spadavecchio wrote: > >>> > >>>> Szilard > >>>> > >>>> You are correct; I misspoke. It appears that the GPU is doing > >>>> *something*, > >>>> but I it would seem it is severely under performing (near the point > where > >>>> its basically doing nothing). > >>>> > >>>> the entire log, not just parts, in particular we want to see the > >>>> > >>>>> header, perf table > >>>>> > >>>>> > >>>> xyz@Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu > >>>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-: > >>>> > >>>> GROMACS is written by: > >>>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par > >>>> Bjelkmar > >>>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian > >>>> Fritsch > >>>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent > >>>> Hindriksen > >>>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten > >>>> Kutzner > >>>> > >>>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter > >>>> Meulenhoff > >>>> Erik Marklund Teemu Murtola Szilard Pall Sander > Pronk > >>>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons > >>>> Sijbers > >>>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten > Wolf > >>>> and the project leaders: > >>>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel > >>>> > >>>> Copyright (c) 1991-2000, University of Groningen, The Netherlands. > >>>> Copyright (c) 2001-2015, The GROMACS development team at > >>>> Uppsala University, Stockholm University and > >>>> the Royal Institute of Technology, Sweden. > >>>> check out http://www.gromacs.org for more information. > >>>> > >>>> GROMACS is free software; you can redistribute it and/or modify it > >>>> under the terms of the GNU Lesser General Public License > >>>> as published by the Free Software Foundation; either version 2.1 > >>>> of the License, or (at your option) any later version. > >>>> > >>>> GROMACS: gmx mdrun, VERSION 5.1.2 > >>>> Executable: /usr/local/gromacs/bin/gmx > >>>> Data prefix: /usr/local/gromacs > >>>> Command line: > >>>> gmx mdrun -deffnm md_0_2 -nb gpu > >>>> > >>>> > >>>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1# > >>>> > >>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible > GPU > >>>> Hardware detected: > >>>> CPU info: > >>>> Vendor: GenuineIntel > >>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz > >>>> SIMD instructions most likely to fit this hardware: AVX2_256 > >>>> SIMD instructions selected at GROMACS compile time: AVX2_256 > >>>> GPU info: > >>>> Number of GPUs detected: 1 > >>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: > >>>> compatible > >>>> > >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) > >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 > >>>> > >>>> Using 1 MPI thread > >>>> Using 8 OpenMP threads > >>>> > >>>> 1 compatible GPU is present, with ID 0 > >>>> 1 GPU auto-selected for this run. > >>>> Mapping of GPU ID to the 1 PP rank in this node: 0 > >>>> > >>>> > >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# > >>>> > >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# > >>>> > >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# > >>>> starting mdrun 'Protein in water' > >>>> 500000 steps, 1000.0 ps. > >>>> > >>>> > >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution > >>>> > >>>> > >>>> heres a snapshot: > >>>> > >>>> # gpu pwr temp sm mem enc dec mclk pclk > >>>> > >>>>> # Idx W C % % % % MHz MHz > >>>>> 0 117 51 56 3 0 0 4513 1847 > >>>>> 0 115 52 54 3 0 0 4513 1847 > >>>>> 0 117 52 47 3 0 0 4513 1847 > >>>>> 0 101 52 52 3 0 0 4513 1848 > >>>>> 0 119 53 54 3 0 0 4513 1847 > >>>>> 0 115 53 54 3 0 0 4513 1848 > >>>>> 0 106 54 54 3 0 0 4513 1848 > >>>>> 0 118 54 54 3 0 0 4513 1849 > >>>>> 0 94 54 52 3 0 0 4513 1848 > >>>>> 0 110 55 52 3 0 0 4513 1849 > >>>>> 0 116 55 53 3 0 0 4513 1854 > >>>>> 0 126 55 51 3 0 0 4513 1854 > >>>>> 0 119 56 51 3 0 0 4513 1855 > >>>>> 0 116 56 52 3 0 0 4513 1855 > >>>>> 0 116 56 53 3 0 0 4513 1855 > >>>>> 0 120 57 53 3 0 0 4513 1855 > >>>>> 0 113 57 54 3 0 0 4513 1855 > >>>>> > >>>>> > >>>> and here's the tail of the performance in .log > >>>> > >>>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) > >>>> > >>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 > >>>>> Using 1 MPI thread > >>>>> Using 8 OpenMP threads > >>>>> 1 compatible GPU is present, with ID 0 > >>>>> 1 GPU auto-selected for this run. > >>>>> Mapping of GPU ID to the 1 PP rank in this node: 0 > >>>>> > >>>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# > >>>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# > >>>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# > >>>>> starting mdrun 'Protein in water' > >>>>> 500000 steps, 1000.0 ps. > >>>>> Writing final coordinates. > >>>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1# > >>>>> Core t (s) Wall t (s) (%) > >>>>> Time: 25285.344 3409.337 741.6 > >>>>> 56:49 > >>>>> (ns/day) (hour/ns) > >>>>> Performance: 25.342 0.947 > >>>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain > Beefheart) > >>>>> > >>>>> > >>>>> As we can see, there is *something* running on the GPU (process id: > >>>> 30784), > >>>> but its never more than 135MiB of memory being used and only uses a > very > >>>> small portion of the gpus power. > >>>> > >>>> Any thoughts are greatly appreciated! > >>>> > >>>> > >>> You cut out the part of the .log file that shows how mdrun actually > spent > >>> its time. This is perhaps the most valuable information. This is why > >>> Szilard asked for you to share a complete .log file (upload it > somewhere, > >>> post a URL) so we can see all relevant details. > >>> > >>> -Justin > >>> > >>> > >>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szil...@gmail.com > > > >>>> wrote: > >>>> > >>>> Hi, > >>>>> > >>>>> If mdrun claims it's using the GPU and you see no errors (and results > >>>>> look reasonable), the GPU is likely being used. That nvidia-smi is > not > >>>>> showing it is admittedly strange. However, this truncated log output > >>>>> does not confirm much, e.g. it does not show the performance table > and > >>>>> the final performance. > >>>>> > >>>>> Could you please run again and look at (and share if approapriate): > >>>>> - the entire log, not just parts, in particular we want to see the > >>>>> header, perf table > >>>>> - the output of nvidia-smi dmon executing *during* mdrun's execution > >>>>> (this monitors the device continuously) > >>>>> > >>>>> Cheers, > >>>>> > >>>>> -- > >>>>> Szilárd > >>>>> > >>>>> > >>>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio > >>>>> <spadavecc...@gmail.com> wrote: > >>>>> > >>>>>> Hello > >>>>>> > >>>>>> After building GROMACS with GPU support and running some simple MD, > I > >>>>>> am > >>>>>> only getting ~20ns/day (which is what I would expect to get out of > my > >>>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which > >>>>>> > >>>>> should > >>>>> > >>>>>> even on moderately large systems, be getting ~100ns/day) > >>>>>> > >>>>>> I've tried launching a mdrun with with the following command: > >>>>>> > >>>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu > >>>>>> > >>>>>> > >>>>>> > >>>>>> With the following in my md.mdp file: > >>>>>> > >>>>>> ; Neighborsearching > >>>>>> > >>>>>> cutoff-scheme = Verlet > >>>>>> > >>>>>> > >>>>>> > >>>>>> The top of the log file for the mdrun notes: > >>>>>> > >>>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible > GPU > >>>>>> > >>>>>>> Hardware detected: > >>>>>>> CPU info: > >>>>>>> Vendor: GenuineIntel > >>>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz > >>>>>>> SIMD instructions most likely to fit this hardware: AVX2_256 > >>>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256 > >>>>>>> GPU info: > >>>>>>> Number of GPUs detected: 1 > >>>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: > >>>>>>> compatible > >>>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision) > >>>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 > >>>>>>> Using 1 MPI thread > >>>>>>> Using 8 OpenMP threads > >>>>>>> 1 GPU user-selected for this run. > >>>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0 > >>>>>>> > >>>>>>> > >>>>>> The output of *nvidia-smi *on my machine is: > >>>>>> > >>>>>> hu Jul 14 11:25:04 2016 > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > +-----------------------------------------------------------------------------+ > >>>>> > >>>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27 > >>>>>>> | > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > |-------------------------------+----------------------+----------------------+ > >>>>> > >>>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile > >>>>>>> > >>>>>> Uncorr. > >>>>> > >>>>>> ECC | > >>>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util > >>>>>>> > >>>>>> Compute > >>>>> > >>>>>> M. | > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > |===============================+======================+======================| > >>>>> > >>>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On | > >>>>>>> N/A | > >>>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3% > >>>>>>> Default | > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > +-------------------------------+----------------------+----------------------+ > >>>>> > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > +-----------------------------------------------------------------------------+ > >>>>> > >>>>>> | Processes: > GPU > >>>>>>> Memory | > >>>>>>> | GPU PID Type Process name > >>>>>>> Usage > >>>>>>> | > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > |=============================================================================| > >>>>> > >>>>>> | 0 1035 G /usr/lib/xorg/Xorg > >>>>>>> 424MiB | > >>>>>>> | 0 2132 G compiz > >>>>>>> 103MiB | > >>>>>>> | 0 5296 G ...s-passed-by-fd > --v8-snapshot-passed-by-fd > >>>>>>> 59MiB | > >>>>>>> | 0 14079 G ...s-passed-by-fd > --v8-snapshot-passed-by-fd > >>>>>>> 31MiB | > >>>>>>> | 0 14234 G ...ing<BrotliEncoding > --disable-features=Doc > >>>>>>> 108MiB | > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > +-----------------------------------------------------------------------------+ > >>>>> > >>>>>> > >>>>>> > >>>>>> and output of *nvcc --version* is: > >>>>>> > >>>>>> nvcc: NVIDIA (R) Cuda compiler driver > >>>>>> > >>>>>>> Copyright (c) 2005-2015 NVIDIA Corporation > >>>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015 > >>>>>>> Cuda compilation tools, release 7.5, V7.5.17 > >>>>>>> > >>>>>> > >>>>>> > >>>>>> Does anyone have any ideas what the problem might be? > >>>>>> > >>>>>> Thanks! > >>>>>> -- > >>>>>> Gromacs Users mailing list > >>>>>> > >>>>>> * Please search the archive at > >>>>>> > >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > >>>>> posting! > >>>>> > >>>>>> > >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >>>>>> > >>>>>> * For (un)subscribe requests visit > >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users > or > >>>>>> > >>>>> send a mail to gmx-users-requ...@gromacs.org. > >>>>> -- > >>>>> Gromacs Users mailing list > >>>>> > >>>>> * Please search the archive at > >>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > >>>>> posting! > >>>>> > >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >>>>> > >>>>> * For (un)subscribe requests visit > >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users > or > >>>>> send a mail to gmx-users-requ...@gromacs.org. > >>>>> > >>>>> > >>> -- > >>> ================================================== > >>> > >>> Justin A. Lemkul, Ph.D. > >>> Ruth L. Kirschstein NRSA Postdoctoral Fellow > >>> > >>> Department of Pharmaceutical Sciences > >>> School of Pharmacy > >>> Health Sciences Facility II, Room 629 > >>> University of Maryland, Baltimore > >>> 20 Penn St. > >>> Baltimore, MD 21201 > >>> > >>> jalem...@outerbanks.umaryland.edu | (410) 706-7441 > >>> http://mackerell.umaryland.edu/~jalemkul > >>> > >>> ================================================== > >>> > >>> -- > >>> Gromacs Users mailing list > >>> > >>> * Please search the archive at > >>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > >>> posting! > >>> > >>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > >>> > >>> * For (un)subscribe requests visit > >>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > >>> send a mail to gmx-users-requ...@gromacs.org. > >>> > >> > >> > > -- > > Gromacs Users mailing list > > > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.