Just wanted to ping this and see if anyone had any ideas :) On Fri, Jul 15, 2016 at 10:30 AM, Vito Spadavecchio <spadavecc...@gmail.com> wrote:
> My apologies! I meant to include the logfile as well. You can view it > here: > > https://www.dropbox.com/s/8ddm226gpgs5kpr/md_0_2.log?dl=0 > > Thanks! > > On Fri, Jul 15, 2016 at 2:45 AM, Justin Lemkul <jalem...@vt.edu> wrote: > >> >> >> On 7/14/16 10:35 PM, Vito Spadavecchio wrote: >> >>> Szilard >>> >>> You are correct; I misspoke. It appears that the GPU is doing >>> *something*, >>> but I it would seem it is severely under performing (near the point where >>> its basically doing nothing). >>> >>> the entire log, not just parts, in particular we want to see the >>> >>>> header, perf table >>>> >>>> >>> xyz@Turing:~/Desktop/xyz_MD/APO_xyz$ gmx mdrun -deffnm md_0_2 -nb gpu >>> :-) GROMACS - gmx mdrun, VERSION 5.1.2 (-: >>> >>> GROMACS is written by: >>> Emile Apol Rossen Apostolov Herman J.C. Berendsen Par >>> Bjelkmar >>> Aldert van Buuren Rudi van Drunen Anton Feenstra Sebastian >>> Fritsch >>> Gerrit Groenhof Christoph Junghans Anca Hamuraru Vincent >>> Hindriksen >>> Dimitrios Karkoulis Peter Kasson Jiri Kraus Carsten >>> Kutzner >>> >>> Per Larsson Justin A. Lemkul Magnus Lundborg Pieter >>> Meulenhoff >>> Erik Marklund Teemu Murtola Szilard Pall Sander Pronk >>> Roland Schulz Alexey Shvetsov Michael Shirts Alfons >>> Sijbers >>> Peter Tieleman Teemu Virolainen Christian Wennberg Maarten Wolf >>> and the project leaders: >>> Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel >>> >>> Copyright (c) 1991-2000, University of Groningen, The Netherlands. >>> Copyright (c) 2001-2015, The GROMACS development team at >>> Uppsala University, Stockholm University and >>> the Royal Institute of Technology, Sweden. >>> check out http://www.gromacs.org for more information. >>> >>> GROMACS is free software; you can redistribute it and/or modify it >>> under the terms of the GNU Lesser General Public License >>> as published by the Free Software Foundation; either version 2.1 >>> of the License, or (at your option) any later version. >>> >>> GROMACS: gmx mdrun, VERSION 5.1.2 >>> Executable: /usr/local/gromacs/bin/gmx >>> Data prefix: /usr/local/gromacs >>> Command line: >>> gmx mdrun -deffnm md_0_2 -nb gpu >>> >>> >>> Back Off! I just backed up md_0_2.log to ./#md_0_2.log.1# >>> >>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU >>> Hardware detected: >>> CPU info: >>> Vendor: GenuineIntel >>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz >>> SIMD instructions most likely to fit this hardware: AVX2_256 >>> SIMD instructions selected at GROMACS compile time: AVX2_256 >>> GPU info: >>> Number of GPUs detected: 1 >>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: >>> compatible >>> >>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) >>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >>> >>> Using 1 MPI thread >>> Using 8 OpenMP threads >>> >>> 1 compatible GPU is present, with ID 0 >>> 1 GPU auto-selected for this run. >>> Mapping of GPU ID to the 1 PP rank in this node: 0 >>> >>> >>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# >>> >>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# >>> >>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# >>> starting mdrun 'Protein in water' >>> 500000 steps, 1000.0 ps. >>> >>> >>> - the output of nvidia-smi dmon executing *during* mdrun's execution >>> >>> >>> heres a snapshot: >>> >>> # gpu pwr temp sm mem enc dec mclk pclk >>> >>>> # Idx W C % % % % MHz MHz >>>> 0 117 51 56 3 0 0 4513 1847 >>>> 0 115 52 54 3 0 0 4513 1847 >>>> 0 117 52 47 3 0 0 4513 1847 >>>> 0 101 52 52 3 0 0 4513 1848 >>>> 0 119 53 54 3 0 0 4513 1847 >>>> 0 115 53 54 3 0 0 4513 1848 >>>> 0 106 54 54 3 0 0 4513 1848 >>>> 0 118 54 54 3 0 0 4513 1849 >>>> 0 94 54 52 3 0 0 4513 1848 >>>> 0 110 55 52 3 0 0 4513 1849 >>>> 0 116 55 53 3 0 0 4513 1854 >>>> 0 126 55 51 3 0 0 4513 1854 >>>> 0 119 56 51 3 0 0 4513 1855 >>>> 0 116 56 52 3 0 0 4513 1855 >>>> 0 116 56 53 3 0 0 4513 1855 >>>> 0 120 57 53 3 0 0 4513 1855 >>>> 0 113 57 54 3 0 0 4513 1855 >>>> >>>> >>> and here's the tail of the performance in .log >>> >>> Reading file md_0_2.tpr, VERSION 5.1.2 (single precision) >>> >>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >>>> Using 1 MPI thread >>>> Using 8 OpenMP threads >>>> 1 compatible GPU is present, with ID 0 >>>> 1 GPU auto-selected for this run. >>>> Mapping of GPU ID to the 1 PP rank in this node: 0 >>>> >>>> Back Off! I just backed up md_0_2.trr to ./#md_0_2.trr.1# >>>> Back Off! I just backed up md_0_2.xtc to ./#md_0_2.xtc.1# >>>> Back Off! I just backed up md_0_2.edr to ./#md_0_2.edr.1# >>>> starting mdrun 'Protein in water' >>>> 500000 steps, 1000.0 ps. >>>> Writing final coordinates. >>>> Back Off! I just backed up md_0_2.gro to ./#md_0_2.gro.1# >>>> Core t (s) Wall t (s) (%) >>>> Time: 25285.344 3409.337 741.6 >>>> 56:49 >>>> (ns/day) (hour/ns) >>>> Performance: 25.342 0.947 >>>> gcq#11: "She's Not Bad, She's Just Genetically Mean" (Captain Beefheart) >>>> >>>> >>>> As we can see, there is *something* running on the GPU (process id: >>> 30784), >>> but its never more than 135MiB of memory being used and only uses a very >>> small portion of the gpus power. >>> >>> Any thoughts are greatly appreciated! >>> >>> >> You cut out the part of the .log file that shows how mdrun actually spent >> its time. This is perhaps the most valuable information. This is why >> Szilard asked for you to share a complete .log file (upload it somewhere, >> post a URL) so we can see all relevant details. >> >> -Justin >> >> >> >>> >>> >>> >>> >>> >>> On Thu, Jul 14, 2016 at 4:54 PM, Szilárd Páll <pall.szil...@gmail.com> >>> wrote: >>> >>> Hi, >>>> >>>> If mdrun claims it's using the GPU and you see no errors (and results >>>> look reasonable), the GPU is likely being used. That nvidia-smi is not >>>> showing it is admittedly strange. However, this truncated log output >>>> does not confirm much, e.g. it does not show the performance table and >>>> the final performance. >>>> >>>> Could you please run again and look at (and share if approapriate): >>>> - the entire log, not just parts, in particular we want to see the >>>> header, perf table >>>> - the output of nvidia-smi dmon executing *during* mdrun's execution >>>> (this monitors the device continuously) >>>> >>>> Cheers, >>>> >>>> -- >>>> Szilárd >>>> >>>> >>>> On Thu, Jul 14, 2016 at 8:28 PM, Vito Spadavecchio >>>> <spadavecc...@gmail.com> wrote: >>>> >>>>> Hello >>>>> >>>>> After building GROMACS with GPU support and running some simple MD, I >>>>> am >>>>> only getting ~20ns/day (which is what I would expect to get out of my >>>>> i7-6700k cpu only. My machine currently has a 1080 GTX in it, which >>>>> >>>> should >>>> >>>>> even on moderately large systems, be getting ~100ns/day) >>>>> >>>>> I've tried launching a mdrun with with the following command: >>>>> >>>>> gmx mdrun -deffnm md_0_1 -gpu_id 0 -nb gpu >>>>> >>>>> >>>>> >>>>> With the following in my md.mdp file: >>>>> >>>>> ; Neighborsearching >>>>> >>>>> cutoff-scheme = Verlet >>>>> >>>>> >>>>> >>>>> The top of the log file for the mdrun notes: >>>>> >>>>> Running on 1 node with total 4 cores, 8 logical cores, 1 compatible GPU >>>>> >>>>>> Hardware detected: >>>>>> CPU info: >>>>>> Vendor: GenuineIntel >>>>>> Brand: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz >>>>>> SIMD instructions most likely to fit this hardware: AVX2_256 >>>>>> SIMD instructions selected at GROMACS compile time: AVX2_256 >>>>>> GPU info: >>>>>> Number of GPUs detected: 1 >>>>>> #0: NVIDIA GeForce GTX 1080, compute cap.: 6.1, ECC: no, stat: >>>>>> compatible >>>>>> Reading file md_0_1.tpr, VERSION 5.1.2 (single precision) >>>>>> Changing nstlist from 10 to 40, rlist from 1 to 1.099 >>>>>> Using 1 MPI thread >>>>>> Using 8 OpenMP threads >>>>>> 1 GPU user-selected for this run. >>>>>> Mapping of GPU ID to the 1 PP rank in this node: 0 >>>>>> >>>>>> >>>>> The output of *nvidia-smi *on my machine is: >>>>> >>>>> hu Jul 14 11:25:04 2016 >>>>> >>>>>> >>>>>> >>>>>> >>>> +-----------------------------------------------------------------------------+ >>>> >>>>> | NVIDIA-SMI 367.27 Driver Version: 367.27 >>>>>> | >>>>>> >>>>>> >>>>>> >>>> |-------------------------------+----------------------+----------------------+ >>>> >>>>> | GPU Name Persistence-M| Bus-Id Disp.A | Volatile >>>>>> >>>>> Uncorr. >>>> >>>>> ECC | >>>>>> | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util >>>>>> >>>>> Compute >>>> >>>>> M. | >>>>>> >>>>>> >>>>>> >>>> |===============================+======================+======================| >>>> >>>>> | 0 GeForce GTX 1080 Off | 0000:01:00.0 On | >>>>>> N/A | >>>>>> | 30% 41C P8 7W / 180W | 730MiB / 8113MiB | 3% >>>>>> Default | >>>>>> >>>>>> >>>>>> >>>> +-------------------------------+----------------------+----------------------+ >>>> >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> +-----------------------------------------------------------------------------+ >>>> >>>>> | Processes: GPU >>>>>> Memory | >>>>>> | GPU PID Type Process name >>>>>> Usage >>>>>> | >>>>>> >>>>>> >>>>>> >>>> |=============================================================================| >>>> >>>>> | 0 1035 G /usr/lib/xorg/Xorg >>>>>> 424MiB | >>>>>> | 0 2132 G compiz >>>>>> 103MiB | >>>>>> | 0 5296 G ...s-passed-by-fd --v8-snapshot-passed-by-fd >>>>>> 59MiB | >>>>>> | 0 14079 G ...s-passed-by-fd --v8-snapshot-passed-by-fd >>>>>> 31MiB | >>>>>> | 0 14234 G ...ing<BrotliEncoding --disable-features=Doc >>>>>> 108MiB | >>>>>> >>>>>> >>>>>> >>>> +-----------------------------------------------------------------------------+ >>>> >>>>> >>>>> >>>>> and output of *nvcc --version* is: >>>>> >>>>> nvcc: NVIDIA (R) Cuda compiler driver >>>>> >>>>>> Copyright (c) 2005-2015 NVIDIA Corporation >>>>>> Built on Tue_Aug_11_14:27:32_CDT_2015 >>>>>> Cuda compilation tools, release 7.5, V7.5.17 >>>>>> >>>>> >>>>> >>>>> Does anyone have any ideas what the problem might be? >>>>> >>>>> Thanks! >>>>> -- >>>>> Gromacs Users mailing list >>>>> >>>>> * Please search the archive at >>>>> >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >>>> posting! >>>> >>>>> >>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >>>>> >>>>> * For (un)subscribe requests visit >>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >>>>> >>>> send a mail to gmx-users-requ...@gromacs.org. >>>> -- >>>> Gromacs Users mailing list >>>> >>>> * Please search the archive at >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >>>> posting! >>>> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >>>> >>>> * For (un)subscribe requests visit >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >>>> send a mail to gmx-users-requ...@gromacs.org. >>>> >>>> >> -- >> ================================================== >> >> Justin A. Lemkul, Ph.D. >> Ruth L. Kirschstein NRSA Postdoctoral Fellow >> >> Department of Pharmaceutical Sciences >> School of Pharmacy >> Health Sciences Facility II, Room 629 >> University of Maryland, Baltimore >> 20 Penn St. >> Baltimore, MD 21201 >> >> jalem...@outerbanks.umaryland.edu | (410) 706-7441 >> http://mackerell.umaryland.edu/~jalemkul >> >> ================================================== >> >> -- >> Gromacs Users mailing list >> >> * Please search the archive at >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >> posting! >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> * For (un)subscribe requests visit >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >> send a mail to gmx-users-requ...@gromacs.org. >> > > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.