Hi Szilárd, Thanks to your tweaked version of the code, I eventually found out that the CUDA capable GPU's id wasn't what I thought it was.
I thought since nvidia-smi displayed 1080 as the second card (GPU number 1), it has id 1, but it actually has id 0. Therefore, setting CUDA_VISIBLE_DEVICES=0 gmx mdrun <ops> did the trick. Thank you to everyone involved. Cheers, Harry ________________________________ From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se <gromacs.org_gmx-users-boun...@maillist.sys.kth.se> on behalf of Szilárd Páll <pall.szil...@gmail.com> Sent: Monday, August 13, 2018 10:52 PM To: Discussion list for GROMACS users Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: Condition: cudaSuccess == cudaPeekAtLastError() Hi, I can not reproduce such an error, not even by emulating your use case. Can you please try to build this slightly tweaked version of the code: https://gerrit.gromacs.org/changes/8172/revisions/fb358c835c312952571a78f32971cfacf42df966/archive?format=tgz and let me know what the error is? Also, please provide the full log and stdandard outputs with/without CUDA_VISIBLE_DEVICES. Cheers, -- Szilárd On Sat, Aug 11, 2018 at 4:24 AM Jia Hong <jiahon...@hotmail.com> wrote: > Hi Mark, > > I've set CUDA_VISIBLE_DEVICES to 1, the exact command was > > export CUDA_VISIBLE_DEVICES=1 > > I chose 1 because that's the GPU id for the 1080 card. Then, in the same > shell I ran the gmx command (both with and without -gpu_id 1 argument). > > As for the log file, I thought it was md_0_01_GPU.log, which I sent as a > follow-up message. Nonetheless, > here's the link to the file > https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0. > [ > https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-unknown-large.png > ]<https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0> > > md_0_01_GPU.log< > https://www.dropbox.com/s/xc8z09wvnl0314m/md_0_01_GPU.log?dl=0> > Shared with Dropbox > www.dropbox.com<http://www.dropbox.com> > > > Cheers, > Harry > > ________________________________ > From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se < > gromacs.org_gmx-users-boun...@maillist.sys.kth.se> on behalf of Mark > Abraham <mark.j.abra...@gmail.com> > Sent: Saturday, August 11, 2018 7:43 AM > To: gmx-us...@gromacs.org > Subject: Re: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: > Condition: cudaSuccess == cudaPeekAtLastError() > > Hi, > > To what did you set CUDA_VISIBLE_DEVICES? Just saying it didn't work > doesn't exclude that it couldn't have worked because it wasn't right :-) > > Also, could you please share a full log file via a sharing service and post > a link. This fragment doesn't help us find out how the detection saw your > system, unfortunately. > > Mark > > On Wed, Aug 8, 2018, 10:53 Jia Hong <jiahon...@hotmail.com> wrote: > > > The log file: > > > > > > Log file opened on Wed Aug 8 16:33:34 2018 > > Host: matthias-processing pid: 28756 rank ID: 0 number of ranks: 1 > > :-) GROMACS - gmx mdrun, 2018.2 (-: > > > > GROMACS is written by: > > Emile Apol Rossen Apostolov Paul Bauer Herman J.C. > > Berendsen > > Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton > Feenstra > > Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru > > Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus > > Carsten Kutzner Per Larsson Justin A. Lemkul Viveca > Lindahl > > Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola > > Szilard Pall Sander Pronk Roland Schulz Alexey > Shvetsov > > Michael Shirts Alfons Sijbers Peter Tieleman Teemu > Virolainen > > Christian Wennberg Maarten Wolf > > and the project leaders: > > Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel > > > > Copyright (c) 1991-2000, University of Groningen, The Netherlands. > > Copyright (c) 2001-2017, The GROMACS development team at > > Uppsala University, Stockholm University and > > the Royal Institute of Technology, Sweden. > > check out http://www.gromacs.org for more information. > > > > GROMACS is free software; you can redistribute it and/or modify it > > under the terms of the GNU Lesser General Public License > > as published by the Free Software Foundation; either version 2.1 > > of the License, or (at your option) any later version. > > > > GROMACS: gmx mdrun, version 2018.2 > > Executable: /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx > > Data prefix: /home/matthias/Documents/gromacs-2018.2/gpu_install > > Working dir: /home/matthias/Documents/MD_simulations/D_EY_ext > > Command line: > > gmx mdrun -deffnm md_0_01_GPU > > > > GROMACS version: 2018.2 > > Precision: single > > Memory model: 64 bit > > MPI library: thread_mpi > > OpenMP support: enabled (GMX_OPENMP_MAX_THREADS = 64) > > GPU support: CUDA > > SIMD instructions: AVX_256 > > FFT library: fftw-3.3.8-sse2-avx-avx2-avx2_128-avx512 > > RDTSCP usage: enabled > > TNG support: enabled > > Hwloc support: disabled > > Tracing support: disabled > > Built on: 2018-08-03 08:06:55 > > Built by: matthias@matthias-processing [CMAKE] > > Build OS/arch: Linux 4.15.0-29-generic x86_64 > > Build CPU vendor: Intel > > Build CPU brand: Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz > > Build CPU family: 6 Model: 45 Stepping: 7 > > Build CPU features: aes apic avx clfsh cmov cx8 cx16 htt intel lahf mmx > > msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdtscp sse2 sse3 > > sse4.1 sse4.2 ssse3 tdt x2apic > > C compiler: /usr/bin/cc GNU 5.4.0 > > C compiler flags: -mavx -O3 -DNDEBUG -funroll-all-loops > > -fexcess-precision=fast > > C++ compiler: /usr/bin/c++ GNU 5.4.0 > > C++ compiler flags: -mavx -std=c++11 -O3 -DNDEBUG > -funroll-all-loops > > -fexcess-precision=fast > > CUDA compiler: /usr/local/cuda-9.0/bin/nvcc nvcc: NVIDIA (R) Cuda > > compiler driver;Copyright (c) 2005-2017 NVIDIA Corporation;Built on > > Fri_Sep__1_21:08:03_CDT_2017;Cuda compilation tools, release 9.0, > V9.0.176 > > CUDA compiler > > > flags:-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_70,code=compute_70;-use_fast_math;-D_FORCE_INLINES;; > > > ;-mavx;-std=c++11;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast; > > CUDA driver: 9.0 > > CUDA runtime: 9.0 > > > > > > ________________________________ > > From: gromacs.org_gmx-users-boun...@maillist.sys.kth.se < > > gromacs.org_gmx-users-boun...@maillist.sys.kth.se> on behalf of Jia > Hong < > > jiahon...@hotmail.com> > > Sent: Wednesday, August 8, 2018 3:50 PM > > To: gromacs.org_gmx-users@maillist.sys.kth.se > > Subject: [gmx-users] GROMACS 2018.2 mdrun GPU Assertion failed: > Condition: > > cudaSuccess == cudaPeekAtLastError() > > > > Hi, > > > > I've recently compiled gromacs 2018.2 on ubuntu 16.04.5 LTS with CUDA 9.0 > > on an Nvidia 1080. I've got the error below on the mdrun command. I've > > searched around ( > > > https://mailman-1.sys.kth.se/pipermail/gromacs.org_gmx-users/2018-February/118631.html > ) > > and I tried the suggested export CUDA_VISIBLE_DEVICES without any luck. > > > > I currently have 2 nvidia GPUs sitting inside the machine, the first > > doesn't have CUDA support (old graphics card) while the second is the > > aforementioned 1080. I've also verified that the 1080 has gpu id 1, and > ran > > the command: > > > > gmx mdrun -deffnm md_0_01_GPU -gpu_id 1 > > > > which gives the same error. > > I've attached the log file too. > > > > (I don't know if this helps, but I manage to compile a CUDA test program > > to verify that my CUDA installation isn't broken: > > > http://ramblingsofagamedevstudent.blogspot.com/2014/03/set-up-visual-studio-2012-for-cuda.html > > ) > > > > Cheers, > > Harry > > > > The output: > > > > > > :-) GROMACS - gmx mdrun, 2018.2 (-: > > > > GROMACS is written by: > > Emile Apol Rossen Apostolov Paul Bauer Herman J.C. > > Berendsen > > Par Bjelkmar Aldert van Buuren Rudi van Drunen Anton > Feenstra > > Gerrit Groenhof Aleksei Iupinov Christoph Junghans Anca Hamuraru > > Vincent Hindriksen Dimitrios Karkoulis Peter Kasson Jiri Kraus > > Carsten Kutzner Per Larsson Justin A. Lemkul Viveca > Lindahl > > Magnus Lundborg Pieter Meulenhoff Erik Marklund Teemu Murtola > > Szilard Pall Sander Pronk Roland Schulz Alexey > Shvetsov > > Michael Shirts Alfons Sijbers Peter Tieleman Teemu > Virolainen > > Christian Wennberg Maarten Wolf > > and the project leaders: > > Mark Abraham, Berk Hess, Erik Lindahl, and David van der Spoel > > > > Copyright (c) 1991-2000, University of Groningen, The Netherlands. > > Copyright (c) 2001-2017, The GROMACS development team at > > Uppsala University, Stockholm University and > > the Royal Institute of Technology, Sweden. > > check out http://www.gromacs.org for more information. > > > > GROMACS is free software; you can redistribute it and/or modify it > > under the terms of the GNU Lesser General Public License > > as published by the Free Software Foundation; either version 2.1 > > of the License, or (at your option) any later version. > > > > GROMACS: gmx mdrun, version 2018.2 > > Executable: /home/matthias/Documents/gromacs-2018.2/gpu_install/bin/gmx > > Data prefix: /home/matthias/Documents/gromacs-2018.2/gpu_install > > Working dir: /home/matthias/Documents/MD_simulations/D_EY_ext > > Command line: > > gmx mdrun -deffnm md_0_01_GPU > > > > > > Back Off! I just backed up md_0_01_GPU.log to ./#md_0_01_GPU.log.10# > > > > ------------------------------------------------------- > > Program: gmx mdrun, version 2018.2 > > Source file: src/gromacs/gpu_utils/gpu_utils.cu (line 762) > > Function: void findGpus(gmx_gpu_info_t*) > > > > Assertion failed: > > Condition: cudaSuccess == cudaPeekAtLastError() > > We promise to return with clean CUDA state! > > > > For more information and tips for troubleshooting, please check the > GROMACS > > website at http://www.gromacs.org/Documentation/Errors > > > > -- > > Gromacs Users mailing list > > > > * Please search the archive at > > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > > posting! > > > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > > > * For (un)subscribe requests visit > > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > > send a mail to gmx-users-requ...@gromacs.org. > > > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org. -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.