Are you running on login nodes or compute nodes (I can’t seem to tell from the configure.log)? If running from login nodes, do they support running with GPU’s? Some clusters will install stub versions of cuda runtime on login nodes (such that configuration can find them), but that won’t actually work in practice.
If this is the case then CUDA will fail to initialize with this exact error. IIRC It wasn’t until CUDA 11.1 that they created a specific error code (cudaErrorStubLibrary) for it. Best regards, Jacob Faibussowitsch (Jacob Fai - booss - oh - vitch) > On Jan 19, 2022, at 12:07, Fande Kong <[email protected]> wrote: > > Thanks, Jacob, and Junchao > > The log was attached. I am using Sawtooth at INL > https://hpc.inl.gov/SitePages/Home.aspx > <https://hpc.inl.gov/SitePages/Home.aspx> > > > Thanks, > > Fande > > On Wed, Jan 19, 2022 at 10:32 AM Jacob Faibussowitsch <[email protected] > <mailto:[email protected]>> wrote: > Hi Fande, > > What machine are you running this on? Please attach configure.log so I can > troubleshoot this. > > Best regards, > > Jacob Faibussowitsch > (Jacob Fai - booss - oh - vitch) > >> On Jan 19, 2022, at 10:04, Fande Kong <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi All, >> >> Upgraded PETSc from 3.16.1 to the current main branch. I suddenly got the >> following error message: >> >> 2d_diffusion]$ ../../../moose_test-dbg -i 2d_diffusion_test.i >> -use_gpu_aware_mpi 0 -gpu_mat_type aijcusparse -gpu_vec_type cuda -log_view >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Missing or incorrect user input >> [0]PETSC ERROR: Cannot eagerly initialize cuda, as doing so results in cuda >> error 35 (cudaErrorInsufficientDriver) : CUDA driver version is insufficient >> for CUDA runtime version >> [0]PETSC ERROR: See https://petsc.org/release/faq/ >> <https://petsc.org/release/faq/> for trouble shooting. >> [0]PETSC ERROR: Petsc Development GIT revision: v3.16.3-618-gad32f7e GIT >> Date: 2022-01-18 16:04:31 +0000 >> [0]PETSC ERROR: ../../../moose_test-dbg on a arch-linux-c-opt named r8i3n0 >> by kongf Wed Jan 19 08:30:13 2022 >> [0]PETSC ERROR: Configure options --with-debugging=no >> --with-shared-libraries=1 --download-fblaslapack=1 --download-metis=1 >> --download-ptscotch=1 --download-parmetis=1 --download-mumps=1 >> --download-strumpack=1 --download-scalapack=1 --download-slepc=1 >> --with-mpi=1 --with-cxx-dialect=C++14 --with-fortran-bindings=0 >> --with-sowing=0 --with-64-bit-indices --with-make-np=24 --with-cuda >> --with-cudac=nvcc --with-cuda-arch=70 --download-kokkos=1 >> [0]PETSC ERROR: #1 initialize() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/impls/cupm/cupmdevice.cxx:298 >> [0]PETSC ERROR: #2 PetscDeviceInitializeTypeFromOptions_Private() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:299 >> [0]PETSC ERROR: #3 PetscDeviceInitializeFromOptions_Internal() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/device/interface/device.cxx:425 >> [0]PETSC ERROR: #4 PetscInitialize_Common() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:963 >> [0]PETSC ERROR: #5 PetscInitialize() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/src/sys/objects/pinit.c:1238 >> [0]PETSC ERROR: #6 SlepcInitialize() at >> /home/kongf/workhome/sawtooth/moosegpu/petsc/arch-linux-c-opt/externalpackages/git.slepc/src/sys/slepcinit.c:275 >> [0]PETSC ERROR: #7 LibMeshInit() at ../src/base/libmesh.C:522 >> [r8i3n0:mpi_rank_0][MPIDI_CH3_Abort] application called >> MPI_Abort(MPI_COMM_WORLD, 95) - process 0: No such file or directory (2) >> >> Thanks, >> >> Fande > > <configure.log>
