On Tue, Oct 20, 2020 at 4:40 AM Héctor Barreiro Cabrera <[email protected]> wrote:
> El jue., 15 oct. 2020 a las 23:32, Barry Smith (<[email protected]>) > escribió: > >> >> We still have the assumption the AIJ matrix always has a copy on the >> GPU. How did you fill up the matrix on the GPU while not having its copy >> on the CPU? >> >> My strategy here was to initialize the structure on the CPU with dummy > values to have the corresponding device arrays allocated. Ideally I would > have initialized the structure on a kernel as well, since my intention is > to keep all data on the GPU (and not hit host memory other than for > debugging). But since the topology of my problem remains constant over > time, this approach proved to be sufficient. I did not find any problem > with my use case so far. > > One thing I couldn't figure out, though, is how to force PETSc to transfer > the data back to host. MatView always displays the dummy values I used for > initialization. Is there a function to do this? > Hmm, this should happen automatically, so we have missed something. How do you change the values on the device? Thanks, Matt > Thanks for the replies, by the way! I'm quite surprised how responsive the > PETSc community is! :) > > Cheers, > Héctor > > >> Barry >> >> When we remove this assumption we have to add a bunch more code for CPU >> only things to make sure they properly get the data from the GPU. >> >> >> On Oct 15, 2020, at 4:16 AM, Héctor Barreiro Cabrera <[email protected]> >> wrote: >> >> Hello fellow PETSc users, >> >> Following up my previous email >> <https://lists.mcs.anl.gov/pipermail/petsc-users/2020-September/042511.html>, >> I managed to feed the entry data to a SeqAICUSPARSE matrix through a CUDA >> kernel using the new MatCUSPARSEGetDeviceMatWrite function (thanks Barry >> Smith and Mark Adams!). However, I am now facing problems when trying to >> use this matrix within a SNES solver with the Eisenstat-Walker method >> enabled. >> >> According to PETSc's error log, the preconditioner is failing to invert >> the matrix diagonal. Specifically it says that: >> [0]PETSC ERROR: Arguments are incompatible >> [0]PETSC ERROR: Zero diagonal on row 0 >> [0]PETSC ERROR: Configure options PETSC_ARCH=win64_vs2019_release >> --with-cc="win32fe cl" --with-cxx="win32fe cl" --with-clanguage=C++ >> --with-fc=0 --with-mpi=0 --with-cuda=1 --with-cudac="win32fe nvcc" >> --with-cuda-dir=~/cuda --download-f2cblaslapack=1 --with-precision=single >> --with-64-bit-indices=0 --with-single-library=1 --with-endian=little >> --with-debugging=0 --with-x=0 --with-windows-graphics=0 >> --with-shared-libraries=1 --CUDAOPTFLAGS=-O2 >> >> The stack trace leads to the diagonal inversion routine: >> [0]PETSC ERROR: #1 MatInvertDiagonal_SeqAIJ() line 1913 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c >> [0]PETSC ERROR: #2 MatSOR_SeqAIJ() line 1944 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\impls\aij\seq\aij.c >> [0]PETSC ERROR: #3 MatSOR() line 4005 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\mat\INTERF~1\matrix.c >> [0]PETSC ERROR: #4 PCPreSolve_Eisenstat() line 79 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\impls\eisens\eisen.c >> [0]PETSC ERROR: #5 PCPreSolve() line 1549 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\pc\INTERF~1\precon.c >> [0]PETSC ERROR: #6 KSPSolve_Private() line 686 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c >> [0]PETSC ERROR: #7 KSPSolve() line 889 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\ksp\ksp\INTERF~1\itfunc.c >> [0]PETSC ERROR: #8 SNESSolve_NEWTONLS() line 225 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\impls\ls\ls.c >> [0]PETSC ERROR: #9 SNESSolve() line 4567 in >> C:\cygwin64\home\HBARRE~1\PETSC-~1\src\snes\INTERF~1\snes.c >> >> I am 100% positive that the diagonal does not contain a zero entry, so my >> suspicions are either that this operation is not supported on the GPU at >> all (MatInvertDiagonal_SeqAIJ seems to access host-side memory) or that I >> am missing some setting to make this work on the GPU. Is this correct? >> >> Thanks! >> >> Cheers, >> Héctor >> >> >> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
