By the way, one more piece of the puzzle that i feel may contribute, If i try setting the Preconditioner in the program as either one of:
call PCSetType(pc,PCSAVIENNACL ,ierr) call PCSetType(pc,PCROWSCALINGVIENNACL,ierr) call PCSetType(pc,PCCHOWILUVIENNACL,ierr) It will give an error at compile time of the like: call PCSetType(pc,PCCHOWILUVIENNACL,ierr) 1 Error: Symbol 'pcchowiluviennacl' at (1) has no IMPLICIT type If i instead use: call PetscOptionsSetValue(PETSC_NULL_OPTIONS,'-pc_type','saviennacl',ierr) Or give it that option at execute time as a command line argument, it setups the matrix as intended and it works for one processor, Let me know if this rings a bell, i just tried updating the petsc repo and reinstalling but it didn't work, Thanks, Manuel On Wed, Aug 29, 2018 at 11:50 AM, Manuel Valera <mvaler...@sdsu.edu> wrote: > Hi everyone, > > Thanks for your responses, i understand communicating on this way to this > level of technicality can be hard, i still think we can work ways to solve > this problem though, > > I can say the following at this point: > > > - Program works without issues in any other non-gpu preconditioner, > just by calling -pc_type ### . > - Program works correctly with ViennaCL preconditioners with mpirun -n > 1, that is on one processor. > - Program breaks with every ViennaCL preconditioner when attempting > any more than 1 processors. > - I haven't tried other GPU preconditioners because ViennaCL gave the > best performance in 1 processor before. > - I have made sure the matrix type is mpiaijviennacl as it is printed > right before the error log. > - I am attaching what i think are the backtrace logs of the run with > two cores, as i have limited experience with this kind of debugging, and i > cannot make sense of what they say, please let me know if you need me to do > something else. > > > Current options i can think of are giving you permission to clone my model > repo, or for you to guide me trough the process remotely, > > Thanks, > > .-.-.-.- > > > The error i get is still: > > Matrix type: mpiaijviennacl > > Of sizes: 125 x 125 > Matrix type: mpiaijviennacl > > Of sizes: 125 x 125 > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: No support for this operation for this object type > [0]PETSC ERROR: Currently only handles ViennaCL matrices > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT > Date: 2018-05-31 17:31:13 +0300 > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed > Aug 29 11:43:25 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 > --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 > --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 > --download-viennacl > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: #3 KSPSetUp() line 381 in /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [1]PETSC ERROR: ------------------------------ > ------------------------------------------ > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, > probably memory access out of range > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [1]PETSC ERROR: likely location of problem given in stack below > [1]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [1]PETSC ERROR: INSTEAD the line number of the start of the function > [1]PETSC ERROR: is given. > [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 > /home/valera/petsc/src/sys/error/errtrace.c > [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/ > error/err.c > [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [1]PETSC ERROR: [1] KSPSetUp line 294 /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [1]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [1]PETSC ERROR: Signal received > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT > Date: 2018-05-31 17:31:13 +0300 > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed > Aug 29 11:43:25 2018 > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 > --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 > --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 > --download-viennacl > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file > -------------------------------------------------------------------------- > MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD > with errorcode 59. > > NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. > You may or may not see output from other processes, depending on > exactly when Open MPI kills them. > -------------------------------------------------------------------------- > [0]PETSC ERROR: ------------------------------ > ------------------------------------------ > [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the > batch system) has told this process to end > [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger > [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/ > documentation/faq.html#valgrind > [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS > X to find memory corruption errors > [0]PETSC ERROR: likely location of problem given in stack below > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] MatSetErrorIfFailure line 116 > /home/valera/petsc/src/mat/utils/gcreate.c > [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45 > /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu > [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ > interface/precon.c > [0]PETSC ERROR: [0] KSPSetUp line 294 /home/valera/petsc/src/ksp/ > ksp/interface/itfunc.c > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > [0]PETSC ERROR: Signal received > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT > Date: 2018-05-31 17:31:13 +0300 > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed > Aug 29 11:43:25 2018 > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug > --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 > --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 > --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 > --download-viennacl > [0]PETSC ERROR: #4 User provided function() line 0 in unknown file > [node50:32783] 1 more process has sent help message help-mpi-api.txt / > mpi-abort > [node50:32783] Set MCA parameter "orte_base_help_aggregate" to 0 to see > all help / error messages > > > On Tue, Aug 28, 2018 at 9:34 PM, Karl Rupp <r...@iue.tuwien.ac.at> wrote: > >> Hi Manuel, >> >> as Barry said, it is hard for us to provide any help without having a >> more complete picture of what is going on. >> >> The error you report seems to come from the AMG preconditioner in >> ViennaCL. This can have many origins. Do other preconditioners run without >> error? Can you run in a debugger and provide a complete backtrace? >> >> Thanks and best regards, >> Karli >> >> >> On 08/29/2018 01:33 AM, Manuel Valera wrote: >> >>> Talked too fast, >>> >>> After fixing that problem, i tried more than one mpi processor and got >>> the following: >>> >>> Matrix type: mpiaijviennacl >>> Of sizes: 125 x 125 >>> Matrix type: mpiaijviennacl >>> Of sizes: 125 x 125 >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: No support for this operation for this object type >>> [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT >>> Date: 2018-05-31 17:31:13 +0300 >>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue >>> Aug 28 16:30:02 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl >>> [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu < >>> http://saviennacl.cu> >>> [0]PETSC ERROR: #2 PCSetUp() line 932 in /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [1]PETSC ERROR: ------------------------------ >>> ------------------------------------------ >>> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >>> probably memory access out of range >>> [1]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>> ocumentation/faq.html#valgrind >>> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [1]PETSC ERROR: likely location of problem given in stack below >>> [1]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [1]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [1]PETSC ERROR: is given. >>> [1]PETSC ERROR: [1] PetscTraceBackErrorHandler line 182 >>> /home/valera/petsc/src/sys/error/errtrace.c >>> [1]PETSC ERROR: [1] PetscError line 352 /home/valera/petsc/src/sys/err >>> or/err.c >>> [1]PETSC ERROR: [1] PCSetUp_SAVIENNACL line 45 >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu < >>> http://saviennacl.cu> >>> >>> [1]PETSC ERROR: [1] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [1]PETSC ERROR: Signal received >>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT >>> Date: 2018-05-31 17:31:13 +0300 >>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue >>> Aug 28 16:30:02 2018 >>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl >>> [1]PETSC ERROR: #1 User provided function() line 0 in unknown file >>> ------------------------------------------------------------ >>> -------------- >>> MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD >>> with errorcode 59. >>> >>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>> You may or may not see output from other processes, depending on >>> exactly when Open MPI kills them. >>> ------------------------------------------------------------ >>> -------------- >>> [0]PETSC ERROR: ------------------------------ >>> ------------------------------------------ >>> [0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the >>> batch system) has told this process to end >>> [0]PETSC ERROR: Try option -start_in_debugger or >>> -on_error_attach_debugger >>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>> ocumentation/faq.html#valgrind >>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac >>> OS X to find memory corruption errors >>> [0]PETSC ERROR: likely location of problem given in stack below >>> [0]PETSC ERROR: --------------------- Stack Frames >>> ------------------------------------ >>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not >>> available, >>> [0]PETSC ERROR: INSTEAD the line number of the start of the >>> function >>> [0]PETSC ERROR: is given. >>> [0]PETSC ERROR: [0] PetscCommDuplicate line 130 >>> /home/valera/petsc/src/sys/objects/tagm.c >>> [0]PETSC ERROR: [0] PetscHeaderCreate_Private line 34 >>> /home/valera/petsc/src/sys/objects/inherit.c >>> [0]PETSC ERROR: [0] ISCreate line 35 /home/valera/petsc/src/vec/is/ >>> is/interface/isreg.c >>> [0]PETSC ERROR: [0] ISCreateGeneral line 668 >>> /home/valera/petsc/src/vec/is/is/impls/general/general.c >>> [0]PETSC ERROR: [0] PCSetUp_SAVIENNACL line 45 >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu < >>> http://saviennacl.cu> >>> [0]PETSC ERROR: [0] PCSetUp line 894 /home/valera/petsc/src/ksp/pc/ >>> interface/precon.c >>> [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> [0]PETSC ERROR: Signal received >>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>> for trouble shooting. >>> [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 GIT >>> Date: 2018-05-31 17:31:13 +0300 >>> [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Tue >>> Aug 28 16:30:02 2018 >>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>> --download-viennacl >>> [0]PETSC ERROR: #3 User provided function() line 0 in unknown file >>> [node50:30582] 1 more process has sent help message help-mpi-api.txt / >>> mpi-abort >>> [node50:30582] Set MCA parameter "orte_base_help_aggregate" to 0 to see >>> all help / error messages >>> >>> >>> >>> It is currently running in 1 mpi processor + GPU but i would like to >>> call at least 16 mpi processors + GPU to do the rest of the data management >>> who is not part of the main laplacian on the mpi and the laplacian solution >>> on the GPU, is this currently possible? >>> >>> Thanks for your help, >>> >>> >>> >>> On Tue, Aug 28, 2018 at 4:21 PM, Manuel Valera <mvaler...@sdsu.edu >>> <mailto:mvaler...@sdsu.edu>> wrote: >>> >>> Ok, i found the culprit and we can close this thread, >>> >>> The problem was a missing variable for setting the maximum columns, >>> which i deleted at some point without realizing. The error message >>> was too ambiguous to catch this so i had to compare with a previous >>> working version of the arguments of MatSetValues, it was evident >>> then. >>> >>> Good news is that i can now set the values with the viennacl types >>> too, >>> >>> Thanks for your kind help, >>> >>> Manuel >>> >>> On Tue, Aug 28, 2018 at 11:25 AM, Smith, Barry F. >>> <bsm...@mcs.anl.gov <mailto:bsm...@mcs.anl.gov>> wrote: >>> >>> >>> 1) PetscMalloc() is never valid or needed in Fortran >>> >>> 2) there is no reason to use DMSetMatrixPreallocateOnly() >>> just use DMCreateMatrix() assuming that using a DM (DMDA, >>> DMPLEX, etc) is suitable for your problem. >>> >>> At this end we are totally guessing at what you are doing >>> and so have little help we can provide. A simple, nonworking >>> code that tries to do what you would need would help us a great >>> deal in understanding that you are trying to do. >>> >>> Barry >>> >>> >>> >>> >>> >>> > On Aug 28, 2018, at 1:18 PM, Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > >>> > Matthew, PetscMalloc gives the same error, >>> > >>> > Barry, it would be very hard for me to get the code to a >>> minimum working example, i guess all i need to understand is how >>> to setup a DM matrix with DMSetMatrixPreallocateOnly() instead >>> of MatMPIAIJSetPreallocation() as we were doing before, is there >>> a simple example who does this in Fortran? >>> > >>> > Is the PetscMalloc call needed? is 'call >>> PetscMalloc(1,row,ierr)' a valid, compilable call to >>> PetscMalloc? what other reason may there be for this error to >>> happen ? >>> > >>> > Just remembering, that trying to setup the matrix with the >>> MatAIJSetPreallocation() brings up an error to acknowledge the >>> viennacl datatypes and that's why i'm trying to make this change >>> on your recommendation, >>> > >>> > Thanks for your help, >>> > >>> > >>> > >>> > >>> > >>> > On Mon, Aug 27, 2018 at 7:35 PM, Smith, Barry F. >>> <bsm...@mcs.anl.gov <mailto:bsm...@mcs.anl.gov>> wrote: >>> > >>> > Send your code in a way we can compile and run it; it must >>> be some simple issue that is hard to communicate in email. >>> > >>> > Barry >>> > >>> > >>> > > On Aug 27, 2018, at 5:51 PM, Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > >>> > > Hello everyone, >>> > > >>> > > I just had time to work on this again, and checked the code >>> for errors on the matrix entries, this is the exact code i was >>> using for creating the matrix without >>> DMSetMatrixPreallocateOnly, using MatMPIAIJSetPreallocation and >>> it worked that way, but trying this way i get the same 'Column >>> too large' error using any number at the column position of >>> MatSetValues, >>> > > >>> > > I have set up my code to print the column argument (n) of >>> MatSetValues and in this case is 7 (lower than 124), it still >>> gives error, even entering a specific number in the MatSetValues >>> column argument position gives the same error. >>> > > >>> > > So next i went back to ex.50 here: >>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/t >>> utorials/ex50.c.html >>> <http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/ >>> tutorials/ex50.c.html> >>> and it has a very similar structure except the PetscMalloc1() >>> call, so i tried adding that and got: >>> > > >>> > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: undefined >>> reference to `petscmalloc1_' >>> > > >>> > > Any ideas on this behaviour? >>> > > >>> > > Thanks so much, >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F. >>> <bsm...@mcs.anl.gov <mailto:bsm...@mcs.anl.gov>> wrote: >>> > > >>> > > Column too large: col 10980 max 124 >>> > > >>> > > You need to check the code that is generating the matrix >>> entries. The matrix has 124 columns but you are attempting to >>> put a value at column 10980 >>> > > >>> > > Barry >>> > > >>> > > >>> > > > On Aug 15, 2018, at 9:44 PM, Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > > >>> > > > Thanks Matthew and Barry, >>> > > > >>> > > > Now my code looks like: >>> > > > >>> > > > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr) >>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>> > > > call DMCreateMatrix(daDummy,A,ierr) >>> > > > call MatSetFromOptions(A,ierr) >>> > > > call MatSetUp(A,ierr) >>> > > > [...] >>> > > > call >>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS >>> ERT_VALUES,ierr) >>> > > > [...] >>> > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >>> > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >>> > > > >>> > > > And i get a different error, now is: >>> > > > >>> > > > [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > > > [0]PETSC ERROR: Argument out of range >>> > > > [0]PETSC ERROR: Column too large: col 10980 max 124 >>> > > > [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>> trouble shooting. >>> > > > [0]PETSC ERROR: Petsc Development GIT revision: >>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>> > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 >>> by valera Wed Aug 15 19:40:00 2018 >>> > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 >>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>> --with-blaslapack-dir=/usr/lib64 --download-viennacl >>> > > > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442 in >>> /home/valera/petsc/src/mat/impls/aij/seq/aij.c >>> > > > [0]PETSC ERROR: #2 MatSetValues() line 1339 in >>> /home/valera/petsc/src/mat/interface/matrix.c >>> > > > >>> > > > >>> > > > Thanks again, >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F. >>> <bsm...@mcs.anl.gov <mailto:bsm...@mcs.anl.gov>> wrote: >>> > > > >>> > > > Should be >>> > > > >>> > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>> > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>> > > > call DMCreateMatrix(daDummy,A,ierr) >>> > > > >>> > > > and remove the rest. You need to set the type of Mat >>> you want the DM to return BEFORE you create the matrix. >>> > > > >>> > > > Barry >>> > > > >>> > > > >>> > > > >>> > > > > On Aug 15, 2018, at 4:45 PM, Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > > > >>> > > > > Ok thanks for clarifying that, i wasn't sure if there >>> were different types, >>> > > > > >>> > > > > Here is a stripped down version of my code, it seems >>> like the preallocation is working now since the matrix >>> population part is working without problem, but here it is for >>> illustration purposes: >>> > > > > >>> > > > > call DMSetMatrixPreallocateOnly(daD >>> ummy,PETSC_TRUE,ierr) >>> > > > > call DMCreateMatrix(daDummy,A,ierr) >>> > > > > call MatSetFromOptions(A,ierr) >>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>> > > > > call >>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N >>> ULL_INTEGER,ierr) >>> > > > > call MatSetUp(A,ierr) >>> > > > > [...] >>> > > > > call >>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS >>> ERT_VALUES,ierr) >>> > > > > [...] >>> > > > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >>> > > > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >>> > > > > >>> > > > > Adding the first line there did the trick, >>> > > > > >>> > > > > Now the problem seems to be the program is not >>> recognizing the matrix as ViennaCL type when i try with more >>> than one processor, i get now: >>> > > > > >>> > > > > [0]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > > > > [0]PETSC ERROR: No support for this operation for this >>> object type >>> > > > > [0]PETSC ERROR: Currently only handles ViennaCL matrices >>> > > > > [0]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>> trouble shooting. >>> > > > > [0]PETSC ERROR: Petsc Development GIT revision: >>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>> > > > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named >>> node50 by valera Wed Aug 15 14:44:22 2018 >>> > > > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 >>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>> --with-blaslapack-dir=/usr/lib64 --download-viennacl >>> > > > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >>> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >>> <http://saviennacl.cu> >>> > > > > [0]PETSC ERROR: #2 PCSetUp() line 932 in >>> /home/valera/petsc/src/ksp/pc/interface/precon.c >>> > > > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in >>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>> > > > > >>> > > > > When running with: >>> > > > > >>> > > > > mpirun -n 1 ./gcmLEP.GPU >>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2 >>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl >>> -pc_type saviennacl -log_view >>> > > > > >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley >>> <knep...@gmail.com <mailto:knep...@gmail.com>> wrote: >>> > > > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > > > It seems to be resumed on: I do not know how to >>> preallocate a DM Matrix correctly. >>> > > > > >>> > > > > There is only one matrix type, Mat. There are no >>> separate DM matrices. A DM can create a matrix for you >>> > > > > using DMCreateMatrix(), but that is a Mat and it is >>> preallocated correctly. I am not sure what you are doing. >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > Matt >>> > > > > >>> > > > > The interesting part is that it only breaks when i need >>> to populate a GPU matrix from MPI, so kudos on that, but it >>> seems i need to do better on my code to get this setup working, >>> > > > > >>> > > > > Any help would be appreciated, >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > >>> > > > > >>> > > > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley >>> <knep...@gmail.com <mailto:knep...@gmail.com>> wrote: >>> > > > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > > > Thanks Matthew, >>> > > > > >>> > > > > I try to do that when calling: >>> > > > > >>> > > > > call >>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N >>> ULL_INTEGER,ierr) >>> > > > > >>> > > > > But i am not aware on how to do this for the DM if it >>> needs something more specific/different, >>> > > > > >>> > > > > The error says that your preallocation is wrong for the >>> values you are putting in. The DM does not control either, >>> > > > > so I do not understand your email. >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > Matt >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley >>> <knep...@gmail.com <mailto:knep...@gmail.com>> wrote: >>> > > > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera >>> <mvaler...@sdsu.edu <mailto:mvaler...@sdsu.edu>> wrote: >>> > > > > Hello PETSc devs, >>> > > > > >>> > > > > I am running into an error when trying to use the >>> MATMPIAIJVIENNACL Matrix type in MPI calls, the same code runs >>> for MATSEQAIJVIENNACL type in one processor. The error happens >>> when calling MatSetValues for this specific configuration. It >>> does not occur when using MPI DMMatrix types only. >>> > > > > >>> > > > > The DM properly preallocates the matrix. I am assuming >>> you do not here. >>> > > > > >>> > > > > Matt >>> > > > > >>> > > > > Any help will be appreciated, >>> > > > > >>> > > > > Thanks, >>> > > > > >>> > > > > >>> > > > > >>> > > > > My program call: >>> > > > > >>> > > > > mpirun -n 2 ./gcmLEP.GPU >>> tc=TestCases/LockRelease/LE_6x6x6/ jid=tiny_cuda_test_n2 >>> -ksp_type cg -dm_vec_type viennacl -dm_mat_type aijviennacl >>> -pc_type saviennacl -log_view >>> > > > > >>> > > > > >>> > > > > The error (repeats after each MatSetValues call): >>> > > > > >>> > > > > [1]PETSC ERROR: --------------------- Error Message >>> -------------------------------------------------------------- >>> > > > > [1]PETSC ERROR: Argument out of range >>> > > > > [1]PETSC ERROR: Inserting a new nonzero at global >>> row/column (75, 50) into matrix >>> > > > > [1]PETSC ERROR: See >>> http://www.mcs.anl.gov/petsc/documentation/faq.html >>> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for >>> trouble shooting. >>> > > > > [1]PETSC ERROR: Petsc Development GIT revision: >>> v3.9.2-549-g779ab53 GIT Date: 2018-05-31 17:31:13 +0300 >>> > > > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named >>> node50 by valera Wed Aug 15 13:10:44 2018 >>> > > > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 >>> --CXXOPTFLAGS=-O2 --FOPTFLAGS=-O2 --with-shared-libraries=1 >>> --with-debugging=1 --with-cuda=1 --CUDAFLAGS=-arch=sm_60 >>> --with-blaslapack-dir=/usr/lib64 --download-viennacl >>> > > > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in >>> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c >>> > > > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in >>> /home/valera/petsc/src/mat/interface/matrix.c >>> > > > > >>> > > > > >>> > > > > My Code structure: >>> > > > > >>> > > > > call DMCreateMatrix(daDummy,A,ierr) >>> > > > > call MatSetFromOptions(A,ierr) >>> > > > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr) >>> > > > > if (numprocs > 1) then ! set matrix type parallel >>> > > > > ! Get local size >>> > > > > call DMDACreateNaturalVector(daDummy,Tmpnat,ierr) >>> > > > > call VecGetLocalSize(Tmpnat,locsize,ierr) >>> > > > > call VecDestroy(Tmpnat,ierr) >>> > > > > ! Set matrix >>> > > > > #ifdef GPU >>> > > > > call MatSetType(A,MATAIJVIENNACL,ierr) >>> > > > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>> > > > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>> > > > > print*,'SETTING GPU TYPES' >>> > > > > #else >>> > > > > call DMSetMatType(daDummy,MATMPIAIJ,ierr) >>> > > > > call DMSetMatType(daDummy,VECMPI,ierr) >>> > > > > call MatSetType(A,MATMPIAIJ,ierr)! >>> > > > > #endif >>> > > > > call >>> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_N >>> ULL_INTEGER,ierr) >>> > > > > else ! set matrix type sequential >>> > > > > #ifdef GPU >>> > > > > call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr) >>> > > > > call DMSetVecType(daDummy,VECSEQVIENNACL,ierr) >>> > > > > call MatSetType(A,MATSEQAIJVIENNACL,ierr) >>> > > > > print*,'SETTING GPU TYPES' >>> > > > > #else >>> > > > > call DMSetMatType(daDummy,MATSEQAIJ,ierr) >>> > > > > call DMSetMatType(daDummy,VECSEQ,ierr) >>> > > > > call MatSetType(A,MATSEQAIJ,ierr) >>> > > > > #endif >>> > > > > call MatSetUp(A,ierr) >>> > > > > call >>> getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >>> > > > > >>> > > > > do k=zstart,zend-1 >>> > > > > do j=ystart,yend-1 >>> > > > > do i=xstart,xend-1 >>> > > > > [..] >>> > > > > call >>> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INS >>> ERT_VALUES,ierr) >>> > > > > [..] >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > What most experimenters take for granted before they >>> begin their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> > > > > -- Norbert Wiener >>> > > > > >>> > > > > https://www.cse.buffalo.edu/~knepley/ >>> <https://www.cse.buffalo.edu/%7Eknepley/> >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > What most experimenters take for granted before they >>> begin their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> > > > > -- Norbert Wiener >>> > > > > >>> > > > > https://www.cse.buffalo.edu/~knepley/ >>> <https://www.cse.buffalo.edu/%7Eknepley/> >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > What most experimenters take for granted before they >>> begin their experiments is infinitely more interesting than any >>> results to which their experiments lead. >>> > > > > -- Norbert Wiener >>> > > > > >>> > > > > https://www.cse.buffalo.edu/~knepley/ >>> <https://www.cse.buffalo.edu/%7Eknepley/> >>> > > > > >>> > > > >>> > > > >>> > > >>> > > >>> > >>> > >>> >>> >>> >>> >