It seems to be resumed on: I do not know how to preallocate a DM Matrix correctly.
The interesting part is that it only breaks when i need to populate a GPU matrix from MPI, so kudos on that, but it seems i need to do better on my code to get this setup working, Any help would be appreciated, Thanks, On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley <knep...@gmail.com> wrote: > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera <mvaler...@sdsu.edu> wrote: > >> Thanks Matthew, >> >> I try to do that when calling: >> >> call MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_ >> NULL_INTEGER,ierr) >> >> But i am not aware on how to do this for the DM if it needs something >> more specific/different, >> > > The error says that your preallocation is wrong for the values you are > putting in. The DM does not control either, > so I do not understand your email. > > Thanks, > > Matt > > >> Thanks, >> >> On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley <knep...@gmail.com> >> wrote: >> >>> On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera <mvaler...@sdsu.edu> >>> wrote: >>> >>>> Hello PETSc devs, >>>> >>>> I am running into an error when trying to use the MATMPIAIJVIENNACL >>>> Matrix type in MPI calls, the same code runs for MATSEQAIJVIENNACL type in >>>> one processor. The error happens when calling MatSetValues for this >>>> specific configuration. It does not occur when using MPI DMMatrix types >>>> only. >>>> >>> >>> The DM properly preallocates the matrix. I am assuming you do not here. >>> >>> Matt >>> >>> >>>> Any help will be appreciated, >>>> >>>> Thanks, >>>> >>>> >>>> >>>> My program call: >>>> >>>> mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ >>>> jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type >>>> aijviennacl -pc_type saviennacl -log_view >>>> >>>> >>>> The error (repeats after each MatSetValues call): >>>> >>>> [1]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [1]PETSC ERROR: Argument out of range >>>> [1]PETSC ERROR: Inserting a new nonzero at global row/column (75, 50) >>>> into matrix >>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 >>>> GIT Date: 2018-05-31 17:31:13 +0300 >>>> [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed >>>> Aug 15 13:10:44 2018 >>>> [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >>>> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >>>> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 >>>> --download-viennacl >>>> [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in >>>> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c >>>> [1]PETSC ERROR: #2 MatSetValues() line 1339 in >>>> /home/valera/petsc/src/mat/interface/matrix.c >>>> >>>> >>>> My Code structure: >>>> >>>> call DMCreateMatrix(daDummy,A,ierr) >>>>> call MatSetFromOptions(A,ierr) >>>>> call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr) >>>>> if (numprocs > 1) then ! set matrix type parallel >>>>> ! Get local size >>>>> call DMDACreateNaturalVector(daDummy,Tmpnat,ierr) >>>>> call VecGetLocalSize(Tmpnat,locsize,ierr) >>>>> call VecDestroy(Tmpnat,ierr) >>>>> ! Set matrix >>>>> #ifdef GPU >>>>> call MatSetType(A,MATAIJVIENNACL,ierr) >>>>> call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >>>>> call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >>>>> print*,'SETTING GPU TYPES' >>>>> #else >>>>> call DMSetMatType(daDummy,MATMPIAIJ,ierr) >>>>> call DMSetMatType(daDummy,VECMPI,ierr) >>>>> call MatSetType(A,MATMPIAIJ,ierr)! >>>>> #endif >>>>> call MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19, >>>>> PETSC_NULL_INTEGER,ierr) >>>>> else ! set matrix type sequential >>>>> #ifdef GPU >>>>> call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr) >>>>> call DMSetVecType(daDummy,VECSEQVIENNACL,ierr) >>>>> call MatSetType(A,MATSEQAIJVIENNACL,ierr) >>>>> print*,'SETTING GPU TYPES' >>>>> #else >>>>> call DMSetMatType(daDummy,MATSEQAIJ,ierr) >>>>> call DMSetMatType(daDummy,VECSEQ,ierr) >>>>> call MatSetType(A,MATSEQAIJ,ierr) >>>>> #endif >>>>> call MatSetUp(A,ierr) >>>>> call getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >>>>> >>>> >>>> >>>>> do k=zstart,zend-1 >>>>> do j=ystart,yend-1 >>>>> do i=xstart,xend-1 >>>>> [..] >>>>> call MatSetValues(A,1,row,sumpos, >>>>> pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >>>>> [..] >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >>> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >