On Mon, Aug 27, 2018 at 6:51 PM Manuel Valera <mvaler...@sdsu.edu> wrote:
> Hello everyone, > > I just had time to work on this again, and checked the code for errors on > the matrix entries, this is the exact code i was using for creating the > matrix without DMSetMatrixPreallocateOnly, using MatMPIAIJSetPreallocation > and it worked that way, but trying this way i get the same 'Column too > large' error using any number at the column position of MatSetValues, > > I have set up my code to print the column argument (n) of MatSetValues and > in this case is 7 (lower than 124), it still gives error, even entering a > specific number in the MatSetValues column argument position gives the same > error. > > So next i went back to ex.50 here: > http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/ex50.c.html > and it has a very similar structure except the PetscMalloc1() call, so i > tried adding that and got: > > /home/valera/ParGCCOM/Src/DMDALaplacian.f90:114: undefined reference to > `petscmalloc1_' > > Any ideas on this behaviour? > This is a macro. I am guessing you are using Fortran. Just use PetscMalloc instead. Thanks, Matt > Thanks so much, > > > > > > > On Thu, Aug 16, 2018 at 11:20 AM, Smith, Barry F. <bsm...@mcs.anl.gov> > wrote: > >> >> Column too large: col 10980 max 124 >> >> You need to check the code that is generating the matrix entries. The >> matrix has 124 columns but you are attempting to put a value at column 10980 >> >> Barry >> >> >> > On Aug 15, 2018, at 9:44 PM, Manuel Valera <mvaler...@sdsu.edu> wrote: >> > >> > Thanks Matthew and Barry, >> > >> > Now my code looks like: >> > >> > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr) >> > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> > call DMCreateMatrix(daDummy,A,ierr) >> > call MatSetFromOptions(A,ierr) >> > call MatSetUp(A,ierr) >> > [...] >> > call >> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >> > [...] >> > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >> > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >> > >> > And i get a different error, now is: >> > >> > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > [0]PETSC ERROR: Argument out of range >> > [0]PETSC ERROR: Column too large: col 10980 max 124 >> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> > [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 >> GIT Date: 2018-05-31 17:31:13 +0300 >> > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera Wed >> Aug 15 19:40:00 2018 >> > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl >> > [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 442 in >> /home/valera/petsc/src/mat/impls/aij/seq/aij.c >> > [0]PETSC ERROR: #2 MatSetValues() line 1339 in >> /home/valera/petsc/src/mat/interface/matrix.c >> > >> > >> > Thanks again, >> > >> > >> > >> > >> > >> > >> > >> > >> > On Wed, Aug 15, 2018 at 7:02 PM, Smith, Barry F. <bsm...@mcs.anl.gov> >> wrote: >> > >> > Should be >> > >> > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> > call DMCreateMatrix(daDummy,A,ierr) >> > >> > and remove the rest. You need to set the type of Mat you want the DM >> to return BEFORE you create the matrix. >> > >> > Barry >> > >> > >> > >> > > On Aug 15, 2018, at 4:45 PM, Manuel Valera <mvaler...@sdsu.edu> >> wrote: >> > > >> > > Ok thanks for clarifying that, i wasn't sure if there were different >> types, >> > > >> > > Here is a stripped down version of my code, it seems like the >> preallocation is working now since the matrix population part is working >> without problem, but here it is for illustration purposes: >> > > >> > > call DMSetMatrixPreallocateOnly(daDummy,PETSC_TRUE,ierr) >> > > call DMCreateMatrix(daDummy,A,ierr) >> > > call MatSetFromOptions(A,ierr) >> > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> > > call >> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >> > > call MatSetUp(A,ierr) >> > > [...] >> > > call >> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >> > > [...] >> > > call MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY, ierr) >> > > call MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY, ierr) >> > > >> > > Adding the first line there did the trick, >> > > >> > > Now the problem seems to be the program is not recognizing the matrix >> as ViennaCL type when i try with more than one processor, i get now: >> > > >> > > [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > > [0]PETSC ERROR: No support for this operation for this object type >> > > [0]PETSC ERROR: Currently only handles ViennaCL matrices >> > > [0]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > > [0]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 >> GIT Date: 2018-05-31 17:31:13 +0300 >> > > [0]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera >> Wed Aug 15 14:44:22 2018 >> > > [0]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl >> > > [0]PETSC ERROR: #1 PCSetUp_SAVIENNACL() line 47 in >> /home/valera/petsc/src/ksp/pc/impls/saviennaclcuda/saviennacl.cu >> > > [0]PETSC ERROR: #2 PCSetUp() line 932 in >> /home/valera/petsc/src/ksp/pc/interface/precon.c >> > > [0]PETSC ERROR: #3 KSPSetUp() line 381 in >> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >> > > >> > > When running with: >> > > >> > > mpirun -n 1 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ >> jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type >> aijviennacl -pc_type saviennacl -log_view >> > > >> > > >> > > Thanks, >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > On Wed, Aug 15, 2018 at 2:32 PM, Matthew Knepley <knep...@gmail.com> >> wrote: >> > > On Wed, Aug 15, 2018 at 5:20 PM Manuel Valera <mvaler...@sdsu.edu> >> wrote: >> > > It seems to be resumed on: I do not know how to preallocate a DM >> Matrix correctly. >> > > >> > > There is only one matrix type, Mat. There are no separate DM >> matrices. A DM can create a matrix for you >> > > using DMCreateMatrix(), but that is a Mat and it is preallocated >> correctly. I am not sure what you are doing. >> > > >> > > Thanks, >> > > >> > > Matt >> > > >> > > The interesting part is that it only breaks when i need to populate a >> GPU matrix from MPI, so kudos on that, but it seems i need to do better on >> my code to get this setup working, >> > > >> > > Any help would be appreciated, >> > > >> > > Thanks, >> > > >> > > >> > > >> > > On Wed, Aug 15, 2018 at 2:15 PM, Matthew Knepley <knep...@gmail.com> >> wrote: >> > > On Wed, Aug 15, 2018 at 4:53 PM Manuel Valera <mvaler...@sdsu.edu> >> wrote: >> > > Thanks Matthew, >> > > >> > > I try to do that when calling: >> > > >> > > call >> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >> > > >> > > But i am not aware on how to do this for the DM if it needs something >> more specific/different, >> > > >> > > The error says that your preallocation is wrong for the values you >> are putting in. The DM does not control either, >> > > so I do not understand your email. >> > > >> > > Thanks, >> > > >> > > Matt >> > > >> > > Thanks, >> > > >> > > On Wed, Aug 15, 2018 at 1:51 PM, Matthew Knepley <knep...@gmail.com> >> wrote: >> > > On Wed, Aug 15, 2018 at 4:39 PM Manuel Valera <mvaler...@sdsu.edu> >> wrote: >> > > Hello PETSc devs, >> > > >> > > I am running into an error when trying to use the MATMPIAIJVIENNACL >> Matrix type in MPI calls, the same code runs for MATSEQAIJVIENNACL type in >> one processor. The error happens when calling MatSetValues for this >> specific configuration. It does not occur when using MPI DMMatrix types >> only. >> > > >> > > The DM properly preallocates the matrix. I am assuming you do not >> here. >> > > >> > > Matt >> > > >> > > Any help will be appreciated, >> > > >> > > Thanks, >> > > >> > > >> > > >> > > My program call: >> > > >> > > mpirun -n 2 ./gcmLEP.GPU tc=TestCases/LockRelease/LE_6x6x6/ >> jid=tiny_cuda_test_n2 -ksp_type cg -dm_vec_type viennacl -dm_mat_type >> aijviennacl -pc_type saviennacl -log_view >> > > >> > > >> > > The error (repeats after each MatSetValues call): >> > > >> > > [1]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > > [1]PETSC ERROR: Argument out of range >> > > [1]PETSC ERROR: Inserting a new nonzero at global row/column (75, 50) >> into matrix >> > > [1]PETSC ERROR: See >> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. >> > > [1]PETSC ERROR: Petsc Development GIT revision: v3.9.2-549-g779ab53 >> GIT Date: 2018-05-31 17:31:13 +0300 >> > > [1]PETSC ERROR: ./gcmLEP.GPU on a cuda-debug named node50 by valera >> Wed Aug 15 13:10:44 2018 >> > > [1]PETSC ERROR: Configure options PETSC_ARCH=cuda-debug >> --with-mpi-dir=/usr/lib64/openmpi --COPTFLAGS=-O2 --CXXOPTFLAGS=-O2 >> --FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >> --CUDAFLAGS=-arch=sm_60 --with-blaslapack-dir=/usr/lib64 --download-viennacl >> > > [1]PETSC ERROR: #1 MatSetValues_MPIAIJ() line 608 in >> /home/valera/petsc/src/mat/impls/aij/mpi/mpiaij.c >> > > [1]PETSC ERROR: #2 MatSetValues() line 1339 in >> /home/valera/petsc/src/mat/interface/matrix.c >> > > >> > > >> > > My Code structure: >> > > >> > > call DMCreateMatrix(daDummy,A,ierr) >> > > call MatSetFromOptions(A,ierr) >> > > call MPI_Comm_size(PETSC_COMM_WORLD, numprocs, ierr) >> > > if (numprocs > 1) then ! set matrix type parallel >> > > ! Get local size >> > > call DMDACreateNaturalVector(daDummy,Tmpnat,ierr) >> > > call VecGetLocalSize(Tmpnat,locsize,ierr) >> > > call VecDestroy(Tmpnat,ierr) >> > > ! Set matrix >> > > #ifdef GPU >> > > call MatSetType(A,MATAIJVIENNACL,ierr) >> > > call DMSetMatType(daDummy,MATMPIAIJVIENNACL,ierr) >> > > call DMSetVecType(daDummy,VECMPIVIENNACL,ierr) >> > > print*,'SETTING GPU TYPES' >> > > #else >> > > call DMSetMatType(daDummy,MATMPIAIJ,ierr) >> > > call DMSetMatType(daDummy,VECMPI,ierr) >> > > call MatSetType(A,MATMPIAIJ,ierr)! >> > > #endif >> > > call >> MatMPIAIJSetPreallocation(A,19,PETSC_NULL_INTEGER,19,PETSC_NULL_INTEGER,ierr) >> > > else ! set matrix type sequential >> > > #ifdef GPU >> > > call DMSetMatType(daDummy,MATSEQAIJVIENNACL,ierr) >> > > call DMSetVecType(daDummy,VECSEQVIENNACL,ierr) >> > > call MatSetType(A,MATSEQAIJVIENNACL,ierr) >> > > print*,'SETTING GPU TYPES' >> > > #else >> > > call DMSetMatType(daDummy,MATSEQAIJ,ierr) >> > > call DMSetMatType(daDummy,VECSEQ,ierr) >> > > call MatSetType(A,MATSEQAIJ,ierr) >> > > #endif >> > > call MatSetUp(A,ierr) >> > > call getCenterInfo(daGrid,xstart,ystart,zstart,xend,yend,zend) >> > > >> > > do k=zstart,zend-1 >> > > do j=ystart,yend-1 >> > > do i=xstart,xend-1 >> > > [..] >> > > call >> MatSetValues(A,1,row,sumpos,pos(0:iter-1),vals(0:iter-1),INSERT_VALUES,ierr) >> > > [..] >> > > >> > > >> > > >> > > >> > > >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > > >> > > https://www.cse.buffalo.edu/~knepley/ >> > > >> > > >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > > >> > > https://www.cse.buffalo.edu/~knepley/ >> > > >> > > >> > > >> > > -- >> > > What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> > > -- Norbert Wiener >> > > >> > > https://www.cse.buffalo.edu/~knepley/ >> > > >> > >> > >> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
