On Mon, Aug 1, 2022 at 11:45 AM E. Ekici <ee...@cam.ac.uk> wrote: > Hi, > > I would like to preallocate my highly sparse matrices to get speed up > during assembling. > > Up to 70000x70000, it works fine in parallel. > > But when I exceed around 70000 rows/columns, I receive following error at > preallocation line; > > petsc4py.PETSc.Error: error code 4031 > [0] MatMPIAIJSetPreallocation() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:4039 > [0] MatMPIAIJSetPreallocation_MPIAIJ() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/mpi/mpiaij.c:2845 > [0] MatSeqAIJSetPreallocation() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:3963 > [0] MatSeqAIJSetPreallocation_SeqAIJ() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/mat/impls/aij/seq/aij.c:4031 > [0] PetscMallocA() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:401 > [0] PetscMallocAlign() at > /home/ekrem/Dev/Venvs/dolfinxcomplex/installation/petsc/src/sys/memory/mal.c:49 > [0] MatSeqAIJSetPreallocation_SeqAIJ > > I have attached my sparsity pattern as *CSR.png*. > > I use petsc4py for matrix generation, and here are the commands that I am > using to generate matrix; > > row # global row numbers > col = # global column numbers > val = # flattened 2D nonzero array > > mat = PETSc.Mat().create(PETSc.COMM_WORLD) > mat.setSizes([(local_size, global_size), (local_size, global_size)]) > mat.setType('mpiaij') > ONNZ = len(row)*np.ones(local_size,dtype=np.int32) >
Is len(row) == local_size? This looks like it is allocating a dense diagonal block and a semi-dense off-diagonal block. Is that true? If so, you are probably running out of memory. Thanks, Matt > mat.setPreallocationNNZ([ONNZ,ONNZ]) # This line throws the error above > mat.setOption(PETSc.Mat.Option.NEW_NONZERO_ALLOCATION_ERR, False) > mat.setUp() > mat.setValues(row, col, val, addv=PETSc.InsertMode.ADD_VALUES) > mat.assemblyBegin() > mat.assemblyEnd() > > Many thanks for your response in advance, > > Kind regards > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>