On Thu, Sep 28, 2023 at 4:09 PM Jaysaval, Piyoosh via petsc-dev < petsc-dev@mcs.anl.gov> wrote:
> Hello PETSc developers, > > > > I am having some issues with using MUMPS solver after the solution phase > from PETSc. I am solving a matrix equation with about 2.17 million DOFs and > for 1764 RHSs. MUMPS successfully solves the system for all RHS; however, > after the solve phase the distributed solution from MUMPS vector/matrix is > scattered back to PETSc mpi vector (done within PETSc) and this is where I > am getting the error. > > > > FYI, when I use 1323 RHSs, there is no issue. Moreover, I had to use > v3.18.6 (or lower) because of some compiling issue with v3.19+ and > SuperLU_DIST on our cluster with older intel mpi. > > > > Any help is greatly appreciated. Thanks. > Have you configured with 64-bit indices? Thanks, Matt > Piyoosh > > > > Here’s is the error message I am getting: > > > > Entering ZMUMPS 5.5.1 from C interface with JOB, N = 3 2165687 > > executing #MPI = 63 and #OMP = 1 > > > > > > > > ****** SOLVE & CHECK STEP ******** > > > > GLOBAL STATISTICS PRIOR SOLVE PHASE ........... > > Number of right-hand-sides = 1764 > > Blocking factor for multiple rhs = 32 > > ICNTL (9) = 1 > > --- (10) = 0 > > --- (11) = 0 > > --- (20) = 1 > > --- (21) = 1 > > --- (30) = 0 > > --- (35) = 0 > > ** Rank of processor needing largest memory in solve : 5 > > ** Space in MBYTES used by this processor for solve : 1173 > > ** Avg. Space in MBYTES per working proc during solve : 820 > > > > Leaving solve with ... > > Time to build/scatter RHS = 0.130768 > > Time in solution step (fwd/bwd) = 41.264175 > > .. Time in forward (fwd) step = 11.261921 > > .. Time in ScaLAPACK root = 3.584752 > > .. Time in backward (bwd) step = 26.515026 > > Time to gather solution(cent.sol)= 0.000000 > > Time to copy/scale dist. solution= 0.724149 > > > > Elapsed time in solve driver= 42.3461 > > > > [23]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [23]PETSC ERROR: Argument out of range > > [23]PETSC ERROR: Scatter indices in iy are out of range > > [23]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting. > > [23]PETSC ERROR: Petsc Release Version 3.18.6, unknown > > [23]PETSC ERROR: /people/jays242/softwares/pgemini-em/pgemini/src/pgemini > on a arch-linux-intel-opt-v3.18.6 named dc230.local by jays242 Thu Sep 28 > 10:44:51 2023 > > [23]PETSC ERROR: Configure options PETSC_ARCH=arch-linux-intel-opt-v3.18.6 > --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 > --with-cc=mpiicc --with-cxx=mpiicpc --with-fc=mpiifort > --with-blaslapack-dir=/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 > -with-scalar-type=complex --download-mumps --download-metis --with-openmp > --download-parmetis --download-superlu_dist > --with-scalapack-lib="-L/share/apps/intel/2020u4/compilers_and_libraries_2020.4.304/linux/mkl/lib/intel64 > -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" > > [23]PETSC ERROR: #1 VecScatterCreate() at > /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/vec/is/sf/interface/vscat.c:736 > > [23]PETSC ERROR: #2 MatMatSolve_MUMPS() at > /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1449 > > [23]PETSC ERROR: #3 MatMatTransposeSolve_MUMPS() at > /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/impls/aij/mpi/mumps/mumps.c:1506 > > [23]PETSC ERROR: #4 MatMatTransposeSolve() at > /qfs/people/jays242/softwares/pgemini-em/petsc-intel/src/mat/interface/matrix.c:3789 > > [24]PETSC ERROR: --------------------- Error Message --------------- > > > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>