I think I found the/a problem. It is failing on the current test when it adds equation 144 to GASM. This test has a 3^3 cell mesh so a 4^3 vertex mesh. There one face with BCs, that means there is a 4x4x3 mesh. With bs==3, there are 144 equations, equation number 144 is the first out-of-bounds equation. This seems like a suspicious coincidence.
It has been a long time since I've looked that code but I also see that this test has two aggregates and the IS's that I give GASM have 54 and 90 equations in them. So it looks like I am taking out the BCs. [I just got your new message ...] Mark On Fri, Jun 24, 2016 at 4:52 PM, Mark Adams <mfad...@lbl.gov> wrote: > OK, but how should I proceed? > > On Fri, Jun 24, 2016 at 4:47 PM, Barry Smith <bsm...@mcs.anl.gov> wrote: > >> >> > On Jun 24, 2016, at 9:35 AM, Mark Adams <mfad...@lbl.gov> wrote: >> > >> > Barry, >> > >> > ASM worked before GASM existed. I've never heard of GASM until >> yesterday, other than seeing the directory in the source. >> >> Pretty impressive you never heard of GASM and yet YOU put the GASM >> code into GAMG >> >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 604) >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 605) /* set defaults */ >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 606) ierr = KSPSetType(smoother, >> KSPCHEBYSHEV);CHKERRQ(ierr); >> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-03-27 19:09:54 -0400 607) >> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2013-11-02 15:56:24 -0400 608) /* set blocks for GASM smoother that >> uses the 'aggregates' */ >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 609) if (pc_gamg->use_aggs_in_gasm) { >> 2d3561bba src/ksp/pc/impls/gamg/gamg.c (Satish Balay >> 2012-03-29 12:58:42 -0500 610) PetscInt sz; >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 611) IS *is; >> a2f3521de src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-05-01 12:48:43 -0400 612) >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 613) sz = nASMBlocksArr[level]; >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 614) is = ASMLocalIDsArr[level]; >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 615) ierr = PCSetType(subpc, >> PCGASM);CHKERRQ(ierr); >> 1b18a24a4 src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2013-11-02 15:56:24 -0400 616) ierr = PCGASMSetOverlap(subpc, >> 0);CHKERRQ(ierr); >> 7f66b68fb src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2015-08-11 09:52:40 -0400 617) if (!sz) { >> 2fa5cd679 src/ksp/pc/impls/gamg/gamg.c (Karl Rupp >> 2013-01-30 22:53:06 -0600 618) IS is; >> ffc955d6d src/ksp/pc/impls/gamg/gamg.c (Mark Adams >> 2012-03-27 19:09:54 -0400 619) PetscInt my0,kk; >> 806fa848a src/ksp/pc/impls/gamg/gamg.c (Barry Smith >> 2013-01-02 02:05:45 -0600 620) ierr = >> MatGetOwnershipRange(Aarr[level], &my0 >> >> I think you put in the GASM but then forgot and thought you put in ASM. >> >> > It has been so long, I could probably hunt for a working version, but >> who cares, we are going to roll back 5 years? >> > >> > >> > > In branch mark/gamg-agg-asm in ksp ex56, 'make runex56': >> > >> > I don't care about this! This is where you have tried to change from >> GASM to ASM which I told you is non-trivial. >> > Give me the example and command line where the GASM version in master >> (or maint) doesn't work where the error message includes ** Max-trans not >> allowed because matrix is distributed >> > >> > Why do you want to debug GASM? I thought we wanted to use ASM. It is >> not clear why converting from GASM to ASM is non-trivial, I could have done >> this from the command line. This use aggregate-asm stuff just does what a >> clever user might do. >> > >> > >> > We are not communicating very well, you jumped from stating GASM >> crashed to monkeying with ASM and now refuse to tell me how to reproduce >> the GASM crash. >> > >> > I don't think you want to mess with GASM, BUT if you run ksp/ex56 with: >> > >> > -pc_gamg_use_agg_gasm -mg_levels_pc_type gasm >> > >> > and remove the existing -mg_levels_pc_type sor, you should see an error >> that Garth and I saw (eg, appended). >> > >> > >> > We have to start by fixing the current code to work with GASM (if it >> ever worked) and then move on to using ASM (which is just an optimization >> of the GASM usage.) >> > >> > >> > Are you sure? I am deep into debugging asm with DDT. I can switch to >> GASM by just calling the GASM set block method and use gasm in the PC, and >> continue with DDT debugging. >> > >> > Let me know. >> > >> > >> > 16:29 master *= ~/Codes/petsc/src/ksp/ksp/examples/tutorials$ make >> runex56 >> > 1,4c1,85 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < Linear solve converged due to CONVERGED_RTOL iterations 8 >> > < [0]main |b-Ax|/|b|=1.940043e-04, |b|=4.969822e+00, emax=9.926090e-01 >> > --- >> > > [1]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [1]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [1]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [1]PETSC ERROR: [2]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [2]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [2]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [2]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [6]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [6]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [6]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [6]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [6]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [4]PETSC ERROR: >> ------------------------------------------------------------------------ >> > > [4]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, >> probably memory access out of range >> > > [4]PETSC ERROR: Try option -start_in_debugger or >> -on_error_attach_debugger >> > > [4]PETSC ERROR: or see >> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >> > > [4]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple >> Mac OS X to find memory corruption errors >> > > [6]PETSC ERROR: likely location of problem given in stack below >> > > [6]PETSC ERROR: --------------------- Stack Frames >> ------------------------------------ >> > > [6]PETSC ERROR: Note: The EXACT line numbers in the stack are not >> available, >> > > [6]PETSC ERROR: INSTEAD the line number of the start of the >> function >> > > [6]PETSC ERROR: is given. >> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ_Local line 1228 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatrices_MPIAIJ line 1131 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIXAIJ line 2163 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI_MPIAIJ line 2418 >> /Users/markadams/Codes/petsc/src/mat/impls/aij/mpi/mpiov.c >> > > [6]PETSC ERROR: [6] MatGetSubMatricesMPI line 6805 >> /Users/markadams/Codes/petsc/src/mat/interface/matrix.c >> > > [6]PETSC ERROR: [6] PCSetUp_GASM line 357 >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/gasm/gasm.c >> > > [6]PETSC ERROR: [6] PCSetUp line 930 >> /Users/markadams/Codes/petsc/src/ksp/pc/interface/precon.c >> > > [6]PETSC ERROR: [6] KSPSetUp line 305 >> /Users/markadams/Codes/petsc/src/ksp/ksp/interface/itfunc.c >> > > [6]PETSC ERROR: [6] PCSetUp_MG line 561 >> /Users/markadams/Codes/petsc/src/ksp/pc/impls/mg/mg.c >> > > [6]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> > > [6]PETSC ERROR: Signal received >> >> >