Re: [petsc-users] KSP_Solve crashes in debug mode

[Barry Smith]

Hmm, it is crashing in PetscFunctionListDLAllPop_Private() which is very recent code with Jacob's name on it.   It is only run in debug mode, hence only crashes there. I don't know why it fails here, somehow memory was corrupted earlier? Try running with -malloc_debug or under valgrind? static PetscErrorCode PetscFunctionListDLAllPop_Private(PetscFunctionList fl) {   PetscFunctionBegin;   if (PetscDefined(USE_DEBUG) && !PetscDefined(HAVE_THREADSAFETY)) {     PetscFunctionListDLAll current = dlallhead, prev = NULL;     /* Remove this entry from the main DL list (if it is in it) */     while (current) {       const PetscFunctionListDLAll next = current->next;       if (current->data == fl) {         if (prev) {           // somewhere in the middle (or end) of the list           prev->next = next;         } else {           // prev = NULL implies current = dlallhead, so front of list           dlallhead = next;         }         PetscCall(PetscFree(current));         break;       }       prev    = current;       current = next;     }   }   PetscFunctionReturn(PETSC_SUCCESS); } On Feb 9, 2023, at 1:56 PM, Sajid Ali Syed <sas...@fnal.gov> wrote: Hi Barry,  The lack of line numbers is due to the fact that this build of PETSc was done via spack which installs it in a temporary directory before moving it to the final location.  I have removed that build and installed PETSc locally (albeit with a simpler configuration) and see the same bug. Logs for this configuration and the error trace with this build are attached with this email.  Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io From: Barry Smith <bsm...@petsc.dev> Sent: Thursday, February 9, 2023 12:02 PM To: Sajid Ali Syed <sas...@fnal.gov> Cc: petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode     Hmm, looks like your build may be funny? It is not in debug mode  frame #2: 0x000000010eda20c8 libpetsc.3.018.dylib`PetscHeaderDestroy_Private + 1436     frame #3: 0x000000010f10176c libpetsc.3.018.dylib`VecDestroy + 808     frame #4: 0x0000000110199f34 libpetsc.3.018.dylib`KSPSolve_Private + 512 In debugger mode it would show the line numbers where the crash occurred and help us determine the problem. I do note the -g being used by the compilers so cannot explain off hand why it does not display the debug information.   Barry On Feb 9, 2023, at 12:42 PM, Sajid Ali Syed via petsc-users <petsc-users@mcs.anl.gov> wrote: Hi PETSc-developers,  In our application we call KSP_Solve as part of a step to propagate a beam through a lattice. I am observing a crash within KSP_Solve for an application only after the 43rd call to KSP_Solve when building the application and PETSc in debug mode, full logs for which are attached with this email (1 MPI rank and 4 OMP threads were used, but this crash occurs with multiple MPI ranks as well ). I am also including the last few lines of the configuration for this build. This crash does not occur when building the application and PETSc in release mode.  Could someone tell me what causes this crash and if anything can be done to prevent it? Thanks in advance. The configuration of this solver is here :  https://github.com/fnalacceleratormodeling/synergia2/blob/sajid/features/openpmd_basic_integration/src/synergia/collective/space_charge_3d_fd_utils.cc#L273-L292 Thank You, Sajid Ali (he/him) | Research Associate Scientific Computing Division Fermi National Accelerator Laboratory s-sajid-ali.github.io <ksp_crash_log>

configure_log_tail_local_install
Description: Binary data

ksp_crash_log_local_install
Description: Binary data