Re: [petsc-users] KSP_Solve crashes in debug mode

[Barry Smith]

Hmm, here is the macro

    #define PetscCallAbort(comm, ...) \
    do { \
      PetscErrorCode ierr_petsc_call_abort_; \
      PetscStackUpdateLine; \
      ierr_petsc_call_abort_ = __VA_ARGS__; \
      if (PetscUnlikely(ierr_petsc_call_abort_ != PETSC_SUCCESS)) { \
        ierr_petsc_call_abort_ = PetscError(PETSC_COMM_SELF, __LINE__, 
PETSC_FUNCTION_NAME, __FILE__, ierr_petsc_call_abort_, PETSC_ERROR_REPEAT, " 
"); \
        (void)MPI_Abort(comm, (PetscMPIInt)ierr_petsc_call_abort_); \
      } \
    } while (0)

it does not seem to increment anything in the stack. So I think call should be 
ok

Perhaps your function has a PetscFunctionBegin, but no PetscFunctionReturn() or 
in some other way increase the stack size without decreasing it?




> On Feb 24, 2023, at 12:39 PM, Sajid Ali Syed <sas...@fnal.gov> wrote:
> 
> Hi Barry, 
> 
> The application calls PetscCallAbort in a loop, i.e.
> 
> for i in range:
>   void routine(PetscCallAbort(function_returning_petsc_error_code))
> 
> From the prior logs it looks like the stack grows every time PetscCallAbort 
> is called (in other words, the stack does not shrink upon successful exit 
> from PetscCallAbort). 
> 
> Is this usage pattern not recommended? Should I be manually checking for 
> success of the `function_returning_petsc_error_code` and throw instead of 
> relying on PetscCallAbort?
> 
> 
> 
> Thank You,
> Sajid Ali (he/him) | Research Associate
> Data Science, Simulation, and Learning Division
> Fermi National Accelerator Laboratory
> s-sajid-ali.github.io <http://s-sajid-ali.github.io/>
> From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>
> Sent: Wednesday, February 22, 2023 6:49 PM
> To: Sajid Ali Syed <sas...@fnal.gov <mailto:sas...@fnal.gov>>
> Cc: Matthew Knepley <knep...@gmail.com <mailto:knep...@gmail.com>>; 
> petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> 
> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>
> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode
>  
> 
>   Hmm, there could be a bug in our handling of the stack when reaches the 
> maximum. It is suppose to just stop collecting additional levels at that 
> point but likely it has not been tested since a lot of refactorizations.
> 
>    What are you doing to have so many stack frames? 
> 
>> On Feb 22, 2023, at 6:32 PM, Sajid Ali Syed <sas...@fnal.gov 
>> <mailto:sas...@fnal.gov>> wrote:
>> 
>> Hi Matt, 
>> 
>> Adding `-checkstack` does not prevent the crash, both on my laptop and on 
>> the cluster. 
>> 
>> What does prevent the crash (on my laptop at least) is changing 
>> `PETSCSTACKSIZE` from 64 to 256 here : 
>> https://github.com/petsc/petsc/blob/main/include/petscerror.h#L1153 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_petsc_petsc_blob_main_include_petscerror.h-23L1153&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=h95E7R5X17258LHwsaKi0qVASp22lBVFOsdrDZFvAOS2iJQd-5FGzfHgq68ShXYR&s=Rfmp69z-e_VacDf-D0n8jt0xA6qq7oRBfgFSgMn1Dj8&e=>
>> 
>> 
>> Thank You,
>> Sajid Ali (he/him) | Research Associate
>> Data Science, Simulation, and Learning Division
>> Fermi National Accelerator Laboratory
>> s-sajid-ali.github.io 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io_&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=h95E7R5X17258LHwsaKi0qVASp22lBVFOsdrDZFvAOS2iJQd-5FGzfHgq68ShXYR&s=KDcd2SRT062jOa-0d8hvQywGEvYtyx9oHol5xp4XMI8&e=>
>> From: Matthew Knepley <knep...@gmail.com <mailto:knep...@gmail.com>>
>> Sent: Wednesday, February 22, 2023 5:23 PM
>> To: Sajid Ali Syed <sas...@fnal.gov <mailto:sas...@fnal.gov>>
>> Cc: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>; 
>> petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> 
>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>
>> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode
>>  
>> On Wed, Feb 22, 2023 at 6:18 PM Sajid Ali Syed via petsc-users 
>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote:
>> One thing to note in relation to the trace attached in the previous email is 
>> that there are no warnings until the 36th call to KSP_Solve. The first error 
>> (as indicated by ASAN) occurs somewhere before the 40th call to KSP_Solve 
>> (part of what the application marks as turn 10 of the propagator). The crash 
>> finally occurs on the 43rd call to KSP_solve.
>> 
>> Looking at the trace, it appears that stack handling is messed up and 
>> eventually it causes the crash. This can happen when
>> PetscFunctionBegin is not matched up with PetscFunctionReturn. Can you try 
>> running this with
>> 
>>   -checkstack
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> Thank You,
>> Sajid Ali (he/him) | Research Associate
>> Data Science, Simulation, and Learning Division
>> Fermi National Accelerator Laboratory
>> s-sajid-ali.github.io 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=q5fD8r624Cr0Ow4AKTmgeLtq_M--q_KdGYMkBNiKOMDa8o82C8P97vdCRcxrqTCF&s=oNWxB3zDYTHODeZK9VCibIVqSo7DnwsJjSr6IgIPs2M&e=>
>> From: Sajid Ali Syed <sas...@fnal.gov <mailto:sas...@fnal.gov>>
>> Sent: Wednesday, February 22, 2023 5:11 PM
>> To: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>
>> Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> 
>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>
>> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode
>>  
>> Hi Barry, 
>> 
>> Thanks a lot for fixing this issue. I ran the same problem on a linux 
>> machine and have the following trace for the same crash (with ASAN turned on 
>> for both PETSc (on the latest commit of the branch) and the application) : 
>> https://gist.github.com/s-sajid-ali/85bdf689eb8452ef8702c214c4df6940 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_s-2Dsajid-2Dali_85bdf689eb8452ef8702c214c4df6940&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=q5fD8r624Cr0Ow4AKTmgeLtq_M--q_KdGYMkBNiKOMDa8o82C8P97vdCRcxrqTCF&s=Z8JyNKYXjUZE4DXYKvjxTOG4HZUA95U6z750WC6gUCo&e=>
>> 
>> The trace seems to indicate a couple of buffer overflows, one of which 
>> causes the crash. I'm not sure as to what causes them. 
>> 
>> Thank You,
>> Sajid Ali (he/him) | Research Associate
>> Data Science, Simulation, and Learning Division
>> Fermi National Accelerator Laboratory
>> s-sajid-ali.github.io 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=q5fD8r624Cr0Ow4AKTmgeLtq_M--q_KdGYMkBNiKOMDa8o82C8P97vdCRcxrqTCF&s=oNWxB3zDYTHODeZK9VCibIVqSo7DnwsJjSr6IgIPs2M&e=>
>> From: Barry Smith <bsm...@petsc.dev <mailto:bsm...@petsc.dev>>
>> Sent: Wednesday, February 15, 2023 2:01 PM
>> To: Sajid Ali Syed <sas...@fnal.gov <mailto:sas...@fnal.gov>>
>> Cc: petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov> 
>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>>
>> Subject: Re: [petsc-users] KSP_Solve crashes in debug mode
>>  
>> 
>> https://gitlab.com/petsc/petsc/-/merge_requests/6075 
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_6075&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=P7R0CW9R-fGNfm2q3yTL-ehqhM5N9-r8hHBLNgDetm9-7jxVqNsujIZ2hdnhVrVX&s=QwRI_DzGnCHagpaQSC4MPPEUnC4aAkbMwdG1eg_QUII&e=>
>>  should fix the possible recursive error condition Matt pointed out
>> 
>> 
>>> On Feb 9, 2023, at 6:24 PM, Matthew Knepley <knep...@gmail.com 
>>> <mailto:knep...@gmail.com>> wrote:
>>> 
>>> On Thu, Feb 9, 2023 at 6:05 PM Sajid Ali Syed via petsc-users 
>>> <petsc-users@mcs.anl.gov <mailto:petsc-users@mcs.anl.gov>> wrote:
>>> I added “-malloc_debug” in a .petscrc file and ran it again. The backtrace 
>>> from lldb is in the attached file. The crash now seems to be at:
>>> 
>>> Process 32660 stopped* thread #1, queue = 'com.apple.main-thread', stop 
>>> reason = EXC_BAD_ACCESS (code=2, address=0x16f603fb8)
>>>     frame #0: 0x0000000112ecc8f8 libpetsc.3.018.dylib`PetscFPrintf(comm=0, 
>>> fd=0x0000000000000000, format=0x0000000000000000) at mprint.c:601
>>>    598               `PetscViewerASCIISynchronizedPrintf()`, 
>>> `PetscSynchronizedFlush()`
>>>    599      @*/
>>>    600      PetscErrorCode PetscFPrintf(MPI_Comm comm, FILE *fd, const char 
>>> format[], ...)
>>> -> 601      {
>>>    602       PetscMPIInt rank;
>>>    603      
>>>    604       PetscFunctionBegin;
>>> (lldb) frame info
>>> frame #0: 0x0000000112ecc8f8 libpetsc.3.018.dylib`PetscFPrintf(comm=0, 
>>> fd=0x0000000000000000, format=0x0000000000000000) at mprint.c:601
>>> (lldb)
>>> The trace seems to indicate some sort of infinite loop causing an overflow.
>>> 
>>> 
>>> Yes, I have also seen this. What happens is that we have a memory error. 
>>> The error is reported inside PetscMallocValidate()
>>> using PetscErrorPrintf, which eventually calls PetscCallMPI, which calls 
>>> PetscMallocValidate again, which fails. We need to
>>> remove all error checking from the prints inside Validate.
>>> 
>>>   Thanks,
>>> 
>>>      Matt
>>>  
>>> PS: I'm using a arm64 mac, so I don't have access to valgrind. 
>>> 
>>> Thank You,
>>> Sajid Ali (he/him) | Research Associate
>>> Scientific Computing Division
>>> Fermi National Accelerator Laboratory
>>> s-sajid-ali.github.io 
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__s-2Dsajid-2Dali.github.io_&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=P7R0CW9R-fGNfm2q3yTL-ehqhM5N9-r8hHBLNgDetm9-7jxVqNsujIZ2hdnhVrVX&s=JA1u9AHcO8HqY5oCgbEy-ghtKRjURlRDwdmxP-9YJac&e=>
>>> 
>>> 
>>> -- 
>>> What most experimenters take for granted before they begin their 
>>> experiments is infinitely more interesting than any results to which their 
>>> experiments lead.
>>> -- Norbert Wiener
>>> 
>>> https://www.cse.buffalo.edu/~knepley/ 
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=P7R0CW9R-fGNfm2q3yTL-ehqhM5N9-r8hHBLNgDetm9-7jxVqNsujIZ2hdnhVrVX&s=CdEZKWQbBYiD2pzU3Az_EDIGUTBNkNHwSoD2n_2098Y&e=>
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments 
>> is infinitely more interesting than any results to which their experiments 
>> lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ 
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=w-DPglgoOUOz8eiEyHKz0g&m=q5fD8r624Cr0Ow4AKTmgeLtq_M--q_KdGYMkBNiKOMDa8o82C8P97vdCRcxrqTCF&s=Hkn4IxPABZIeY0m9o_VGFHJ4ntffqbtyd3fddpbZw7I&e=>