[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Eryk Sun]

Eryk Sun  added the comment:

In theory, a crash could be prevented in most cases by setting a larger stack 
guarantee (i.e. region of guard pages) via SetThreadStackGuarantee() [1] and 
using a vectored exception handler [2]. The exception handler can set a flag in 
the thread state that indicates stack-overflow recovery is in progress and then 
return EXCEPTION_CONTINUE_EXECUTION. The guaranteed stack space will be 
available, but there are no guard pages, so another stack overflow in this 
context will crash with an access violation. The stack guarantee should be 
large enough to raise and unwind a RecursionError. As the stack unwinds, if the 
recovery flag is still set, try calling _resetstkoflw() [3] to restore the 
guard region. If it succeeds, clear the flag in the thread state.

For giggles, here's a toy example using ctypes:

import ctypes
import sys

kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
ucrt = ctypes.CDLL('ucrtbase', use_errno=True)

EXCEPTION_CONTINUE_EXECUTION = 0x

stack_overflow = False

@ctypes.WINFUNCTYPE(ctypes.c_long, ctypes.c_void_p)
def handler(p):
global stack_overflow
stack_overflow = True
return EXCEPTION_CONTINUE_EXECUTION

kernel32.AddVectoredExceptionHandler(1, handler)

def recursive():
if stack_overflow:
raise RecursionError
recursive()

# Normally the stack has 2 or 3 guard pages, which is actually
# enough to recover in this example, but let's increase it to
# 9 pages (8 plus an extra that the memory manager adds). You
# can inspect this with Sysinternals VMMap.
size = ctypes.c_ulong(8 * 4096)
kernel32.SetThreadStackGuarantee(ctypes.byref(size))

sys.setrecursionlimit(100)

for n in range(5):
try:
recursive()
except RecursionError:
if stack_overflow and ucrt._resetstkoflw():
stack_overflow = False
print("recovered from stack overflow:", n)

---
[1] 
https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-setthreadstackguarantee
[2] 
https://docs.microsoft.com/en-us/windows/win32/api/errhandlingapi/nf-errhandlingapi-addvectoredexceptionhandler
[3] 
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/resetstkoflw?view=msvc-160

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

PABLO LOBATO DE LA CRUZ  added the comment:

I see. Thank you very much Steve :)

El vie, 29 oct 2021 a las 0:29, Steve Dower ()
escribió:

>
> Steve Dower  added the comment:
>
> > But why does it happen only on Windows?
>
> Because it's the operating system that is terminating the process :)
> Other operating systems behave differently when you exceed the stack,
> and may also allocate different amounts of stack space, making it less
> likely that you hit it.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Steve Dower]

Steve Dower  added the comment:

> But why does it happen only on Windows?

Because it's the operating system that is terminating the process :) 
Other operating systems behave differently when you exceed the stack, 
and may also allocate different amounts of stack space, making it less 
likely that you hit it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

PABLO LOBATO DE LA CRUZ  added the comment:

I see thanks for answering so quickly. But why does it happen only on
Windows?

El jue, 28 oct 2021 a las 23:09, Steve Dower ()
escribió:

>
> Steve Dower  added the comment:
>
> This is almost certainly because of how Windows handles stack overflow
> exceptions, and the fact that there's no way for us to detect it reliably.
>
> There's some work going on to reduce the C stack depth when calling
> heavily nested Python code (see issue45256), but there's nothing else we
> can really do I'm afraid.
>
> I'm marking *this* issue as wontfix, but hopefully we can make some
> improvement on this general issue through other issues. Thanks for
> reporting it!
>
> --
> resolution:  -> wont fix
> stage:  -> resolved
> status: open -> closed
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[Steve Dower]

Steve Dower  added the comment:

This is almost certainly because of how Windows handles stack overflow 
exceptions, and the fact that there's no way for us to detect it reliably.

There's some work going on to reduce the C stack depth when calling heavily 
nested Python code (see issue45256), but there's nothing else we can really do 
I'm afraid.

I'm marking *this* issue as wontfix, but hopefully we can make some improvement 
on this general issue through other issues. Thanks for reporting it!

--
resolution:  -> wont fix
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45645] Deep recursion terminates script execution with no error (Windows, Python 3.9)

[PABLO LOBATO DE LA CRUZ]

New submission from PABLO LOBATO DE LA CRUZ :

Deep recursion crashes on Windows (Python 3.9) when the depth limit is 
increased and no error is shown.
Seems to work fine on other systems that I have tried (Linux and MacOS).
Please find attached the script to reproduce the error.
Expected and other systems output:

Calculated N2
Try block finished correctly
This is the finally block

Output:

Calculated N2

--
components: Windows
files: forstack.py
messages: 405185
nosy: pablolob, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Deep recursion terminates script execution with no error (Windows, 
Python 3.9)
type: crash
versions: Python 3.9
Added file: https://bugs.python.org/file50407/forstack.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com