[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
stage: patch review -> resolved
status: pending -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

The issue has been solved, all buildbots should now pass. Will continue to 
monitor the situation. Thanks for the report @kj!

--
priority: release blocker -> 
resolution:  -> fixed
status: open -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:


New changeset 87f502231c6d5b04a4d8aa23fba24fcf5303aebb by Batuhan Taskaya in 
branch '3.9':
[3.9] bpo-11105: reduce the recursion limit for tests. (GH-26605)
https://github.com/python/cpython/commit/87f502231c6d5b04a4d8aa23fba24fcf5303aebb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:


New changeset bd6f0d3eadfe5623657db6aeb69b94d21f86f4a0 by Batuhan Taskaya in 
branch '3.10':
[3.10] bpo-11105: reduce the recursion limit for tests. (GH-26607)
https://github.com/python/cpython/commit/bd6f0d3eadfe5623657db6aeb69b94d21f86f4a0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:


New changeset 8004c4570b1d1277ea8754e22b5eb60e63f5026c by Batuhan Taskaya in 
branch 'main':
bpo-11105: document the new test.support.infinite_recursion context manager 
(GH-26604)
https://github.com/python/cpython/commit/8004c4570b1d1277ea8754e22b5eb60e63f5026c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25190
pull_request: https://github.com/python/cpython/pull/26607

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25188
pull_request: https://github.com/python/cpython/pull/26605

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25187
pull_request: https://github.com/python/cpython/pull/26604

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:


New changeset e58d762c1fb4ad5e021d016c80c2bc4513632d2f by Batuhan Taskaya in 
branch 'main':
bpo-11105: reduce the recursion limit for tests (GH-26550)
https://github.com/python/cpython/commit/e58d762c1fb4ad5e021d016c80c2bc4513632d2f


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Jeremy Kloth]

Change by Jeremy Kloth :


--
nosy: +jkloth
nosy_count: 11.0 -> 12.0
pull_requests: +25186
pull_request: https://github.com/python/cpython/pull/26578

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[ppperry]

Change by ppperry :


--
nosy:  -ppperry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

>b) always skip the test on windows => we can do that but it might be 
> counterintuitive for the future

Well, is not that the test is flaky technically, this means that the feature 
doesn't work on Windows (non release builds). So the reasoning has to be why we 
want/need to not support this on Windows. Otherwise we need to customize the 
limit on debug builds.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

After playing with it for a couple hours and without much success of creating a 
test environment (only using buildbots), I decided not to introduce hard 
limits. Even though they make the original tests to pass, they don't solve the 
problem overall and also more important part is that the 'hard limits' might 
cause regressions for people who do compile() calls. 

For normal windows builds (as @kj noted) something might work in the current 
revision and we might just break it with introducing hard limits. Since the 
trees are tend to get really branchy, I don't think it is a good idea.

I'm open to any proposals/plans 

Extra: In the worst case that we can't come up with something (the AST 
converter functions are really long 2000+ LoC C functions so it is possible 
that there might be stuff that eats a lot of space on the stack), we can either
   a) revert => not a good option, this is not a regression on the python 
itself. It is a fix for other os's and windows release builds
   b) always skip the test on windows => we can do that but it might be 
counterintuitive for the future
   c) use a really low recursion limit for the test_recursion_* for windows => 
I'm open to fallback to this if nothing comes up. 

we might need to revert this though as is it is not a regression. It used to 
crash with the same exact error, just outside of the test suite, and now since 
it works for linux/macos/others + windows for release builds I wonder whether 
can just skip the test on windows and keep it as is in the worst scenario).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25141
pull_request: https://github.com/python/cpython/pull/26554

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25137
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/26550

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> The stack size for a window build is currently set to 2MB, which is usually 
> lesser than *nix 8MB. So I think an easy solution is to increase the stack 
> size for windows builds.

> I'm guessing release builds aren't affected because some of the 
> Py_EnterRecursiveCall helper functions are probably inlined and thus use less 
> of the stack.

> Opinions are greatly appreciated.

I don't think that we should make a global change for this case, AFAIK some of 
the core parts of the interpreter maintain their own recursion checks with 
different handling of windows limits. E.g;

https://github.com/python/cpython/blob/fa106a685c1f199aca5be5c2d0277a14cc9057bd/Python/marshal.c#L25-L40

We might need to end up with the same motion and do the handling by ourselves. 
Wdyt @pablogsal @kj?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
nosy: +lukasz.langa
priority: normal -> release blocker
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Batuhan Taskaya  added the comment:

> Batuhan, can you take a look?

Yes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

Batuhan, can you take a look?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Ken Jin]

Ken Jin  added the comment:

The newly added test ``test_recursion_direct`` seems to trigger a stack 
overflow on windows in debug mode instead of a RecursionError. Release mode 
isn't affected and the test passes there.

One of the buildbots reflects this too: 
https://buildbot.python.org/all/#/builders/146/builds/337/steps/4/logs/stdio

I can avoid the crash by lowering the recursion limit in Python from 1000 to 
500. The stack size for a window build is currently set to 2MB, which is 
usually lesser than *nix 8MB. So I think an easy solution is to increase the 
stack size for windows builds.

I'm guessing release builds aren't affected because some of the 
Py_EnterRecursiveCall helper functions are probably inlined and thus use less 
of the stack.

Opinions are greatly appreciated.

--
nosy: +kj

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Pablo Galindo Salgado]

Change by Pablo Galindo Salgado :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:


New changeset de58b319af3a72440a74e807cf8a1194ed0c6d8c by Batuhan Taskaya in 
branch '3.9':
[3.9] bpo-11105: Do not crash when compiling recursive ASTs (GH-20594) 
(GH-26522)
https://github.com/python/cpython/commit/de58b319af3a72440a74e807cf8a1194ed0c6d8c


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
pull_requests: +25116
pull_request: https://github.com/python/cpython/pull/26522

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11105] Compiling recursive Python ASTs crash the interpreter

[Batuhan Taskaya]

Change by Batuhan Taskaya :


--
components: +Interpreter Core -None
priority: low -> normal
title: Compiling evil ast crashes interpreter -> Compiling recursive Python 
ASTs crash the interpreter
versions: +Python 3.11, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com