[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice
Hi Mark, Le 05/03/2021 à 18:06, Mark Shannon a écrit : Hi Antoine, On 05/03/2021 4:07 pm, Antoine Pitrou wrote: On Fri, 5 Mar 2021 15:03:59 + Mark Shannon wrote: There are two issues here. Portability and changes to behaviour. Regarding portability, I have to admit that PEP is rather vague. That's my fault; I should have done more implementation first :( FWIW, I have an implementation that should be portable. https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation I looked through this diff and I'm not sure how this works robustly. This seems to assume: - each thread's stack size is known and is a compile-time constant We just need to know that the stack size is at least the compile-time constant, we don't need to know exactly what it is. If the stack size is less than the compile-time constant, then a crash is a possibility. However, since PEP 651 would reduce stack consumption, a crash would have been less likely than it currently is. For all supported platforms, there is a default stack size and it is considerably larger than the chosen value of 512k. Right, but Python more or less works on a large range of non-formally supported platforms. With your proposal and a hardcoded 512 kB minimum stack size, Python may crash almost immediately on such platforms. I don't know if we care about that (we could tell third-party maintainers to change the #define to an appropriate value), but it's still something to will make porting a bit more cumbersome. - Python owns the thread and can compute the stack limit reliably from the current stack frame There are two points of entry into Python code. From the high level API and thread_run(). Hmm, actually, there are many entry points into Python code. C code can create its own thread and call arbitrary CPython APIs from it (after having initialized the tstate using the PyGILState API). That call inside a C-created thread can happen after an arbitrary amount of stack was already consumed by the calling C code. But, this is no worse than the status quo, and probably better due to reduced C stack use. I'm not sure what you mean with "no worse than the statu quo". Supporting a smaller recursion limit than advertised is one thing, crashing almost immediately when executing Python code (because of trying to access memory outside beyond the reserved stack area) is another. Currently, you would only crash when recursing a lot. - it's ok to write memory beyond the current stack top (I can imagine verification tools such as Valgrind complaining about that, though this can be worked around using suppressions) It's not much of a stack if you can't grow it :) The stack size is typically limited (see `ulimit -s`). Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MKLBLT4OGAYGPJFIATSGQVD4MV7TEYC3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice
Hi Antoine, On 05/03/2021 4:07 pm, Antoine Pitrou wrote: On Fri, 5 Mar 2021 15:03:59 + Mark Shannon wrote: There are two issues here. Portability and changes to behaviour. Regarding portability, I have to admit that PEP is rather vague. That's my fault; I should have done more implementation first :( FWIW, I have an implementation that should be portable. https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation I looked through this diff and I'm not sure how this works robustly. This seems to assume: - each thread's stack size is known and is a compile-time constant We just need to know that the stack size is at least the compile-time constant, we don't need to know exactly what it is. If the stack size is less than the compile-time constant, then a crash is a possibility. However, since PEP 651 would reduce stack consumption, a crash would have been less likely than it currently is. For all supported platforms, there is a default stack size and it is considerably larger than the chosen value of 512k. - Python owns the thread and can compute the stack limit reliably from the current stack frame There are two points of entry into Python code. From the high level API and thread_run(). Since thread_run is called when a thread is started, it is guaranteed to have the full stack available. The high level API (PyRun_SimpleStringFlags and friends) is a bit more problematic. If there isn't sufficient stack space, then a crash is a possibility. But, this is no worse than the status quo, and probably better due to reduced C stack use. - the OS allocates stack pages in units of BLOCK_SIZE or more (currently 8kiB) The BLOCK_SIZE is just a number. A larger numbers means fewer slow checks, but wastes more stack. It doesn't matter what the underlying platform's granularity is. - the OS doesn't use a segmentation scheme that limits stack accesses with a finer granularity than page (or "block") size I don't think that would matter. - it's ok to write memory beyond the current stack top (I can imagine verification tools such as Valgrind complaining about that, though this can be worked around using suppressions) It's not much of a stack if you can't grow it :) I wouldn't be surprised if valgrind complains, though. I would be curious if you could elaborate on these points. The above scheme does require some care from users of the high-level C-API and should be tested thoroughly for new platforms. But it is a *lot* more robust than what we have now. Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WT6I75DELQM43VP5QYAYQGMG3RFPHA3F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice
On Fri, 5 Mar 2021 15:03:59 + Mark Shannon wrote: > > There are two issues here. Portability and changes to behaviour. > > Regarding portability, I have to admit that PEP is rather vague. > That's my fault; I should have done more implementation first :( > FWIW, I have an implementation that should be portable. > https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation I looked through this diff and I'm not sure how this works robustly. This seems to assume: - each thread's stack size is known and is a compile-time constant - Python owns the thread and can compute the stack limit reliably from the current stack frame - the OS allocates stack pages in units of BLOCK_SIZE or more (currently 8kiB) - the OS doesn't use a segmentation scheme that limits stack accesses with a finer granularity than page (or "block") size - it's ok to write memory beyond the current stack top (I can imagine verification tools such as Valgrind complaining about that, though this can be worked around using suppressions) I would be curious if you could elaborate on these points. Regards Antoine. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HFV33HIVTWLIJXI4WQAGK6ZLWYQHKJPX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 651, Robust Stack Overflow Handling, Rejection notice
Hi, Thanks for taking the time to consider the PEP. Although the PEP was rejected, I still believe that the safety guarantees in PEP 651 are worth adding to Python in the future. To do that (maybe for 3.11), I need to understand your concerns better. Would you clarify a few points for me? On 03/03/2021 7:19 pm, Python Steering Council wrote: Hi Mark, Thank you for submitting PEP 651. The Steering Council has spent the past two weeks reviewing PEP 651. After careful consideration, we have decided to reject the PEP. The following were the key points that led us to this decision: * The benefits are not compelling enough. Deep recursion is not a common tool in Python, and even with PEP 651 it would not be efficient enough to make it a common tool. * The benefit of PEP 651 is negated as soon as a non-Python function is involved in the recursion, making the likelihood of it being useful even smaller. It also creates easy pitfalls for users who do end up relying on recursion. Could you give an example pitfall? * We believe the PEP understates the disruption created by the technical solution of multiple Python stack frames per C call. Although this may be solvable, it will certainly cause substantial disruption to existing debuggers, tracers, and state inspection tools as they need to adapt to this change (which may not be trivial). This is presumably the key objection. Is there a particular tool that you feel would be problematic? I have only looked at gdb and py-spy. * As the way to approach this will be platform-specific (as some parts of the proposal are not portable), this can cause generic Python code to behave differently on different platforms, making this kind of code less portable and less predictable. There are two issues here. Portability and changes to behaviour. Regarding portability, I have to admit that PEP is rather vague. That's my fault; I should have done more implementation first :( FWIW, I have an implementation that should be portable. https://github.com/python/cpython/compare/master...markshannon:pep-overflow-implementation Regarding changes to behaviour, I don't see how "generic" Python code would behave differently on different platforms, except for cases where it already does. In some cases, the PEP would have improved the situation. For example: sys.setrecursionlimit(5000) def f(): f() Currently, it raises a RecursionError on linux, but crashes the interpreter on Windows. With PEP 651 it would have raised a RecursionError on both platforms. Am I missing something here? Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L3U7FRGRFAXOVLA6YEAG3TSQB7EDQ2TU/ Code of Conduct: http://python.org/psf/codeofconduct/
