[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Gregory P. Smith]

Change by Gregory P. Smith :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Andrew Svetlov]

Andrew Svetlov  added the comment:

'catch (boost::python::error_already_set e)' is equal to `except BaseException 
as e:`
In Python, blind catching base exception is dangerous, the code should re-raise 
it usually.
The same is true for boost::python usage.

> how would it tell the difference between a "real" exception raised from 
> within whatever function is currently returning and the "fake" StopIteration 
> exception that is in the error global while an async function returns?

There is no "fake" exception. async function is a kind of Python generator 
object that uses StopIteration exception for finishing.
The same is true for a regular Python iterator; nothing asyncio specific.

I suggest writing a functional equivalent for `except Exception as e: print(e)` 
instead of catching BaseException error.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Zac Bentley]

Zac Bentley  added the comment:

For context (copied from the linked GitHub issue), this has affected at least 
two OSS projects:

https://github.com/apache/pulsar/issues/11823
https://github.com/tbodt/v8py/issues/45

The proposed alternative ("PyErr_Fetch, then PyErr_Display, then 
PyErr_Restore") does not work well: if code runs in a boost-python destructor 
which runs during async return time, how would it tell the difference between a 
"real" exception raised from within whatever function is currently returning 
and the "fake" StopIteration exception that is in the error global while an 
async function returns? Aka, this code would not have any way to function 
properly:

async def func():
thing = MyNativeCodeObject()
raise ValueError()

The destructor for MyNativeCodeObject would have to perform a complex check, 
and PyErr_Restore/do nothing IFF the following are all true:
- Is there an exception currently being handled/in the global?
- Is that exception a StopIteration?
- Is the interpreter currently in the process of returning from an async 
function (I'm not sure how to check this)?


The fact that non-StopIteration exceptions are handled correctly (e.g. the 
ValueError in the above example makes it out of the async call during "await 
func()") means, I think, that there's already some secondary exception storage 
mechanism that's used to stash and restore user exceptions while the async 
machinery raises StopIteration to return from an async function. Could that 
same mechanism be used to "hide" the StopIteration from user code that runs 
during async-return time?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Zac Bentley]

Zac Bentley  added the comment:

Irit, the documentation is clear.

However, the problem is that the presence of StopIteration in the exception 
global during async return time causes very unexpected behavior.

If, during async return time (in the linked issue, I do this in a boost-python 
object destructor), I write code like this:

try {
boost::python::call(somefunction);
} catch (boost::python::error_already_set e) {
PyErr_Print();
}

then two things go wrong:
1. The "catch" block runs even if "somefunction" did not raise, because 
StopIteration is already in the exception handler.
2. When PyErr_Print clears StopIteration, the currently-returning async 
function returns None, regardless of what it was supposed to return. I consider 
this to be interpreter-induced data corruption.

I'm not sure what happens if user code raises an additional exception.

Do you think this is a Python bug, or a Boost bug? I'm not familiar enough with 
interpreter internals to know whether or not there are other ways besides 
boost-python destructors to get code to run "during async return". If there 
are, I would definitely consider this a Python bug: PyErr_Print should not be 
able to corrupt interpreter/return state.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Irit Katriel]

Irit Katriel  added the comment:

Zac, the documentation for PyErr_Print [1] states that it clears the error 
indicator. This is akin to handling the exception. Was this your intention?

If you want to print the exception without clearing it, I think you would 
probably need to call PyErr_Fetch, then PyErr_Display, then PyErr_Restore.

[1] https://docs.python.org/3/c-api/exceptions.html#c.PyErr_PrintEx

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45033] Calls to PyErr_PrintEx in destructors cause calling async functions to incorrectly return None

[Zac Bentley]

New submission from Zac Bentley :

If an object's destructor contains native code which calls PyErr_PrintEx, and 
that object's refcount drops to zero as the result of an async function 
returning, the async function incorrectly returns None.

I first identified this behavior while using Boost-python. A more detailed 
description, and steps to reproduce, are in the issue report I filed on that 
library: https://github.com/boostorg/python/issues/374

I'm not very familiar with interpreter internals, so it is possible that this 
is expected behavior. However, it does seem like at least a leaky abstraction 
between the mechanics of async calls (which use exception based control flow 
internally) and the PyErr_PrintEx function, which is typically invoked by 
callers interested in finding out about errors that they caused, not errors 
that are both caused elsewhere and whose propagation is important to preserving 
call stack state.

--
components: C API, Interpreter Core, asyncio
messages: 400448
nosy: asvetlov, yselivanov, zbentley
priority: normal
severity: normal
status: open
title: Calls to PyErr_PrintEx in destructors cause calling async functions to 
incorrectly return None
type: behavior
versions: Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com