[issue18861] Problems with recursive automatic exception chaining

[Irit Katriel]

Irit Katriel  added the comment:

I think this problem is actually simpler than what we've been discussing. 

First, note that by-and-large our current system works:

>>> try:
...   raise VE(1)
... except VE as e1:
...   try:
... raise VE(2)
...   except VE as e2:
... raise VE(3) from e2
... 
Traceback (most recent call last):
  File "", line 2, in 
ValueError: 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 5, in 
ValueError: 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "", line 7, in 
ValueError: 3



Here VE(2) is the cause of VE(3) and VE(1) is the context of VE(2), so VE(1) is 
not hidden by the fact that VE(3) has a cause.

The reason that Nick's example didn't work is because he is doing

raise VE(3) from VE(2)

i.e., creating a new exception VE(2) that doesn't have VE(1) as a context. I'm 
not sure there is a use case for this, so I don't think we need to worry about 
it.


The case of None does need fixing. We use None to indicate that there was no 
cause (while suppressing context). But None can't have a context, so VE(1) gets 
lost. We could, instead of None, use a NoException(Exception) class. This 
exception would be chained with the current exc_info() as context so that it 
works like VE(1) as above, and the traceback printing logic will know that it 
needs to be omitted from the output.


This proposal involves no changes to the exception propagation mechanism of the 
interpreter. It would require these changes:

1. Define the new exception type
2. in do_raise, in the case of raise-from the None case changes
3. traceback display code needs to be updated to omit NoExceptions

--
versions: +Python 3.11 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Irit Katriel]

Irit Katriel  added the comment:

For the issue of which contexts need to be suppressed when __suppress_context__ 
is true, I think we don't need to keep the whole tree of exceptions. It might 
be enough to save the dynamic "nesting depth of the try-excepts".  Here is a 
solution with two integers per exception, plus one on the thread state. Would 
it work?


We add an int field called except_depth to the thread state (incremented in 
ceval just before the try of a try-except and decremented after the last 
except: block).

Whenever an exception is caught, this value is copied to it (to a new int we 
put on BaseException, call it depth).

The __suppress_context__ field we change from bool to int (alternatively: add a 
new field and leave this one alone). Instead of setting it to True when there 
is __cause__, we set it to the except-depth of the raise...from that raised it.

Then the code displaying the traceback can use the depths information to 
determine which contexts to suppress (the ones up to and not including the 
first one with depth < the suppress_context value).

Returning to Nick's examples from 2013-11-08 14:34:

  # No context on Exception3 is exactly what we want
  try:
  try:
  raise Exception1
  except Exception1:
  raise Exception2
  except Exception2 as exc
  raise Exception3 from Exception2

Exception1 - depth = 2
Exception2 - depth = 1
Exception3 - depth = ?  suppress_context = 1
(suppress both Exception1 and Exception2, they have depths >= 1)


  # Not seeing Exception1 as the context for Exception3 is surprising!
  try:
  raise Exception1
  except Exception1:
  try:
  raise Exception2
  except Exception2 as exc
  raise Exception3 from Exception2
  
Exception1 - depth = 1
Exception2 - depth = 2
Exception3 - depth = ?  suppress_context = 2
(suppress only Exception2, Exception1 is too shallow)
  

To consider more elaborate nesting:

  try:
  try:
  raise Exception0
  except T: # Does not catch Exception0
  pass
  except Exception1:
  try:
  raise Exception2
  except Exception2 as exc
  raise Exception3 from Exception2

Whatever happens in the topmost try-block, if Exception0 is the context of 
anything raised inside the "except Exception1" block, then its depth is 1 
because it was caught by this except. So it won't be suppressed by anything 
raised at a deeper level.

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Martin Panter]

Change by Martin Panter :


--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Chris Jerdonek]

Change by Chris Jerdonek :


--
nosy: +chris.jerdonek

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nathaniel Smith]

Changes by Nathaniel Smith :


--
nosy: +njs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Actually, this won't the subTest example because that actually *suppresses* the 
errors, and reports them later. Annotations only help when the exception is 
allowed to escape.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Adjusting the target version, since it isn't feasible to move away from the 
current behaviour any earlier than Python 3.5.

--
versions: +Python 3.5 -Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Something else that this might make simpler: the unittest.TestCase.subTest API 
is currently kind of ugly. If the subtest details could be stored as a frame 
annotation instead...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

So, I've been pondering the idea of traceback/frame annotations and exception 
trees a bit.

And what I'm wondering is if it may make sense to have the ability to annotate 
*frames* at runtime, and these annotations can be qualified by module names. 
So, for example, you might be able to write things like:

sys.annotate_frame(codecs, encoding, the_encoding)
sys.annotate_frame(codecs, decoding, the_encoding)
sys.annotate_frame(traceback, hide, True)
sys.annotate_frame(traceback, context, exc)

And then the traceback display machinery would be updated to do something 
useful with the annotations.

I'm not sure how ExitStack would cope with that (or other code that fakes 
tracebacks) but it's something to ponder.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Walter suggested annotating the exceptions directly might work: 
https://mail.python.org/pipermail/python-dev/2013-November/130155.html

However, that potentially runs into nesting problems (e.g. the idna codec 
invokes the ascii codec), although Walter suggested a simpler mechanism that 
just involved appending to a list.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

It may not immediately look like it, but I think issue 17828 offers an example 
of a related problem. In that issue, I didn't want to *change* the exception 
raised, I wanted to annotate it to say Hey, something I called raised an 
exception, here's some relevant local state to help you figure out what is 
going on (in that case, the information to be added is the specific codec 
being invoked and whether it is an encoding or decoding operation).

Setting the context is somewhat similar - you don't just want to know which 
specific exception happened to be active when the eventually caught exception 
was first raised - you also want to know which already active exception 
handlers you passed through while unwinding the stack.

So really, what may be desirable here is almost an annotated traceback, where 
the interpreter can decide to hang additional information off the frame 
currently being unwound in the exceptions traceback, while leaving the 
exception itself alone.

That's definitely PEP territory, but there are two distinct problems with the 
current exception chaining mechanism to help drive a prospective design for 3.5 
(the status quo doesn't bother me enough for me to work on it myself, but if 
you're interested Nikolaus...)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Unrelated to my previous comment, I'm also wondering if this may actually 
represent a behavioural difference between contextlib.ExitStack and the 
interpreter's own exception handling machinery.

ExitStack uses a recursive-iterative transformation for its stack unwinding 
(see issue 14963), and it needs to do a bit of fiddling to get the context 
right (see issue 19092).

While the contextlib test suite goes to great lengths to try to ensure the 
semantics of normal stack unwinding are preserved, that definitely doesn't 
currently cover this case, and I'm thinking the way it works may actually be 
more like the behaviour Nikolaus expected in the original post (i.e. setting 
the context as the stack is unwound rather than when the replacement exception 
is raised).

--
nosy: +alonho, hniksic

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nikolaus Rath]

Nikolaus Rath added the comment:

Hi Nick,

I am interested in working on this, but I have never worked on the C parts of 
cpython before. Do you think this is a feasible project to start with? To me it 
looks a bit daunting, I'd certainly need some mentoring to even know where to 
start with this.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Benjamin Peterson]

Benjamin Peterson added the comment:

The first thing to do is to carefully specificy what the behavior should be.

2013/11/10 Nikolaus Rath rep...@bugs.python.org:

 Nikolaus Rath added the comment:

 Hi Nick,

 I am interested in working on this, but I have never worked on the C parts of 
 cpython before. Do you think this is a feasible project to start with? To me 
 it looks a bit daunting, I'd certainly need some mentoring to even know where 
 to start with this.

 --

 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue18861
 ___

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Yes, I suggest using ExitStack to figure out the behaviour we *want* first,
before diving into the messy practical details of how to make that a
reality in CPython. We somehow have to get the state of the exception
object and its traceback to represent an appropriate stack *tree*, rather
than the traditionally assumed linear stack.

It also occurred to me there's another potentially related issue: frame
hiding, where we want to avoid showing infrastructure code in end user
tracebacks. importlib currently has a very hacky version of that. The
Jinja2 template library uses a different approach.

The reason I bring these other problems up is because I think they
illustrate a theme around altering how a traceback is displayed that may be
amenable to a common solution (preferably one that is contextlib and
asyncio friendly).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

I actually think Nikratio is right about the way this *should* work 
(intuitively).

I'm just not sure it's feasible to *implement* those semantics in CPython 
without significant changes to the way exception handling works - I don't 
believe the exception chaining code can currently tell the difference between 
the cases:

  # No context on Exception3 is exactly what we want
  try:
  try:
  raise Exception1
  except Exception1:
  raise Exception2
  except Exception2 as exc
  raise Exception3 from Exception2

  # Not seeing Exception1 as the context for Exception3 is surprising!
  try:
  raise Exception1
  except Exception1:
  try:
  raise Exception2
  except Exception2 as exc
  raise Exception3 from Exception2

In a certain sense, exceptions need to be able to have *multiple* contexts to 
handle this case properly without losing data. Frames would need to tag 
exceptions appropriately with the context details as an unhandled exception 
passed through a frame that was currently running an exception handler.

So even though it doesn't require new syntax, I think it *does* require a PEP 
if we're going to change this (and we still haven't fully dealt with the 
consequence of the last one - the display options for tracebacks are still a 
bit limited)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Benjamin Peterson]

Benjamin Peterson added the comment:

raise ... from None explictly silences the printing of exception context, so I 
don't see the problem.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nick Coghlan]

Nick Coghlan added the comment:

Nothing is lost either way, it just isn't displayed by default.

While I can see the appeal of treating an outer context differently from an 
inner one, I don't see any way for the exception context suppression 
machinery to tell the difference.

The traceback *display* machinery might be able to figure it out by looking at 
the traceback details, but such a search could be quite expensive.

--
nosy: +ncoghlan

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nikolaus Rath]

Nikolaus Rath added the comment:

Benjamin: I think that in most cases the intention of a .. from None is to 
disable printing of a stack trace for a specific exception when it occurs in 
the try .. except block. (In the example, that would be suppressing the 
stacktrace for the Second exception, because it is explicity converted to 
Third). I do not think that this ought to effect the printing of exceptions 
that occured previously and higher-up in the call chain.

In other words, I think it is natural to expect that

def foo():
   try:
  do_stuff
   except Something:
  raise SomethingElse from None

disables printing the stack trace for `Something` exceptions when they occur in 
the try..except block -- but not to hide the exception context for exceptions 
that occured before foo() was even called.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nikolaus Rath]

Nikolaus Rath added the comment:

*ping* 

No one any comments on this at all?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +benjamin.peterson

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nikolaus Rath]

New submission from Nikolaus Rath:

Consider this:

$ python3 test_exc.py 
Traceback (most recent call last):
  File test_exc.py, line 14, in module
fail1()
  File test_exc.py, line 11, in fail1
fail2()
  File test_exc.py, line 5, in fail2
raise RuntimeError('Third') from None
RuntimeError: Third

$ cat test_exc.py 
def fail2():
try:
raise RuntimeError('Second')
except RuntimeError:
raise RuntimeError('Third') from None

def fail1():
try:
raise RuntimeError('First')
except:
fail2()
raise

fail1()


Any exception raised in fail2() is the immediate consequence of the 'First' 
exception should thus be chained to the 'First' exception.

However, if somewhere in the call stack under fail2() an exception is caught 
and re-raised from None (to convert between exception types), this also results 
in a loss of the chain to the initial exception.


The correct stacktrace (in my opinion) would be:
Traceback (most recent call last):
  File test_exc.py, line 9, in fail1
raise RuntimeError('First')
RuntimeError: First

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File test_exc.py, line 14, in module
fail1()
  File test_exc.py, line 11, in fail1
fail2()
  File test_exc.py, line 5, in fail2
raise RuntimeError('Third')
RuntimeError: Third

--
components: Interpreter Core
messages: 196344
nosy: Nikratio
priority: normal
severity: normal
status: open
title: Problems with recursive automatic exception chaining
type: behavior
versions: Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18861] Problems with recursive automatic exception chaining

[Nikolaus Rath]

Nikolaus Rath added the comment:

The second paragraph should of course read ...consequence of the 'First' 
exception *and* should thus be chained...

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18861
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com