Re: How to pop the interpreter's stack?
Steven D'Aprano wrote:
On Sun, 26 Dec 2010 09:15:32 -0800, Ethan Furman wrote:
Steven D'Aprano wrote:
Right. But I have thought of a clever trick to get the result KJ was
asking for, with the minimum of boilerplate code. Instead of this:
def _pre_spam(args):
if condition(args):
raise SomeException("message")
if another_condition(args):
raise AnotherException("message")
if third_condition(args):
raise ThirdException("message")
def spam(args):
_pre_spam(args)
do_useful_work()
you can return the exceptions instead of raising them (exceptions are
just objects, like everything else!), and then add one small piece of
boilerplate to the spam() function:
def _pre_spam(args):
if condition(args):
return SomeException("message")
if another_condition(args):
return AnotherException("message")
if third_condition(args):
return ThirdException("message")
def spam(args):
exc = _pre_spam(args)
if exc: raise exc
do_useful_work()
-1
You failed to mention that cleverness is not a prime requisite of the
python programmer -- in fact, it's usually frowned upon. The big
problem with the above code is you are back to passing errors in-band,
pretty much completely defeating the point of have an out-of-band
channel.
How is that any worse than making _pre_spam() a validation function that
returns a bool?
def spam(args):
flag = _pre_spam(args)
if flag: raise SomeException()
do_useful_work()
Also -1.
Is that also frowned upon for being too clever?
Frowned upon for being out-of-band, and not as much fun as being clever.
;) I'm pretty sure you've expressed similar sentiments in the past
(although my memory could be failing me).
More to the point, the OP had code that said:
args, kwargs = __pre_spam(*args, **kwargs)
and __pre_spam was either passing back verified (and possibly modified)
parameters, or raising an exception.
~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Steven D'Aprano wrote:
...
>
I think critics of my position have forgotten what it's like to learning
the language. One of the most valuable skills to learn is to *ignore
parts of the traceback* -- a skill that takes practice and familiarity
with the library or function in use. To those who are less familiar with
the function, it can be very difficult to determine which parts of the
traceback are relevant and which aren't. In this case, the caller has
nothing to do with _compile, and the traceback looks like it's an
internal bug in a subroutine, when in fact it is actually due to bad
input. The experienced developer learns (by trial and error, possibly) to
ignore nearly half of the error message in this case.
And it can still be some work to figure out which parts of the traceback
are relevant, even after a couple years...
...
It need not be that way. This could, in principle, be left up to the
developer of the public function to specify (somehow!) that some specific
exceptions are expected, and should be treated as coming from public()
rather than from some internal subroutine. I don't have a concrete
proposal for such, although I did suggest a work-around. I expected
disinterest ("I don't see the point"). I didn't expect the level of
hostility to the idea that exceptions should (if and when possible) point
to the source of the error rather than some accidental implementation-
specific subroutine. Go figure.
My objection is not to the idea, but to the ad-hoc methods that would
currently be required. Resorting to passing exceptions in-band is a
step backwards. If python had a way to specify what level an exception
should be reported from, I might be interested. At this point, if
sparing the user one level of traceback was that high a priority to me,
I would make the validation be either a decorator, or have the
validation *be* the main routine, and the *real work* routine be the
private one.
~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Sun, 26 Dec 2010 17:12:50 -0800, misno...@gmail.com wrote:
> On Dec 23, 3:22 am, Steven D'Aprano +comp.lang.pyt...@pearwood.info> wrote:
>> You seem to have completely missed that there will be no bug report,
>> because this isn't a bug. (Or if it is a bug, the bug is elsewhere,
>> external to the function that raises the exception.) It is part of the
>> promised API. The fact that the exception is generated deep down some
>> chain of function calls is irrelevant.
>
> While the idea of being able to do this (create a general validation
> exception) sounds mildly appealing at first, what happens when the
> module implementing this documented API and documented error, has a bug?
> It seems that the user, or even module developer in the midst of
> writing, would now have no tools to easily tackle the problem, and no
> useful information to submit in the required bug report.
That's very true, but the same applies to *any* use of encapsulation. Any
time you hide information, you hide information (duh!). This doesn't stop
us from doing this:
def public(x):
if condition: return _private(x)
elif other_condition: return _more_private(x+1)
else: return _yet_another_private(x-1)
If you call public(42), and get the wrong answer, it's a bug, but the
source of the bug is hidden from the caller. If you have access to the
source code, you can work out where the bug lies (which of the three
private functions is buggy?) given the argument, but the return result
itself does not expose any information about where the bug lies. This is
considered an unavoidable but acceptable side-effect of an otherwise
desirable state of affairs: information hiding and encapsulation. The
caller being unaware of where and how the result is calculated is
considered a good thing, and the fact that it occasionally adds to the
debugging effort is considered such a trivial cost that it normally isn't
remarked upon, except by lunatics and die-hard fans of spaghetti code
using GOTO. But I repeat myself.
Why should exceptions *necessarily* be different? As I've repeatedly
acknowledged, for an unexpected exception (a bug), the developer needs
all the help he can get, and the current behaviour is the right way to do
it. You won't hear me argue differently. But for a documented, explicit,
expected, deliberate exception, Python breaks encapsulation by exposing
the internal details of any internal subroutines used to generate that
exception. This leads to messy tracebacks that obscure the source of bugs
in the caller's code:
>>> import re
>>> re.compile(r"(")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/re.py", line 190, in compile
return _compile(pattern, flags)
File "/usr/lib/python2.6/re.py", line 245, in _compile
raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis
I think critics of my position have forgotten what it's like to learning
the language. One of the most valuable skills to learn is to *ignore
parts of the traceback* -- a skill that takes practice and familiarity
with the library or function in use. To those who are less familiar with
the function, it can be very difficult to determine which parts of the
traceback are relevant and which aren't. In this case, the caller has
nothing to do with _compile, and the traceback looks like it's an
internal bug in a subroutine, when in fact it is actually due to bad
input. The experienced developer learns (by trial and error, possibly) to
ignore nearly half of the error message in this case.
In principle, the traceback could be roughly half as big, which means the
caller would find it half as difficult to read and understand:
>>> re.compile(r"(")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/re.py", line 190, in compile
raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis
With a one-to-one correspondence between the function called, and the
function reporting an error, it is easier to recognise that the error
lies in the input rather than some internal error in some subroutine you
have nothing to do with. Unfortunately there's no straightforward way to
consistently get this in Python without giving up the advantages of
delegating work to subroutines.
It need not be that way. This could, in principle, be left up to the
developer of the public function to specify (somehow!) that some specific
exceptions are expected, and should be treated as coming from public()
rather than from some internal subroutine. I don't have a concrete
proposal for such, although I did suggest a work-around. I expected
disinterest ("I don't see the point"). I didn't expect the level of
hostility to the idea that exceptions should (if and when possible) point
to the source of the error rather than some accidental implementation-
specific subroutine. Go figure.
--
Steven
--
http://mail.python.org/mailman/listinfo/
Re: How to pop the interpreter's stack?
On Sun, 26 Dec 2010 09:15:32 -0800, Ethan Furman wrote:
> Steven D'Aprano wrote:
>> Right. But I have thought of a clever trick to get the result KJ was
>> asking for, with the minimum of boilerplate code. Instead of this:
>>
>>
>> def _pre_spam(args):
>> if condition(args):
>> raise SomeException("message")
>> if another_condition(args):
>> raise AnotherException("message")
>> if third_condition(args):
>> raise ThirdException("message")
>>
>> def spam(args):
>> _pre_spam(args)
>> do_useful_work()
>>
>>
>> you can return the exceptions instead of raising them (exceptions are
>> just objects, like everything else!), and then add one small piece of
>> boilerplate to the spam() function:
>>
>>
>> def _pre_spam(args):
>> if condition(args):
>> return SomeException("message")
>> if another_condition(args):
>> return AnotherException("message")
>> if third_condition(args):
>> return ThirdException("message")
>>
>> def spam(args):
>> exc = _pre_spam(args)
>> if exc: raise exc
>> do_useful_work()
>
> -1
>
> You failed to mention that cleverness is not a prime requisite of the
> python programmer -- in fact, it's usually frowned upon. The big
> problem with the above code is you are back to passing errors in-band,
> pretty much completely defeating the point of have an out-of-band
> channel.
How is that any worse than making _pre_spam() a validation function that
returns a bool?
def spam(args):
flag = _pre_spam(args)
if flag: raise SomeException()
do_useful_work()
Is that also frowned upon for being too clever?
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 25, 2:49 pm, Steven D'Aprano wrote:
> On Sat, 25 Dec 2010 09:17:27 -0500, Robert Kern wrote:
> > On 12/24/10 5:14 PM, Ethan Furman wrote:
>
> >> There are also times when I change the exception being raised to match
> >> what python expects from that type of object -- for example, from
> >> WhatEverException to KeyError for a dict-like object. So in this regard
> >> I agree with Steven.
>
> > Steven isn't arguing that particular point here, nor is anyone arguing
> > against it.
>
> Emphasis on *here*.
>
> You will note that in Python 3, if you raise an exception inside an
> except block, both the *original* and the new exception are printed. This
> is great for fixing bugs inside except blocks, but terribly disruptive
> for catching one error and raising another error in it's place, e.g.:
>
> try:
> x+0
> except ValueError, TypeError as e:
> # x is not a numeric value, e.g. a string or a NAN.
> raise MyError('x is not a number')
>
> The explicit raise is assumed to indicate a bug in the except block, and
> the original exception is printed as well.
>
> But that's a separate issue from what is being discussed here. What we're
> discussing here is the idea that a function should be able to delegate
> work to private subroutines without the caller being aware of that fact.
> When you return a value, the caller doesn't see the internal details of
> how you calculated the value, but if you deliberately raise an exception,
> the caller does. Often this is the right thing to do, but sometimes it
> isn't. E.g. you can't delegate input validation to a subroutine and raise
> inside the subroutine without obfuscating the traceback.
>>> import re
>>> re.compile(r"(")
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.6/re.py", line 190, in compile
return _compile(pattern, flags)
File "/usr/lib/python2.6/re.py", line 245, in _compile
raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis
OHMYGOD HOW DARE the standard library allow the traceback list an
internal function that does input valididation!
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 26, 11:09 am, kj wrote: > In writes: > > >Except that the *caller* never gets the traceback (unless if it deliberately > >inspects the stack for some metaprogramming reason). It gets the exception, > >and > >that is the same no matter what you do. The developer/user gets the > >traceback, > >and those implementation details *are* often important to them. > > Just look at what Python shows you if you pass the wrong number of > arguments to a function: > > >>> def spam(x, y, z): pass > ... > >>> spam(1, 2) > > Traceback (most recent call last): > File "", line 1, in > TypeError: spam() takes exactly 3 arguments (2 given) > > That's it. The traceback stops at the point of the error. Python > doesn't show you all the underlying C-coded machinery that went > into detecting the error and emitting the error message. *No one* > needs this information at this point. All I'm saying is that I > want to do the same thing with my argument validation code as Python > does with its argument validation code: keep it out of sight. When > my argument validation code fires an exception ***there's no bug > in **my** code***. It's doing exactly what it's supposed to do. > Therefore, there's no need for me to debug anything, and certainly > no need for me to inspect the traceback all the way to the exception. > The bug is in the code that called my function with the wrong > arguments. The developer of that code has no more use for seeing > the traceback all the way to where my code raises the exception > than I have for seeing the traceback of Python's underlying C code > when I get an error like the one shown above. Python makes no attempt to hide its machinery in tracebacks (that I'm aware of); in fact stack points from internal Python functions, classes, and modules appear in tracebacks all the time. The reason you don't see traceback lines for Python's argument validation is it's written in C. If it bothers you that much, you're welcome to write you own argument validation in C, too. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 23, 3:22 am, Steven D'Aprano wrote: > You seem to have completely missed that there will be no bug report, > because this isn't a bug. (Or if it is a bug, the bug is elsewhere, > external to the function that raises the exception.) It is part of the > promised API. The fact that the exception is generated deep down some > chain of function calls is irrelevant. While the idea of being able to do this (create a general validation exception) sounds mildly appealing at first, what happens when the module implementing this documented API and documented error, has a bug? It seems that the user, or even module developer in the midst of writing, would now have no tools to easily tackle the problem, and no useful information to submit in the required bug report. -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/26/2010 2:14 PM, kj wrote: > In Ethan Furman > writes: > >> You failed to mention that cleverness is not a prime requisite of the >> python programmer -- in fact, it's usually frowned upon. > > That's the party line, anyway. I no longer believe it. I've been > crashing against one bit of cleverness after another in Python's > unification of types and classes... Well if you can find a way to implement a class system that doesn't use clever tricks *in its implementation* please let me know. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 PyCon 2011 Atlanta March 9-17 http://us.pycon.org/ See Python Video! http://python.mirocommunity.org/ Holden Web LLC http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/25/2010 2:50 PM, Steven D'Aprano wrote:
On Fri, 24 Dec 2010 10:51:32 -0800, John Nagle wrote:
On 12/24/2010 3:24 AM, Carl Banks wrote:
On Dec 24, 1:24 am, Steven D'Aprano wrote:
All I'm
suggesting is that there should be a way of reducing the boilerplate
needed for this idiom:
def _validate_arg(x):
if x == 'bad input': return False
return True
def f(arg):
if not _validate_arg(arg):
raise ValueError
process(arg)
to something more natural that doesn't needlessly expose
implementation details that are completely irrelevant to the caller.
How about
raise ValueError("Bad input %s to validate_arg" % (repr(arg),))
You can pass arguments to most exceptions, and the content of the
exception is determined entirely by the code raising it.
I know that exceptions can take arguments (usually string error
messages). I was writing in short-hand. My apologies, I thought that
would have been obvious :(
Perhaps you have missed the context of the discussion. The context is
that the called function delegates the job of validating input to a
private function, which should be hidden from the caller (it's private,
not part of the public API, subject to change, hidden, etc.) but
tracebacks expose that information, obscuring the cause of the fault.
(The fault being bad input to the public function, not an accidental bug
in the private function.)
If end users are seeing uncaught tracebacks, the program is broken.
Well, perhaps, but that's a separate issue. We're talking about the
caller of the function seeing internal details, not the end use.
No, that is the issue, unless the program itself is examining the
stack traceback data. Python exception-catching has no notion of
what code raised the exception. Only the contents of the exception
object are normally available. So the "private function" is always
"hidden", unless you're debugging, in which case it shouldn't be
hidden.
Traceback is purely a debugging feature. In some Python
implementations, such as Shed Skin, you don't get tracebacks unless
you're running under a debugger.
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
In Ethan Furman writes: >You failed to mention that cleverness is not a prime requisite of the >python programmer -- in fact, it's usually frowned upon. That's the party line, anyway. I no longer believe it. I've been crashing against one bit of cleverness after another in Python's unification of types and classes... -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
In writes: >Except that the *caller* never gets the traceback (unless if it deliberately >inspects the stack for some metaprogramming reason). It gets the exception, >and >that is the same no matter what you do. The developer/user gets the traceback, >and those implementation details *are* often important to them. Just look at what Python shows you if you pass the wrong number of arguments to a function: >>> def spam(x, y, z): pass ... >>> spam(1, 2) Traceback (most recent call last): File "", line 1, in TypeError: spam() takes exactly 3 arguments (2 given) That's it. The traceback stops at the point of the error. Python doesn't show you all the underlying C-coded machinery that went into detecting the error and emitting the error message. *No one* needs this information at this point. All I'm saying is that I want to do the same thing with my argument validation code as Python does with its argument validation code: keep it out of sight. When my argument validation code fires an exception ***there's no bug in **my** code***. It's doing exactly what it's supposed to do. Therefore, there's no need for me to debug anything, and certainly no need for me to inspect the traceback all the way to the exception. The bug is in the code that called my function with the wrong arguments. The developer of that code has no more use for seeing the traceback all the way to where my code raises the exception than I have for seeing the traceback of Python's underlying C code when I get an error like the one shown above. ~kj -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Steven D'Aprano wrote:
Right. But I have thought of a clever trick to get the result KJ was
asking for, with the minimum of boilerplate code. Instead of this:
def _pre_spam(args):
if condition(args):
raise SomeException("message")
if another_condition(args):
raise AnotherException("message")
if third_condition(args):
raise ThirdException("message")
def spam(args):
_pre_spam(args)
do_useful_work()
you can return the exceptions instead of raising them (exceptions are
just objects, like everything else!), and then add one small piece of
boilerplate to the spam() function:
def _pre_spam(args):
if condition(args):
return SomeException("message")
if another_condition(args):
return AnotherException("message")
if third_condition(args):
return ThirdException("message")
def spam(args):
exc = _pre_spam(args)
if exc: raise exc
do_useful_work()
-1
You failed to mention that cleverness is not a prime requisite of the
python programmer -- in fact, it's usually frowned upon. The big
problem with the above code is you are back to passing errors in-band,
pretty much completely defeating the point of have an out-of-band channel.
~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Steven D'Aprano wrote: On Sat, 25 Dec 2010 09:17:27 -0500, Robert Kern wrote: On 12/24/10 5:14 PM, Ethan Furman wrote: There are also times when I change the exception being raised to match what python expects from that type of object -- for example, from WhatEverException to KeyError for a dict-like object. So in this regard I agree with Steven. Steven isn't arguing that particular point here, nor is anyone arguing against it. Emphasis on *here*. You will note that in Python 3, if you raise an exception inside an except block, both the *original* and the new exception are printed. This is great for fixing bugs inside except blocks, but terribly disruptive for catching one error and raising another error in it's place... Yes, this is where I was agreeing with Steven. While I love python3, the current nested exception behavior is horrible. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Carl Banks wrote: Python is not, and never has been, about hiding internal details. It's about openness, and there's no reason a traceback should hide internal details any more than a class should--in fact hiding information in the traceback is far worse, because you're suppressing information that could be crucial for debugging. +100 ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 25, 6:21 am, Robert Kern wrote: > On 12/24/10 4:24 AM, Steven D'Aprano wrote: > > > On Thu, 23 Dec 2010 22:38:05 -0800, Carl Banks wrote: > >> OTOH, going the extra mile to hide useful information from a user is > >> asinine. As a user, I will decide for myself how I want to use > >> implementation-defined information, and I don't want the implementor to > >> decide this for me. It's bad enough if an implementor fails to provide > >> information out of laziness, but when they deliberately do extra work to > >> hide information, that's self-importance and arrogance. > > > But that of course is nonsense, because as the user you don't decide > > anything of the sort. The developer responsible for writing the function > > decides what information he provides you, starting with whether you get > > an exception at all, where it comes from, the type of exception, and the > > error message (if any). > > Carl isn't arguing that the user is or should be responsible for this sort of > thing. He is arguing that developers should be responsible for doing this in > such a way that is beneficial for the developer/user down the road. I'm not even arguing that; I think I would be content if the developer merely doesn't actively work to harm the user. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 25, 2:49 pm, Steven D'Aprano wrote: > But that's a separate issue from what is being discussed here. What we're > discussing here is the idea that a function should be able to delegate > work to private subroutines without the caller being aware of that fact. I can't fathom any possible reason why this could be considered a good thing, especially in Python where the culture is, "We're all adults here". By the same logic, it would be a good idea to prevent the user from accessing private members of a class, internal variables of a module, or importing an internal module directly. There is a convention in Python: internal objects are preceded by underscore. The fact that your internal function is marked this way in the traceback is more than enough information to the user that this is an internal function. Python is not, and never has been, about hiding internal details. It's about openness, and there's no reason a traceback should hide internal details any more than a class should--in fact hiding information in the traceback is far worse, because you're suppressing information that could be crucial for debugging. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Fri, 24 Dec 2010 10:51:32 -0800, John Nagle wrote:
> On 12/24/2010 3:24 AM, Carl Banks wrote:
>> On Dec 24, 1:24 am, Steven D'Aprano> +comp.lang.pyt...@pearwood.info> wrote:
> All I'm
>>> suggesting is that there should be a way of reducing the boilerplate
>>> needed for this idiom:
>>>
>>> def _validate_arg(x):
>>> if x == 'bad input': return False
>>> return True
>>>
>>> def f(arg):
>>> if not _validate_arg(arg):
>>> raise ValueError
>>> process(arg)
>>>
>>> to something more natural that doesn't needlessly expose
>>> implementation details that are completely irrelevant to the caller.
>
> How about
>
> raise ValueError("Bad input %s to validate_arg" % (repr(arg),))
>
> You can pass arguments to most exceptions, and the content of the
> exception is determined entirely by the code raising it.
I know that exceptions can take arguments (usually string error
messages). I was writing in short-hand. My apologies, I thought that
would have been obvious :(
Perhaps you have missed the context of the discussion. The context is
that the called function delegates the job of validating input to a
private function, which should be hidden from the caller (it's private,
not part of the public API, subject to change, hidden, etc.) but
tracebacks expose that information, obscuring the cause of the fault.
(The fault being bad input to the public function, not an accidental bug
in the private function.)
> If end users are seeing uncaught tracebacks, the program is broken.
Well, perhaps, but that's a separate issue. We're talking about the
caller of the function seeing internal details, not the end use.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Sat, 25 Dec 2010 09:17:27 -0500, Robert Kern wrote:
> On 12/24/10 5:14 PM, Ethan Furman wrote:
>
>> There are also times when I change the exception being raised to match
>> what python expects from that type of object -- for example, from
>> WhatEverException to KeyError for a dict-like object. So in this regard
>> I agree with Steven.
>
> Steven isn't arguing that particular point here, nor is anyone arguing
> against it.
Emphasis on *here*.
You will note that in Python 3, if you raise an exception inside an
except block, both the *original* and the new exception are printed. This
is great for fixing bugs inside except blocks, but terribly disruptive
for catching one error and raising another error in it's place, e.g.:
try:
x+0
except ValueError, TypeError as e:
# x is not a numeric value, e.g. a string or a NAN.
raise MyError('x is not a number')
The explicit raise is assumed to indicate a bug in the except block, and
the original exception is printed as well.
But that's a separate issue from what is being discussed here. What we're
discussing here is the idea that a function should be able to delegate
work to private subroutines without the caller being aware of that fact.
When you return a value, the caller doesn't see the internal details of
how you calculated the value, but if you deliberately raise an exception,
the caller does. Often this is the right thing to do, but sometimes it
isn't. E.g. you can't delegate input validation to a subroutine and raise
inside the subroutine without obfuscating the traceback.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/24/10 4:24 AM, Steven D'Aprano wrote: On Thu, 23 Dec 2010 22:38:05 -0800, Carl Banks wrote: OTOH, going the extra mile to hide useful information from a user is asinine. As a user, I will decide for myself how I want to use implementation-defined information, and I don't want the implementor to decide this for me. It's bad enough if an implementor fails to provide information out of laziness, but when they deliberately do extra work to hide information, that's self-importance and arrogance. But that of course is nonsense, because as the user you don't decide anything of the sort. The developer responsible for writing the function decides what information he provides you, starting with whether you get an exception at all, where it comes from, the type of exception, and the error message (if any). Carl isn't arguing that the user is or should be responsible for this sort of thing. He is arguing that developers should be responsible for doing this in such a way that is beneficial for the developer/user down the road. Once this information has been passed on to you, you're free to do anything you like with it, but you never get to choose what information you get -- I'm not suggesting any change there. All I'm suggesting is that there should be a way of reducing the boilerplate needed for this idiom: def _validate_arg(x): if x == 'bad input': return False return True def f(arg): if not _validate_arg(arg): raise ValueError process(arg) to something more natural that doesn't needlessly expose implementation details that are completely irrelevant to the caller. Except that the *caller* never gets the traceback (unless if it deliberately inspects the stack for some metaprogramming reason). It gets the exception, and that is the same no matter what you do. The developer/user gets the traceback, and those implementation details *are* often important to them. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/24/10 5:14 PM, Ethan Furman wrote: There are also times when I change the exception being raised to match what python expects from that type of object -- for example, from WhatEverException to KeyError for a dict-like object. So in this regard I agree with Steven. Steven isn't arguing that particular point here, nor is anyone arguing against it. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Carl Banks wrote: On Dec 24, 1:24 am, Steven D'Aprano wrote: On Thu, 23 Dec 2010 22:38:05 -0800, Carl Banks wrote: OTOH, going the extra mile to hide useful information from a user is asinine. As a user, I will decide for myself how I want to use implementation-defined information, and I don't want the implementor to decide this for me. It's bad enough if an implementor fails to provide information out of laziness, but when they deliberately do extra work to hide information, that's self-importance and arrogance. But that of course is nonsense, because as the user you don't decide anything of the sort. As a user I can criticize the decision of the implementor to needlessly filter information, and declare that it's borne out of the author's arrogance in thinking he knows what I want when I get a traceback. I can also opine that Python language shouldn't make it easy for library implementors to be arrogant like this. The developer responsible for writing the function decides what information he provides you, starting with whether you get an exception at all, where it comes from, the type of exception, and the error message (if any). Once this information has been passed on to you, you're free to do anything you like with it, but you never get to choose what information you get -- I'm not suggesting any change there. All I'm suggesting is that there should be a way of reducing the boilerplate needed for this idiom: def _validate_arg(x): if x == 'bad input': return False return True def f(arg): if not _validate_arg(arg): raise ValueError process(arg) to something more natural that doesn't needlessly expose implementation details that are completely irrelevant to the caller. Arrogance. Who gave you the right to decide what is completely irrelevant to user? I, as the user, decide what's relevant. If I want implementation-dependent information, it's my business. I don't want the language to make it easy for arrogant people, who think they know what information I want better than I do, to hide that information from me. One of the many things I love about Python is that it stays out of the way of me getting my work done. I think a truly pythonic program/library/module must do the same. So in this regard I agree with Carl. There are also times when I change the exception being raised to match what python expects from that type of object -- for example, from WhatEverException to KeyError for a dict-like object. So in this regard I agree with Steven. For kj's concern, which seems to be along the lines of functional as opposed to custom object, I don't think the traceback should be monkied with -- either use a decorator to keep the traceback short, or give the _pre_func name a good name and don't worry about it. I know when I see a traceback, I start at the bottom and only work my way up if I need to. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/24/2010 3:24 AM, Carl Banks wrote:
On Dec 24, 1:24 am, Steven D'Aprano wrote:
All I'm
suggesting is that there should be a way of reducing the boilerplate
needed for this idiom:
def _validate_arg(x):
if x == 'bad input': return False
return True
def f(arg):
if not _validate_arg(arg):
raise ValueError
process(arg)
to something more natural that doesn't needlessly expose implementation
details that are completely irrelevant to the caller.
How about
raise ValueError("Bad input %s to validate_arg" % (repr(arg),))
You can pass arguments to most exceptions, and the content of the
exception is determined entirely by the code raising it.
If end users are seeing uncaught tracebacks, the program is broken.
It's usually worth it to catch EnvironmentError near the outermost
level of the program, since most non program bug events, like I/O
and network errors. will raise some subclass of EnvironmentError.
John Nagle
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 24, 1:24 am, Steven D'Aprano wrote: > On Thu, 23 Dec 2010 22:38:05 -0800, Carl Banks wrote: > > OTOH, going the extra mile to hide useful information from a user is > > asinine. As a user, I will decide for myself how I want to use > > implementation-defined information, and I don't want the implementor to > > decide this for me. It's bad enough if an implementor fails to provide > > information out of laziness, but when they deliberately do extra work to > > hide information, that's self-importance and arrogance. > > But that of course is nonsense, because as the user you don't decide > anything of the sort. As a user I can criticize the decision of the implementor to needlessly filter information, and declare that it's borne out of the author's arrogance in thinking he knows what I want when I get a traceback. I can also opine that Python language shouldn't make it easy for library implementors to be arrogant like this. > The developer responsible for writing the function > decides what information he provides you, starting with whether you get > an exception at all, where it comes from, the type of exception, and the > error message (if any). Once this information has been passed on to you, > you're free to do anything you like with it, but you never get to choose > what information you get -- I'm not suggesting any change there. All I'm > suggesting is that there should be a way of reducing the boilerplate > needed for this idiom: > > def _validate_arg(x): > if x == 'bad input': return False > return True > > def f(arg): > if not _validate_arg(arg): > raise ValueError > process(arg) > > to something more natural that doesn't needlessly expose implementation > details that are completely irrelevant to the caller. Arrogance. Who gave you the right to decide what is completely irrelevant to user? I, as the user, decide what's relevant. If I want implementation-dependent information, it's my business. I don't want the language to make it easy for arrogant people, who think they know what information I want better than I do, to hide that information from me. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Thu, 23 Dec 2010 22:38:05 -0800, Carl Banks wrote: >> Do you accept that, as a general principle, unhandled errors should be >> reported as close as possible to where the error occurs? If your answer >> to that is No, then where do you think unhandled errors should be >> reported? > > "No", and "where the error is detected". That is, what Python does now. > Trying to figure out where the error "occurred" is fool's errand. But isn't that what debugging is all about -- finding where the error occurred and fixing it? Hardly a fool's errand. > The > former isn't even well-defined, let alone something a compiler or user > can be expected to reliably report. Sometimes the error doesn't even > "occur" in the same call stack. Thank you for taking the time to respond. I think your objection misses the point I'm trying to make completely. But since this argument is rather academic, and it's Christmas Eve here, I'll just make one last comment and leave it at that: > OTOH, going the extra mile to hide useful information from a user is > asinine. As a user, I will decide for myself how I want to use > implementation-defined information, and I don't want the implementor to > decide this for me. It's bad enough if an implementor fails to provide > information out of laziness, but when they deliberately do extra work to > hide information, that's self-importance and arrogance. But that of course is nonsense, because as the user you don't decide anything of the sort. The developer responsible for writing the function decides what information he provides you, starting with whether you get an exception at all, where it comes from, the type of exception, and the error message (if any). Once this information has been passed on to you, you're free to do anything you like with it, but you never get to choose what information you get -- I'm not suggesting any change there. All I'm suggesting is that there should be a way of reducing the boilerplate needed for this idiom: def _validate_arg(x): if x == 'bad input': return False return True def f(arg): if not _validate_arg(arg): raise ValueError process(arg) to something more natural that doesn't needlessly expose implementation details that are completely irrelevant to the caller. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 23, 8:55 pm, Steven D'Aprano wrote:
> On Thu, 23 Dec 2010 02:54:52 -0800, Carl Banks wrote:
> > On Dec 22, 7:22 pm, Steven D'Aprano > +comp.lang.pyt...@pearwood.info> wrote:
> >> There should be a mechanism for Python functions to distinguish between
> >> unexpected exceptions (commonly known as "bugs"), which should be
> >> reported as coming from wherever they come from, and documented,
> >> expected exceptions, which should be reported as coming from the
> >> function regardless of how deep the function call stack really is.
>
> > No, -100. The traceback isn't the place for this. I've never disagreed
> > with you more, and I've disagreed with you and awful lot.
>
> Okay, it's your right to disagree, but I am trying to understand your
> reasons for disagreeing, and I simply don't get it.
>
> I'm quite frustrated that you don't give any reasons for why you think
> it's not just unnecessary but actually *horrible* to hide implementation
> details such as where data validation is performed. I hope you'll try to
> explain *why* you think it's a bad idea, rather than just continue
> throwing out dismissive statements about "self-important" programmers
> (your earlier post to KJ) and "never disagreed more" (to me).
>
> Do you accept that, as a general principle, unhandled errors should be
> reported as close as possible to where the error occurs?
> If your answer to that is No, then where do you think unhandled errors
> should be reported?
"No", and "where the error is detected". That is, what Python does
now. Trying to figure out where the error "occurred" is fool's
errand. The former isn't even well-defined, let alone something a
compiler or user can be expected to reliably report. Sometimes the
error doesn't even "occur" in the same call stack.
There's a similar fine line between a bug exception and bad input
exception, and it's foolish to distinguish them in a reliable way: in
particular bugs can easily be mistaken for bad input.
OTOH, going the extra mile to hide useful information from a user is
asinine. As a user, I will decide for myself how I want to use
implementation-defined information, and I don't want the implementor
to decide this for me. It's bad enough if an implementor fails to
provide information out of laziness, but when they deliberately do
extra work to hide information, that's self-importance and arrogance.
The traceback IS NOT THE PLACE for these kinds of games.
> Now, given the scenario I proposed earlier:
>
> >>> f('bad input') <=== error occurs here
>
> Traceback (most recent call last):
> File "", line 1, in
> File "", line 2, in f
> File "", line 2, in g
> File "", line 2, in h
> File "", line 2, in i
> File "", line 2, in j
> File "", line 2, in k <=== far from the source of error
> ValueError
>
> do you concede that the actual error occurs at the time 'bad input' is
> passed to f, and not further down the stack where k happens to raise an
> exception? If not, where do you think the error occurs, and why?
This question is irrelevant. It doesn't matter where the mistake is
made.
Carl Banks
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Thu, 23 Dec 2010 02:54:52 -0800, Carl Banks wrote:
> On Dec 22, 7:22 pm, Steven D'Aprano +comp.lang.pyt...@pearwood.info> wrote:
>> There should be a mechanism for Python functions to distinguish between
>> unexpected exceptions (commonly known as "bugs"), which should be
>> reported as coming from wherever they come from, and documented,
>> expected exceptions, which should be reported as coming from the
>> function regardless of how deep the function call stack really is.
>
> No, -100. The traceback isn't the place for this. I've never disagreed
> with you more, and I've disagreed with you and awful lot.
Okay, it's your right to disagree, but I am trying to understand your
reasons for disagreeing, and I simply don't get it.
I'm quite frustrated that you don't give any reasons for why you think
it's not just unnecessary but actually *horrible* to hide implementation
details such as where data validation is performed. I hope you'll try to
explain *why* you think it's a bad idea, rather than just continue
throwing out dismissive statements about "self-important" programmers
(your earlier post to KJ) and "never disagreed more" (to me).
Do you accept that, as a general principle, unhandled errors should be
reported as close as possible to where the error occurs?
If your answer to that is No, then where do you think unhandled errors
should be reported?
Now, given the scenario I proposed earlier:
>>> f('bad input')<=== error occurs here
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in f
File "", line 2, in g
File "", line 2, in h
File "", line 2, in i
File "", line 2, in j
File "", line 2, in k<=== far from the source of error
ValueError
do you concede that the actual error occurs at the time 'bad input' is
passed to f, and not further down the stack where k happens to raise an
exception? If not, where do you think the error occurs, and why?
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 22, 7:22 pm, Steven D'Aprano wrote: > There should be a mechanism for Python functions to distinguish between > unexpected exceptions (commonly known as "bugs"), which should be > reported as coming from wherever they come from, and documented, expected > exceptions, which should be reported as coming from the function > regardless of how deep the function call stack really is. No, -100. The traceback isn't the place for this. I've never disagreed with you more, and I've disagreed with you and awful lot. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Wed, 22 Dec 2010 13:53:20 -0800, Carl Banks wrote:
> On Dec 22, 8:52 am, kj wrote:
>> In Robert Kern
>> writes:
>>
>> >Obfuscating the location that an exception gets raised prevents a lot
>> >of debugging...
>>
>> The Python interpreter does a lot of that "obfuscation" already, and I
>> find the resulting tracebacks more useful for it.
>>
>> An error message is only useful to a given audience if that audience
>> can use the information in the message to modify what they are doing to
>> avoid the error.
>
> So when the audience files a bug report it's not useful for them to
> include the whole traceback?
Well, given the type of error KJ has been discussing, no, it isn't useful.
Fault: function raises documented exception when passed input that
is documented as being invalid
What steps will reproduce the problem?
1. call the function with invalid input
2. read the exception that is raised
3. note that it is the same exception as documented
What is the expected output? What do you see instead?
Excepted somebody to hit me on the back of the head and tell me
not to call the function with invalid input. Instead I just got
an exception.
You seem to have completely missed that there will be no bug report,
because this isn't a bug. (Or if it is a bug, the bug is elsewhere,
external to the function that raises the exception.) It is part of the
promised API. The fact that the exception is generated deep down some
chain of function calls is irrelevant.
The analogy is this: imagine a function that delegates processing of the
return result to different subroutines:
def func(arg):
if arg > 0:
return _inner1(arg)
else:
return _inner2(arg)
This is entirely irrelevant to the caller. When they receive the return
result from calling func(), they have no way of knowing where the result
came from, and wouldn't care even if they could. Return results hide
information about where the result was calculated, as they should. Why
shouldn't deliberate, explicit, documented exceptions be treated the same?
Tracebacks expose the implementation details of where the exception was
generated. This is the right behaviour if the exception is unexpected --
a bug internal to func -- since you need knowledge of the implementation
of func in order to fix the unexpected exception. So far so good -- we
accept that Python's behaviour under these circumstances is correct.
But this is not the right behaviour when the exception is expected, e.g.
an explicitly raised exception in response to an invalid argument. In
this case, the traceback exposes internal details of no possible use to
the caller. What does the caller care if func() delegates (e.g.) input
checking to a subroutine? The subroutine is an irrelevant implementation
detail. The exception is promised output of the function, just as much so
as if it were a return value.
Consider the principle that exceptions should be dealt with as close as
possible to the actual source of the problem:
>>> f('good input')
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in f
File "", line 2, in g
File "", line 2, in h
File "", line 2, in i
File "", line 2, in j
File "", line 2, in k<=== error occurs here, and shown here
ValueError
But now consider the scenario where the error is not internal to f, but
external. The deeper down the stack trace you go, the further away from
the source of the error you get. The stack trace now obscures the source
of the error, rather than illuminating it:
>>> f('bad input')<=== error occurs here
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in f
File "", line 2, in g
File "", line 2, in h
File "", line 2, in i
File "", line 2, in j
File "", line 2, in k<=== far from the source of error
ValueError
There's no point in inspecting function k for a bug when the problem has
nothing to do with k. The problem is that the input fails to match the
pre-conditions for f. From the perspective of the caller, the error has
nothing to do with k, k is a meaningless implementation detail, and the
source of the error is the mismatch between the input and what f expects.
And so by the principle of dealing with exceptions as close as possible
to the source of the error, we should desire this traceback instead:
>>> f('bad input')<=== error occurs here
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in f<=== matches where the error occurs
ValueError
In the absence of any practical way for function f to know whether an
arbitrary exception in a subroutine is a bug or not, the least-worst
decision is Python's current behaviour: take the conservative, risk-
adverse path and assume the worst, treat the exception as a bug in the
subroutine, and expose the entire stack trace.
But, I suggest, we can do better using the usual Python strategy of
implementin
Re: How to pop the interpreter's stack?
In <1f47c36d-a509-4d05-ba79-62b4a534b...@j19g2000prh.googlegroups.com> Carl Banks writes: >On Dec 22, 8:52=A0am, kj wrote: >> In Robert Kern t.k...@gmail.com> writes: >> >> >Obfuscating the location that an exception gets raised prevents a lot of >> >debugging... >> >> The Python interpreter does a lot of that "obfuscation" already, and I >> find the resulting tracebacks more useful for it. >> >> An error message is only useful to a given audience if that audience >> can use the information in the message to modify what they are >> doing to avoid the error. >> =A0It is of no use (certainly no *immediate* >> use) to this audience to see tracebacks that go deep into code that >> they don't know anything about and cannot change. >So when the audience files a bug report it's not useful for them to >include the whole traceback? Learn to read, buster. I wrote *immediate* use. ~kj -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 22, 8:52 am, kj wrote: > In Robert Kern > writes: > > >Obfuscating the location that an exception gets raised prevents a lot of > >debugging... > > The Python interpreter does a lot of that "obfuscation" already, and I > find the resulting tracebacks more useful for it. > > An error message is only useful to a given audience if that audience > can use the information in the message to modify what they are > doing to avoid the error. So when the audience files a bug report it's not useful for them to include the whole traceback? > It is of no use (certainly no *immediate* > use) to this audience to see tracebacks that go deep into code that > they don't know anything about and cannot change. Seriously, quit trying to do the user favors. There's nothing that pisses me off than a self-important developer thinking he knows what the user wants more than the user does. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
In Robert Kern
writes:
>Obfuscating the location that an exception gets raised prevents a lot of
>debugging...
The Python interpreter does a lot of that "obfuscation" already, and I
find the resulting tracebacks more useful for it.
An error message is only useful to a given audience if that audience
can use the information in the message to modify what they are
doing to avoid the error. It is of no use (certainly no *immediate*
use) to this audience to see tracebacks that go deep into code that
they don't know anything about and cannot change.
For example, consider this:
#-
def foo(x, **k): pass
def bar(*a, **k):
if len(a) > 1: raise TypeError('too many args')
def baz(*a, **k): _pre_baz(*a, **k)
def _pre_baz(*a, **k):
if len(a) > 1: raise TypeError('too many args')
if __name__ == '__main__':
from traceback import print_exc
try: foo(1, 2)
except: print_exc()
try: bar(1, 2)
except: print_exc()
try: baz(1, 2)
except: print_exc()
#-
(The code in the "if __name__ == '__main__'" section is meant to
simulate the general case in which the functions defined in this file
are called by third-party code.) When you run this code the output is
this (a few blank lines added for clarity):
Traceback (most recent call last):
File "/tmp/ex2.py", line 5, in
try: foo(1, 2)
TypeError: foo() takes exactly 1 argument (2 given)
Traceback (most recent call last):
File "/tmp/ex2.py", line 7, in
try: bar(1, 2)
File "/tmp/example.py", line 4, in bar
if len(a) > 1: raise TypeError('too many args')
TypeError: too many args
Traceback (most recent call last):
File "/tmp/ex2.py", line 9, in
try: baz(1, 2)
File "/tmp/example.py", line 6, in baz
def baz(*a, **k): _pre_baz(*a, **k)
File "/tmp/example.py", line 9, in _pre_baz
if len(a) > 1: raise TypeError('too many args')
TypeError: too many args
In all cases, the *programming* errors are identical: functions called
with the wrong arguments. The traceback from foo(1, 2) tells me this
very clearly, and I'm glad that Python is not also giving me the
traceback down to where the underlying C code throws the exception: I
don't need to see all this machinery.
In contrast, the tracebacks from bar(1, 2) and baz(1, 2) obscure the
fundamental problem with useless detail. From the point of view of
the programmer that is using these functions, it is of no use to know
that the error resulted from some "raise TypeError" statement
somewhere, let alone that this happened in some obscure, private
function _pre_baz.
Perhaps I should have made it clearer in my original post that the
tracebacks I want to clean up are those from exceptions *explicitly*
raised by my argument-validating helper function, analogous to
_pre_baz above. I.e. I want that when my spam function is called
(by code written by someone else) with the wrong arguments, the
traceback looks more like this
Traceback (most recent call last):
File "/some/python/code.py", line 123, in
spam(some, bad, args)
TypeError: the second argument is bad
than like this:
Traceback (most recent call last):
File "/some/python/code.py", line 123, in
spam(some, bad, args)
File "/my/niftymodule.py", line 456, in niftymodule
_pre_spam(*a, **k)
File "/my/niftymodule.py", line 789, in __pre_spam
raise TypeError('second argument to spam is bad')
TypeError: the second argument is bad
In my opinion, the idea that more is always better in a traceback
is flat out wrong. As the example above illustrates, the most
useful traceback is the one that stops at the deepest point where
the *intended audience* for the traceback can take action, and goes
no further. The intended audience for the errors generated by my
argument-checking functions should see no further than the point
where they called a function incorrectly.
~kj
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/16/10 6:33 PM, Steven D'Aprano wrote:
On Thu, 16 Dec 2010 10:39:34 -0600, Robert Kern wrote:
On 12/16/10 10:23 AM, Steven D'Aprano wrote:
On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:
Tim Arnold wrote:
"Ethan Furman" wrote in message
news:mailman.4.1292379995.6505.python-l...@python.org...
kj wrote:
The one thing I don't like about this strategy is that the
tracebacks of exceptions raised during the execution of __pre_spam
include one unwanted stack level (namely, the one corresponding to
__pre_spam itself).
[...]
A decorator was one of the items kj explicity didn't want. Also,
while it would have a shallower traceback for exceptions raised during
the __pre_spam portion, any exceptions raised during spam itself would
then be one level deeper than desired... while that could be masked by
catching and (re-?)raising the exception in the decorator, Steven had
a very good point about why that is a bad idea -- namely, tracebacks
shouldn't lie about where the error is.
True, very true... but many hours later, it suddenly hit me that what
KJ was asking for wasn't *necessarily* such a bad idea. My thought is,
suppose you have a function spam(x) which raises an exception. If it's
a *bug*, then absolutely you need to see exactly where the error
occurred, without the traceback being mangled or changed in any way.
But what if the exception is deliberate, part of the function's
documented behaviour? Then you might want the exception to appear to
come from the function spam even if it was actually generated inside
some private sub-routine.
Obfuscating the location that an exception gets raised prevents a lot of
debugging (by inspection or by pdb), even if the exception is
deliberately raised with an informative error message. Not least, the
code that decides to raise that exception may be buggy. But even if the
actual error is outside of the function (e.g. the caller is passing bad
arguments), you want to at least see what tests the __pre_spam function
is doing in order to decide to raise that exception.
And how do you think you see that from the traceback? The traceback
prints the line which actually raises the exception (and sometimes not
even that!), which is likely to be a raise statement:
import example
example.func(42)
Traceback (most recent call last):
File "", line 1, in
File "example.py", line 3, in func
raise ValueError('bad value for x')
ValueError: bad value for x
The actual test is:
def func(x):
if x> 10 and x%2 == 0:
raise ValueError('bad value for x')
but you can't get that information from the traceback.
But I can get the line number and trivially go look it up. If we elide that
stack frame, I have to go hunting and possibly make some guesses. Depending on
the organization of the code, I may have to make some guesses anyways, but if I
keep the decision to raise an exception close to the actual raising of the
exception, it makes things a lot easier.
Python's exception system has to handle two different situations: buggy
code, and bad data. It's not even clear whether there is a general
distinction to be made between the two, but even if there's not a general
distinction, there's certainly a distinction which we can *sometimes*
make. If a function contains a bug, we need all the information we can
get, including the exact line that causes the fault. But if the function
deliberately raises an exception due to bad input, we don't need any
information regarding the internals of the function (assuming that the
exception is sufficiently detailed, a big assumption I grant you!). If I
re-wrote the above func() like this:
def func(x):
if !(x<= 10):
if x%2 != 0:
pass
else:
raise ValueError('bad value for x')
return
I would have got the same traceback, except the location of the exception
would have been different (line 6, in a nested if-block). To the caller,
whether I had written the first version of func() or the second is
irrelevant. If I had passed the input validation off to a second
function, that too would be irrelevant.
The caller doesn't care about tracebacks one way or the other, either. Only
someone *viewing* the traceback cares as well as debuggers like pdb. Eliding the
stack frame neither helps nor harms the caller, but it does substantially harm
the developer viewing tracebacks or using a debugger.
I don't expect Python to magically know whether an exception is a bug or
not, but there's something to be said for the ability to turn Python
functions into black boxes with their internals invisible, like C
functions already are. If (say) math.atan2(y, x) raises an exception, you
have no way of knowing whether atan2 is a single monolithic function, or
whether it is split into multiple pieces. The location of the exception
is invisible to the caller: all you can see is that atan2 raised an
exception.
And that has frustrated my debugging efforts more often than I can
Re: How to pop the interpreter's stack?
On 12/16/2010 7:33 PM, Steven D'Aprano wrote:
Python's exception system has to handle two different situations: buggy
code, and bad data. It's not even clear whether there is a general
distinction to be made between the two, but even if there's not a general
distinction, there's certainly a distinction which we can *sometimes*
make.
The two are intertwined. Production code that passes bad data to a
function without catching the exception is buggy.
def func(x):
if !(x<= 10):
if x%2 != 0:
pass
else:
raise ValueError('bad value for x')
return
I would have got the same traceback,
A traceback is printed only if the code passes bad data and does not
catch the exception. Tracebacks are for developers, not for users.
--
Terry Jan Reedy
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On 12/14/2010 6:31 PM, Ethan Furman wrote: kj wrote: The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). __pre_spam should be completely invisible and unobtrusive I am unaware of any way to accomplish what you desire. I also think this is one of those things that's not worth fighting -- how often are you going to see such a traceback? When somebody makes a coding mistake? Right. If you are worried about what the user sees in a traceback, you are doing it wrong. Consider reporting detailed error information via the logging module, for example. John Nagle -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Dec 15, 2:16 am, Steven D'Aprano wrote: > On Tue, 14 Dec 2010 21:14:35 +, kj wrote: > > Consider this code: > > > def spam(*args, **kwargs): > > args, kwargs = __pre_spam(*args, **kwargs) > > > # args & kwargs are OK: proceed > > # ... > > > def __pre_spam(*args, **kwargs): > > # validate args & kwargs; > > # return canonicalized versions of args & kwargs; # on failure, > > raise some *informative* exception # ... > > return canonicalized_args, canonicalized_kwargs > > Double leading underscores don't have any special meaning in the global > scope. Save yourself an underscore and call it _pre_spam instead :) > > In fact, even if spam and __pre_spam are methods, it's probably a good > idea to avoid the double-underscore name mangling. It's usually more > trouble than it's worth. > > > I write functions like __pre_spam for one reason only: to remove clutter > > from a corresponding spam function that has a particularly complex > > argument-validation/canonicalization stage. In effect, spam > > "outsources" to __pre_spam the messy business of checking and > > conditioning its arguments. > > A perfectly sensible arrangement. > > > The one thing I don't like about this strategy is that the tracebacks of > > exceptions raised during the execution of __pre_spam include one > > unwanted stack level (namely, the one corresponding to __pre_spam > > itself). > > But why is it unwanted? The traceback shows where the error occurs -- it > occurs in __pre_spam, not spam, or __post_spam, or spam_caller, or > anywhere else. Even if it's possible, having the traceback *lie* about > where it occurs is a bad idea which will cause confusion to anyone trying > to maintain the software in the future. I don't agree with kj's usage, but I have customized the traceback to remove items before. In my case it was to remove lines for endemic wrapper functions. The traceback lines showing the wrapper functions in the stack were useless, and since pretty much every function was wrapped it meant half the lines in that traceback were useless. (Really. I was scanning the loaded modules and adding wrappers to every function found. Never mind why.) I only printed the wrapper line if it was the very top of the stack. > I can't think of any way to do it, You override sys.excepthook to print lines from the traceback selectively. Carl Banks -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Thu, 16 Dec 2010 10:39:34 -0600, Robert Kern wrote:
> On 12/16/10 10:23 AM, Steven D'Aprano wrote:
>> On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote:
>>
>>> Tim Arnold wrote:
"Ethan Furman" wrote in message
news:mailman.4.1292379995.6505.python-l...@python.org...
> kj wrote:
>> The one thing I don't like about this strategy is that the
>> tracebacks of exceptions raised during the execution of __pre_spam
>> include one unwanted stack level (namely, the one corresponding to
>> __pre_spam itself).
>> [...]
>>> A decorator was one of the items kj explicity didn't want. Also,
>>> while it would have a shallower traceback for exceptions raised during
>>> the __pre_spam portion, any exceptions raised during spam itself would
>>> then be one level deeper than desired... while that could be masked by
>>> catching and (re-?)raising the exception in the decorator, Steven had
>>> a very good point about why that is a bad idea -- namely, tracebacks
>>> shouldn't lie about where the error is.
>>
>> True, very true... but many hours later, it suddenly hit me that what
>> KJ was asking for wasn't *necessarily* such a bad idea. My thought is,
>> suppose you have a function spam(x) which raises an exception. If it's
>> a *bug*, then absolutely you need to see exactly where the error
>> occurred, without the traceback being mangled or changed in any way.
>>
>> But what if the exception is deliberate, part of the function's
>> documented behaviour? Then you might want the exception to appear to
>> come from the function spam even if it was actually generated inside
>> some private sub-routine.
>
> Obfuscating the location that an exception gets raised prevents a lot of
> debugging (by inspection or by pdb), even if the exception is
> deliberately raised with an informative error message. Not least, the
> code that decides to raise that exception may be buggy. But even if the
> actual error is outside of the function (e.g. the caller is passing bad
> arguments), you want to at least see what tests the __pre_spam function
> is doing in order to decide to raise that exception.
And how do you think you see that from the traceback? The traceback
prints the line which actually raises the exception (and sometimes not
even that!), which is likely to be a raise statement:
>>> import example
>>> example.func(42)
Traceback (most recent call last):
File "", line 1, in
File "example.py", line 3, in func
raise ValueError('bad value for x')
ValueError: bad value for x
The actual test is:
def func(x):
if x > 10 and x%2 == 0:
raise ValueError('bad value for x')
but you can't get that information from the traceback.
Python's exception system has to handle two different situations: buggy
code, and bad data. It's not even clear whether there is a general
distinction to be made between the two, but even if there's not a general
distinction, there's certainly a distinction which we can *sometimes*
make. If a function contains a bug, we need all the information we can
get, including the exact line that causes the fault. But if the function
deliberately raises an exception due to bad input, we don't need any
information regarding the internals of the function (assuming that the
exception is sufficiently detailed, a big assumption I grant you!). If I
re-wrote the above func() like this:
def func(x):
if !(x <= 10):
if x%2 != 0:
pass
else:
raise ValueError('bad value for x')
return
I would have got the same traceback, except the location of the exception
would have been different (line 6, in a nested if-block). To the caller,
whether I had written the first version of func() or the second is
irrelevant. If I had passed the input validation off to a second
function, that too would be irrelevant.
I don't expect Python to magically know whether an exception is a bug or
not, but there's something to be said for the ability to turn Python
functions into black boxes with their internals invisible, like C
functions already are. If (say) math.atan2(y, x) raises an exception, you
have no way of knowing whether atan2 is a single monolithic function, or
whether it is split into multiple pieces. The location of the exception
is invisible to the caller: all you can see is that atan2 raised an
exception.
> Tracebacks are inherently over-verbose. This is necessarily true because
> no algorithm (or clever programmer) can know all the pieces of
> information that the person debugging may want to know a priori. Most
> customizations of tracebacks *add* more verbosity rather than reduce it.
> Removing one stack level from the traceback barely makes the traceback
> more readable and removes some of the most relevant information.
Right. But I have thought of a clever trick to get the result KJ was
asking for, with the minimum of boilerplate code. Instead of this:
def _pre_spam(args):
if condition(args):
raise S
Re: How to pop the interpreter's stack?
On 12/16/10 10:23 AM, Steven D'Aprano wrote: On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote: Tim Arnold wrote: "Ethan Furman" wrote in message news:mailman.4.1292379995.6505.python-l...@python.org... kj wrote: The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). [...] A decorator was one of the items kj explicity didn't want. Also, while it would have a shallower traceback for exceptions raised during the __pre_spam portion, any exceptions raised during spam itself would then be one level deeper than desired... while that could be masked by catching and (re-?)raising the exception in the decorator, Steven had a very good point about why that is a bad idea -- namely, tracebacks shouldn't lie about where the error is. True, very true... but many hours later, it suddenly hit me that what KJ was asking for wasn't *necessarily* such a bad idea. My thought is, suppose you have a function spam(x) which raises an exception. If it's a *bug*, then absolutely you need to see exactly where the error occurred, without the traceback being mangled or changed in any way. But what if the exception is deliberate, part of the function's documented behaviour? Then you might want the exception to appear to come from the function spam even if it was actually generated inside some private sub-routine. Obfuscating the location that an exception gets raised prevents a lot of debugging (by inspection or by pdb), even if the exception is deliberately raised with an informative error message. Not least, the code that decides to raise that exception may be buggy. But even if the actual error is outside of the function (e.g. the caller is passing bad arguments), you want to at least see what tests the __pre_spam function is doing in order to decide to raise that exception. Tracebacks are inherently over-verbose. This is necessarily true because no algorithm (or clever programmer) can know all the pieces of information that the person debugging may want to know a priori. Most customizations of tracebacks *add* more verbosity rather than reduce it. Removing one stack level from the traceback barely makes the traceback more readable and removes some of the most relevant information. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Thu, 16 Dec 2010 07:29:25 -0800, Ethan Furman wrote: > Tim Arnold wrote: >> "Ethan Furman" wrote in message >> news:mailman.4.1292379995.6505.python-l...@python.org... >>> kj wrote: The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). [...] > A decorator was one of the items kj explicity didn't want. Also, while > it would have a shallower traceback for exceptions raised during the > __pre_spam portion, any exceptions raised during spam itself would then > be one level deeper than desired... while that could be masked by > catching and (re-?)raising the exception in the decorator, Steven had a > very good point about why that is a bad idea -- namely, tracebacks > shouldn't lie about where the error is. True, very true... but many hours later, it suddenly hit me that what KJ was asking for wasn't *necessarily* such a bad idea. My thought is, suppose you have a function spam(x) which raises an exception. If it's a *bug*, then absolutely you need to see exactly where the error occurred, without the traceback being mangled or changed in any way. But what if the exception is deliberate, part of the function's documented behaviour? Then you might want the exception to appear to come from the function spam even if it was actually generated inside some private sub-routine. So, with qualifications, I have half changed my mind. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
Tim Arnold wrote: "Ethan Furman" wrote in message news:mailman.4.1292379995.6505.python-l...@python.org... kj wrote: The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). __pre_spam should be completely invisible and unobtrusive I am unaware of any way to accomplish what you desire. I also think this is one of those things that's not worth fighting -- how often are you going to see such a traceback? When somebody makes a coding mistake? I would say change the name (assuming yours was a real example) to something more meaningful like _spam_arg_verifier and call it good. Alternatively, perhaps you could make a more general arg_verifier that could be used for all such needs, and then your traceback would have: caller spam arg_verifier and that seems useful to me (it is, in fact, how I have mine set up). Hope this helps! ~Ethan~ I thought people would advise using a decorator for this one. Wouldn't that work? thanks, --Tim A decorator was one of the items kj explicity didn't want. Also, while it would have a shallower traceback for exceptions raised during the __pre_spam portion, any exceptions raised during spam itself would then be one level deeper than desired... while that could be masked by catching and (re-?)raising the exception in the decorator, Steven had a very good point about why that is a bad idea -- namely, tracebacks shouldn't lie about where the error is. ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
"Ethan Furman" wrote in message news:mailman.4.1292379995.6505.python-l...@python.org... > kj wrote: >> The one thing I don't like about this strategy is that the tracebacks >> of exceptions raised during the execution of __pre_spam include one >> unwanted stack level (namely, the one corresponding to __pre_spam >> itself). >> >> __pre_spam should be completely invisible and unobtrusive > > I am unaware of any way to accomplish what you desire. I also think this > is one of those things that's not worth fighting -- how often are you > going to see such a traceback? When somebody makes a coding mistake? I > would say change the name (assuming yours was a real example) to something > more meaningful like _spam_arg_verifier and call it good. > > Alternatively, perhaps you could make a more general arg_verifier that > could be used for all such needs, and then your traceback would have: > > caller > > spam > > arg_verifier > > and that seems useful to me (it is, in fact, how I have mine set up). > > Hope this helps! > > ~Ethan~ I thought people would advise using a decorator for this one. Wouldn't that work? thanks, --Tim -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
On Tue, 14 Dec 2010 21:14:35 +, kj wrote: > Consider this code: > > > def spam(*args, **kwargs): > args, kwargs = __pre_spam(*args, **kwargs) > > # args & kwargs are OK: proceed > # ... > > > def __pre_spam(*args, **kwargs): > # validate args & kwargs; > # return canonicalized versions of args & kwargs; # on failure, > raise some *informative* exception # ... > return canonicalized_args, canonicalized_kwargs Double leading underscores don't have any special meaning in the global scope. Save yourself an underscore and call it _pre_spam instead :) In fact, even if spam and __pre_spam are methods, it's probably a good idea to avoid the double-underscore name mangling. It's usually more trouble than it's worth. > I write functions like __pre_spam for one reason only: to remove clutter > from a corresponding spam function that has a particularly complex > argument-validation/canonicalization stage. In effect, spam > "outsources" to __pre_spam the messy business of checking and > conditioning its arguments. A perfectly sensible arrangement. > The one thing I don't like about this strategy is that the tracebacks of > exceptions raised during the execution of __pre_spam include one > unwanted stack level (namely, the one corresponding to __pre_spam > itself). But why is it unwanted? The traceback shows where the error occurs -- it occurs in __pre_spam, not spam, or __post_spam, or spam_caller, or anywhere else. Even if it's possible, having the traceback *lie* about where it occurs is a bad idea which will cause confusion to anyone trying to maintain the software in the future. I can't think of any way to do it, but frankly I haven't thought too hard about it. I'm glad I can't think of any way of doing it, because the thought of having tracebacks lie about where they come from gives me the shivers. Imagine debugging when you've edited the source but are still running the old version, and now the reported line numbers don't match up with the source file -- it would be like that, only worse. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: How to pop the interpreter's stack?
kj wrote: The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). __pre_spam should be completely invisible and unobtrusive I am unaware of any way to accomplish what you desire. I also think this is one of those things that's not worth fighting -- how often are you going to see such a traceback? When somebody makes a coding mistake? I would say change the name (assuming yours was a real example) to something more meaningful like _spam_arg_verifier and call it good. Alternatively, perhaps you could make a more general arg_verifier that could be used for all such needs, and then your traceback would have: caller spam arg_verifier and that seems useful to me (it is, in fact, how I have mine set up). Hope this helps! ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
How to pop the interpreter's stack?
Consider this code: def spam(*args, **kwargs): args, kwargs = __pre_spam(*args, **kwargs) # args & kwargs are OK: proceed # ... def __pre_spam(*args, **kwargs): # validate args & kwargs; # return canonicalized versions of args & kwargs; # on failure, raise some *informative* exception # ... return canonicalized_args, canonicalized_kwargs I write functions like __pre_spam for one reason only: to remove clutter from a corresponding spam function that has a particularly complex argument-validation/canonicalization stage. In effect, spam "outsources" to __pre_spam the messy business of checking and conditioning its arguments. The one thing I don't like about this strategy is that the tracebacks of exceptions raised during the execution of __pre_spam include one unwanted stack level (namely, the one corresponding to __pre_spam itself). __pre_spam should be completely invisible and unobtrusive, as if it had been textually "inlined" into spam prior to the code's interpretation. And I want to achieve this without in any way cluttering spam with try/catches, decorators, and whatnot. (After all, the whole point of introducing __pre_spam is to declutter spam.) It occurs to me, in my innocence (since I don't know the first thing about the Python internals), that one way to achieve this would be to have __pre_spam trap any exceptions (with a try/catch around its entire body), and somehow pop its frame from the interpreter stack before re-raising the exception. (Or some clueful/non-oxymoronic version of this.) How feasible is this? And, if it is quite unfeasible, is there some other way to achieve the same overall design goals described above? TIA! ~kj -- http://mail.python.org/mailman/listinfo/python-list
