Re: exception handling in complex Python programs
Lie a écrit :
On Aug 21, 12:59 am, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
On Wed, 20 Aug 2008 09:23:22 -0700, [EMAIL PROTECTED] wrote:
On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote:
def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:
def do_something(filename):
try:
f = open(filename)
except:
handle exception
No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.
If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.
And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.
Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...)
Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.
BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:
def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.
# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.
y = some_function(x)
# y should now be between -1 and 1.
assert -1 y 1
do_something_with(y)
and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible
Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:
def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...
How could you report the exception any earlier than immediately?
I'm sure different people would have different view on what
immediately means. The LBYL supporters define immediately as
immediately after a definite potential problem is detected (by ifs or
assertion), while the EAFP supporters define immediately as
immediately after a problem arises. No side is right or wrong, both
have weakness and strength, but python-style programs are encouraged
to use EAFP-style exception handling whenever feasible, but with the
spirit of the Zen: Special cases aren't special enough to break the
rules, although practicality beats purity, if LBYL makes things much
easier in that case, and doing EAFP would just convolute the code,
then practicality beats purity.
Hear hear...
--
http://mail.python.org/mailman/listinfo/python-list
RE: exception handling in complex Python programs
Lie wrote: Ah... now I understand what the Zen is talking about when it said: Now is better then never, although never is often better than *right* now. If you don't have all the necessary resources to fix an exception right now, don't try to fix it, instead let it propagate, and allow it to be handled in a level where there is enough information how to fix it. Well, I believe the original intent was more along the lines of adding features, etc to Python, but it's apropos here as well. I think we should change except: into expect:, it would confuse less, would it? It signifies that the program expects so and so kinds of exceptional situations. Whilst the connotations are good, and I think create the right mindset, it ain't gonna happen. Cheers, Tim Delaney -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Bruno Desthuilliers @ Thursday 21 August 2008 22:54: magloca a écrit : Bruno Desthuilliers @ Thursday 21 August 2008 17:31: If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. This is one of the main concerns with which I started this c.l.py thread ! I think it's a pity that we have no way of anticipating and constraining the exceptions thrown by our code, Java's checked exception system has proven to be a total disaster. Could you elaborate on that? I'm not disagreeing with you (or agreeing, for that matter); I'd just really like to know what you mean by a total disaster. One of the most (in)famous Java coding pattern is the empty catchall clause. Read Chris Mellon and Richard Levasseur posts in this thread for more details - they already covered the whole point. Thanks, I missed those. Having read them, I *am* agreeing with you. Personally, I also dislike the predominance in the Java world of only giving type information -- IllegalValueException, ObjectRetrievalFailureException, and whatever. What was the illegal value? What object couldn't be retrieved? How hard is it to use the with-error-message version of the Exception constructor, and include something that might actually be helpful in debugging? m. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit : The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), Defensive programming makes sense in the context of a low-level language like C where errors can lead to dramatic results. In high-level languages like Python, the worse thing that an unhandled exception can cause is an abrupt termination of the process and a nice traceback on screen. ... and leave your datas in inconsistent state. So, what C or any other language could do worse to your application ? In this context, defensive programming is mostly a waste of time - if you can't *correctly* handle the exception where it happens, then doing nothing is the better solution. If I don't buy the argument I actually agree with the conclusion. Each component of a program should try to manage only errors tied to their own logic and let pass others up to the gui logic for rendering errors the good way, persistence logic to rollback unwanted changes, and application logic to continue execution the right way. This is hard to do in C because you have no way to trap an error which happen randomly in the program, ie. a segfault will interrupt the execution anyway. -- _ Maric Michaud -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 21, 12:59 am, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
On Wed, 20 Aug 2008 09:23:22 -0700, [EMAIL PROTECTED] wrote:
On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote:
def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:
def do_something(filename):
try:
f = open(filename)
except:
handle exception
No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.
If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.
And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.
Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...)
Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.
BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:
def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.
# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.
y = some_function(x)
# y should now be between -1 and 1.
assert -1 y 1
do_something_with(y)
and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible
Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:
def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...
How could you report the exception any earlier than immediately?
I'm sure different people would have different view on what
immediately means. The LBYL supporters define immediately as
immediately after a definite potential problem is detected (by ifs or
assertion), while the EAFP supporters define immediately as
immediately after a problem arises. No side is right or wrong, both
have weakness and strength, but python-style programs are encouraged
to use EAFP-style exception handling whenever feasible, but with the
spirit of the Zen: Special cases aren't special enough to break the
rules, although practicality beats purity, if LBYL makes things much
easier in that case, and doing EAFP would just convolute the code,
then practicality beats purity.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Maric Michaud a écrit : Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit : The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), Defensive programming makes sense in the context of a low-level language like C where errors can lead to dramatic results. In high-level languages like Python, the worse thing that an unhandled exception can cause is an abrupt termination of the process and a nice traceback on screen. ... and leave your datas in inconsistent state. Not all applications persist data, so this is an application-specific problem, to be solved at the application level - IOW, there's no one-size-fits-all solution here. Anyway: transactions management is not what I meant when talking about defensive programming. As far as I'm concerned, the only place where the defensive approach really makes sense whatever the language is when dealing with external datas (user inputs etc). So, what C or any other language could do worse to your application ? An error in a C program can do *way* worse than leave an application's data in inconsistent state. See ART for more on this: http://dialspace.dial.pipex.com/prod/dialspace/town/green/gfd34/art/ In this context, defensive programming is mostly a waste of time - if you can't *correctly* handle the exception where it happens, then doing nothing is the better solution. If I don't buy the argument cf above - maybe you buy it after all ?-) I actually agree with the conclusion. Each component of a program should try to manage only errors tied to their own logic and let pass others up to the gui logic for rendering errors the good way, persistence logic to rollback unwanted changes, and application logic to continue execution the right way. This is hard to do in C because you have no way to trap an error which happen randomly in the program, ie. a segfault will interrupt the execution anyway. Indeed. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 21, 2:34 pm, Bruno Desthuilliers bruno. [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] a écrit : On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote: def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... You're running on a multitasking modern machine, right? What happens when some other process deletes filename, or changes its permissions, in the time after you check for access but before you actually open it? This is a good point - if you want to use the correct way of opening files, and you don't want to worry about tracking down exception types, then we can probably agree that the following is the simplest, easiest-to-remember way: def do_something(filename): try: f = open(filename) except: handle exception ... Still not correct IMHO - bare except clauses are BAD. You want: try: f = open(filename) except IOError, e: handle exception Opening files is a special case where EAFP is the only correct solution (AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...) Which defeats the whole point of dynamic typing... and other similar assertions in routines. The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), Defensive programming makes sense in the context of a low-level language like C where errors can lead to dramatic results. In high-level languages like Python, the worse thing that an unhandled exception can cause is an abrupt termination of the process and a nice traceback on screen. In this context, defensive programming is mostly a waste of time - if you can't *correctly* handle the exception where it happens, then doing nothing is the better solution. Ah... now I understand what the Zen is talking about when it said: Now is better then never, although never is often better than *right* now. If you don't have all the necessary resources to fix an exception right now, don't try to fix it, instead let it propagate, and allow it to be handled in a level where there is enough information how to fix it. My 2 cents... Steven D'Aprano says: Exceptions can and often are anticipated. E.g. if you write code that opens a URL, you better anticipate that the server might reject your connection. You better expect to be asked for a cookie, or authentication. If you check for robots.txt, you better expect that it might not exist. That's all normal execution. I think we should change except: into expect:, it would confuse less, would it? It signifies that the program expects so and so kinds of exceptional situations. The try: should also be changed to... perhaps in:, block:, onthiscode:, etc (this paragraph is written with my taste buds on the part of my face below my eye and above my jaw) -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Fri, 22 Aug 2008 06:43:58 -0700 (PDT), Lie wrote: I think we should change except: into expect:, it would confuse less, would it? It signifies that the program expects so and so kinds of exceptional situations. The try: should also be changed to... perhaps in:, block:, onthiscode:, etc (this paragraph is written with my taste buds on the part of my face below my eye and above my jaw) IMO it's not even worth considering. Breaking all the python software that exists and moreover breaking the habits of programmists to gain *nothing* is a waste of time of so many people, that we should just forget it. -- Regards, Wojtek Walczak, http://tosh.pl/gminick/ -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Le Friday 22 August 2008 15:03:21 Bruno Desthuilliers, vous avez écrit : Maric Michaud a écrit : Le Thursday 21 August 2008 09:34:47 Bruno Desthuilliers, vous avez écrit : The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), Defensive programming makes sense in the context of a low-level language like C where errors can lead to dramatic results. In high-level languages like Python, the worse thing that an unhandled exception can cause is an abrupt termination of the process and a nice traceback on screen. ... and leave your datas in inconsistent state. Not all applications persist data, so this is an application-specific problem, to be solved at the application level - IOW, there's no one-size-fits-all solution here. ... or lose open connection, reset sessions, etc.. it doesn't really matter what is lost when a program crash, if it can be restarted without changes it is hardly what I'd qualify a dramatic result. It doesn't depend on the language what implications of an unhandled error are, It is always application-specific. Anyway: transactions management is not what I meant when talking about defensive programming. As far as I'm concerned, the only place where the defensive approach really makes sense whatever the language is when dealing with external datas (user inputs etc). I agree, this is my whole point, whatever the language is. So, what C or any other language could do worse to your application ? An error in a C program can do *way* worse than leave an application's data in inconsistent state. See ART for more on this: http://dialspace.dial.pipex.com/prod/dialspace/town/green/gfd34/art/ I didn't read dramatic results in that sense, but with the meaning of a result that the program itself cannot handle. If the whole system crash due to an unhandled error in a program, once missile's dropped it doesn't really matter in which language it was written. Reliability of a system, as high-availability of an application, is mostly a problem beyond the scope of application level error checking. In this context, defensive programming is mostly a waste of time - if you can't *correctly* handle the exception where it happens, then doing nothing is the better solution. If I don't buy the argument cf above - maybe you buy it after all ?-) I actually agree with the conclusion. Each component of a program should try to manage only errors tied to their own logic and let pass others up to the gui logic for rendering errors the good way, persistence logic to rollback unwanted changes, and application logic to continue execution the right way. This is hard to do in C because you have no way to trap an error which happen randomly in the program, ie. a segfault will interrupt the execution anyway. Indeed. -- http://mail.python.org/mailman/listinfo/python-list -- _ Maric Michaud -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
En Thu, 21 Aug 2008 14:48:45 -0300, magloca [EMAIL PROTECTED] escribió: Bruno Desthuilliers @ Thursday 21 August 2008 17:31: Java's checked exception system has proven to be a total disaster. Could you elaborate on that? I'm not disagreeing with you (or agreeing, for that matter); I'd just really like to know what you mean by a total disaster. Please allow me to share a few links: http://www.mindview.net/Etc/Discussions/CheckedExceptions http://radio.weblogs.com/0122027/stories/2003/04/01/JavasCheckedExceptionsWereAMistake.html and see also a message from Richard Levasseur today in this very thread explaining why checked exceptions are just an annoyance that buys you nothing. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
One common-place thing I've noticed in a lot of python code is that every package or module has a main Error class, and all sub-types inherit from that class. So you just catch mylib.Error, and you're going to catch all the exceptions that package generates directly. There seems to be a lot of concern about what exceptions a functions might throw, and what to do when it happens. The way I see it, there are only two types of exceptions: those you can recover from, and those you can't. The only ones -really- worth catching are those you can recover from. There's a middle-ground type of 'cleanup', to free any resources, but those generally go in a finally block, not an except block. For the ones you can't handle, it doesn't matter if you catch them or not. If you catch them, what do you do? Log an error message, then rethrow it. You're still throwing an exception, so you haven't really gained anything. You might repackage it and put additional information in the exception so you can do something at a higher level. What that is, I don't know. I don't think I've ever passed information up in an exception that was of use to the program, and I'm hard pressed to think of any information you could provide that could - fix- the problem. If you can derive recoverable information, then why rethrow? Thats pretty much a recoverable situation, so there's not need to rethrow. In java, there are checked exceptions, which are nice - they tell you what a function might throw, so you know what to catch. I don't think this improves the quality of anything, though. It just annoys the developer. What they end up doing is writing an application-specific exception class, and -everything- gets rethrown as that, and everything begins to declare it throws AppError. Whats worse is that you have heavily, heavily repackaged exceptions: SQLError - AppError - MessageError - AppError - MessageError (yes, i've seen this before). That is almost completely useless. Sure, you could dig down to SQLError, but how do you know to do that? If you knew how far you should dig down, that means you know what the problem was, in which case, you could have prevented it or aborted early. Whats worse, if you have all these re-packaging catch blocks and they just log something generic, which becomes common with all the catching going on. Couldn't do foo!, Bar operation failed!, or Couldn't fetch filters from database (why? We don't know, its catching an AppError instead of something more specific), and then they rethrow the exception. While trying to debug something not-during development, those messages are completely useless, in fact, they're more than useless. They're just more cruft to sift through in a log file. Additionally, most root causes of an error are going to originate where the input comes from the user. Handling anything at levels deeper than that isn't going to gain you much. PrepareQuery threw an error because of a missing field? Thats great. Where'd it come from? There are 100 calls to PrepareQuery. There's only a few calls to ReadUserInput(), and a single informative log message of Query failed, unknown field; fields=a, b, c is much better than 100 lines of traceback smattered with Unknown field and Unable to prepare query. Finally, they give a false sense of security. I'm catching everything it could throw, so everything will be ok if an exception is thrown! I guess thats true. I guess. The only real advantage is the whole program won't crash with the ever-helpful, single line of Segmentation fault. An improvement, but it doesn't prevent anything. In a complex system, an error can occur in any function at anytime. Adding 'throws' to a method definition doesn't change that. I guess my point is: everything Chris Mellon said was spot on. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 19, 7:19 pm, eliben [EMAIL PROTECTED] wrote: Python provides a quite good and feature-complete exception handling snip Thanks for the interesting discussion. Armed by the new information and few online sources, I blogged a summary for myself on the topic of robust exception handling in Python: http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/ -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
[EMAIL PROTECTED] a écrit : On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote: def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... You're running on a multitasking modern machine, right? What happens when some other process deletes filename, or changes its permissions, in the time after you check for access but before you actually open it? This is a good point - if you want to use the correct way of opening files, and you don't want to worry about tracking down exception types, then we can probably agree that the following is the simplest, easiest-to-remember way: def do_something(filename): try: f = open(filename) except: handle exception ... Still not correct IMHO - bare except clauses are BAD. You want: try: f = open(filename) except IOError, e: handle exception Opening files is a special case where EAFP is the only correct solution (AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...) Which defeats the whole point of dynamic typing... and other similar assertions in routines. The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), Defensive programming makes sense in the context of a low-level language like C where errors can lead to dramatic results. In high-level languages like Python, the worse thing that an unhandled exception can cause is an abrupt termination of the process and a nice traceback on screen. In this context, defensive programming is mostly a waste of time - if you can't *correctly* handle the exception where it happens, then doing nothing is the better solution. My 2 cents... -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
[EMAIL PROTECTED] a écrit : (snip) Here is an example: a simple query tool for a tiny mock SQL relational database. With a method (called select_all) you can perform the equivalent of a select query on the database. The contents of the query are specified with triples of the form [field, comparison_operator, value] for instance ['name', operator.equals, cmd_name]. You can also specify an order by field which is None by default. In the code written, there is an assertion that the order-by field is either None or a valid field name (we can't order by a nonexistent field!). If the assertion isn't there, then I will get an error on this line: key_extractor = KeyExtractor(q_column_names.index(order_by_column)) In this particular case, I will get a ValueError (what does ValueError mean again? And what is this KeyExtractor?) since the index method will fail. I wrote the tiny relational database a long time ago, and I really don't want to put pressure on my mental cache by thinking about the internal logic of this chunk of code. After scratching my head for a while, I'll probably figure it out. Now imagine that you instead get an error on this line: assert order_by_column in q_column_names Now the programming error slaps me with a fish and yells STOP! YOU CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!. As far as I'm concerned, this is a case where I would explicitely raise an exception (either a ValueError with an explicit message or a library-defined exception type). -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Thu, 21 Aug 2008 00:34:21 -0700, eliben wrote:
On Aug 19, 7:19 pm, eliben [EMAIL PROTECTED] wrote:
Python provides a quite good and feature-complete exception handling
snip
Thanks for the interesting discussion. Armed by the new information and
few online sources, I blogged a summary for myself on the topic of
robust exception handling in Python:
http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/
Just a few random points. You say:
Exceptions are better than returning error status codes. Some languages
(like Python) leave you with no choice as the whole language core and
standard libraries throw exceptions.
Of course you have a choice. Your function can return anything you want:
def mysqrt(x):
try:
return math.sqrt(x)
except ValueError:
return Code 37
I've written functions that return an object on success and None if the
function failed. In the context of what I was doing, that made more sense
than raising an exception.
Furthermore, the str.find() method returns -1 for not found instead of
raising an exception. There are probably other examples as well.
You also wrote:
Exceptions exist for exceptional situations: unanticipated events that
are not a part of normal execution.
Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.
When a programmer calls str.find('substring') he doesn’t expect an
exception to be thrown if the substring isn’t found.
But if he called str.index() then he does expect an exception to be
thrown, just like for list.index() and dict[key] can raise exceptions.
They are neither bugs nor unexpected.
This is what he called find for. A better approach is to return a
special value like None or -1.
Sometimes, maybe. But usually not, because that just complicates the
calling code. You end up writing code that repeatedly checks that the
result isn't a special value before doing anything.
Often a better tactic is to write your code assuming that the result is
the unexceptional case, and then wrap it in a try...except block to catch
the exceptional cases.
When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not.
I strongly disagree. try...except is like break or continue. Yes, it
breaks the linear flow of control, but not in a wild, dangerous way like
goto.
It is possible to write bad code with exceptions, but you can write bad
code with anything.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
eliben a écrit : On Aug 19, 7:19 pm, eliben [EMAIL PROTECTED] wrote: Python provides a quite good and feature-complete exception handling snip Thanks for the interesting discussion. Armed by the new information and few online sources, I blogged a summary for myself on the topic of robust exception handling in Python: http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/ A couple comments (mostly python-specific, so I post them here): When used for flow-control, exceptions are like goto. There might be a few esoteric cases in which they’re appropriate, but 99.99% of the time they are not. Python itself uses exceptions for flow control in iterators. For some exceptions, like programming errors (e.g. IndexError, TypeError, NameError etc.) exceptions are best left to the programmer / user, because “handling” them will just hide real bugs. Depends on the context. There are cases where you expect these kind of errors - like when dealing with program inputs, inspecting objects etc. As a QD example: while True: raw_num = raw_input(enter a number) try: num = float(raw_num) except TypeError, ValueError: print sorry, '%s' is not a valid number % raw_num else: # ok break This is also the reason why you should be extremely careful with except: clauses that catch everything. These will not only catch the exceptions you intended, but all of them. And remember that SysExit and KeyboardInterrupt *are* exceptions too... Document the exceptions thrown by your code If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. My 2 cents. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/
Just a few random points. You say:
Exceptions are better than returning error status codes. Some languages
(like Python) leave you with no choice as the whole language core and
standard libraries throw exceptions.
Of course you have a choice. Your function can return anything you want:
Of course. I didn't mean that the language prohibits returning error
codes, just that you can't use it without employing exception
handling. I've fixed the wording to make it clearer.
You also wrote:
Exceptions exist for exceptional situations: unanticipated events that
are not a part of normal execution.
Exceptions can and often are anticipated. E.g. if you write code that
opens a URL, you better anticipate that the server might reject your
connection. You better expect to be asked for a cookie, or
authentication. If you check for robots.txt, you better expect that it
might not exist. That's all normal execution.
This is a point I'm not 100% in accord with. I still think that
exceptions are for exceptional situations. I've removed the word
unanticipated though, because it probably has no place in that
sentence. However, I think that if one of your valid execution paths
is w/o robots.txt, you should not use an exception to check whether
it's there. This indeed uses the bad side of exceptions, splitting
the exetution to two paths.
Check if robots.txt is there. If it is, open it. If you can't open it,
*that* is an exception, but if it's just not there, well it's part of
your application logic. I believe this isn't against EAFP.
I'm not sure I'm making the distinction clear here, it's a fine point.
When a programmer calls str.find('substring') he doesn’t expect an
exception to be thrown if the substring isn’t found.
But if he called str.index() then he does expect an exception to be
thrown, just like for list.index() and dict[key] can raise exceptions.
They are neither bugs nor unexpected.
But why are there two versions that are the same except for the
behavior in case it wasn't found ? My wishful imagination is precisely
because of the reasons I've named. If you *know* it's there,
use .index() - then, if it fails, it's an exception, but if a part of
your logic is finding an item that might be missing, use a special
value because you want to keep the logic in a single path.
When used for flow-control, exceptions are like goto. There might be a
few esoteric cases in which they’re appropriate, but 99.99% of the time
they are not.
I strongly disagree. try...except is like break or continue. Yes, it
breaks the linear flow of control, but not in a wild, dangerous way like
goto.
try...except can 'exit' to several 'catch points', unlike break/
continue. Furthermore, try...except can bring execution to another
hierarchy level if it's not caught where it's thrown, so it's much
more like goto in these senses. To find where the execution may go
you'll find yourself searhching for the exception name over your
source files, looking for the exception class name in some except
clause. Sounds like looking for a goto label.
P.S. Thanks a lot for taking the time to comment
Eli
--
http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 21, 12:40 pm, Bruno Desthuilliers bruno. [EMAIL PROTECTED] wrote: eliben a écrit : On Aug 19, 7:19 pm, eliben [EMAIL PROTECTED] wrote: Python provides a quite good and feature-complete exception handling snip Thanks for the interesting discussion. Armed by the new information and few online sources, I blogged a summary for myself on the topic of robust exception handling in Python: http://eli.thegreenplace.net/2008/08/21/robust-exception-handling/ A couple comments (mostly python-specific, so I post them here): Thanks for the feedback. My comments below: When used for flow-control, exceptions are like goto. There might be a few esoteric cases in which they’re appropriate, but 99.99% of the time they are not. Python itself uses exceptions for flow control in iterators. Yep, I'm aware of StopIteration, but I'm not sure whether it's a good or a bad feature. I'm a bit wary of the programming style it might encourage in inexperienced programmers. When this behavior is hidden inside the implementation of 'for', fair enough. But when you have to catch exceptions just to walk over some iterable explicitly, I'm not sure the designers of this Python feature made the correct choices here. For some exceptions, like programming errors (e.g. IndexError, TypeError, NameError etc.) exceptions are best left to the programmer / user, because “handling” them will just hide real bugs. Depends on the context. There are cases where you expect these kind of errors - like when dealing with program inputs, inspecting objects etc. As a QD example: while True: raw_num = raw_input(enter a number) try: num = float(raw_num) except TypeError, ValueError: print sorry, '%s' is not a valid number % raw_num else: # ok break I agree. This is also the reason why you should be extremely careful with except: clauses that catch everything. These will not only catch the exceptions you intended, but all of them. And remember that SysExit and KeyboardInterrupt *are* exceptions too... Document the exceptions thrown by your code If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. This is one of the main concerns with which I started this c.l.py thread ! I think it's a pity that we have no way of anticipating and constraining the exceptions thrown by our code, and that we should strive to make it more explicit. The function accepting a file is a case in point. You know what you do with this file, so why can't you know what exceptions might be thrown ? If you're trying to open it, IOError (and OSError ?), etc. Besides, as I noted in the article, perhaps you want to hide some of inner-level exceptions in your own, to keep encapsulation. Eli -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
eliben a écrit : On Aug 21, 12:40 pm, Bruno Desthuilliers bruno. [EMAIL PROTECTED] wrote: eliben a écrit : On Aug 19, 7:19 pm, eliben [EMAIL PROTECTED] wrote: (snip) Document the exceptions thrown by your code If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. This is one of the main concerns with which I started this c.l.py thread ! I think it's a pity that we have no way of anticipating and constraining the exceptions thrown by our code, Java's checked exception system has proven to be a total disaster. and that we should strive to make it more explicit. The function accepting a file is a case in point. You know what you do with this file, so why can't you know what exceptions might be thrown ? Reread more carefully. I wrote a *file-like* object, not a file. This is the whole point of duck typing. Given that any file-like object will work ok with my function, I have *no way* to know what exceptions this object may raise. If you're trying to open it, Trying to open a file *object* ? heck, it's supposed to be already opened at this stage. And yes, it's a pretty common pattern in Python (which is why I choose this example). -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Bruno Desthuilliers @ Thursday 21 August 2008 17:31: If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. This is one of the main concerns with which I started this c.l.py thread ! I think it's a pity that we have no way of anticipating and constraining the exceptions thrown by our code, Java's checked exception system has proven to be a total disaster. Could you elaborate on that? I'm not disagreeing with you (or agreeing, for that matter); I'd just really like to know what you mean by a total disaster. m. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
magloca a écrit : Bruno Desthuilliers @ Thursday 21 August 2008 17:31: If you mean the exceptions *explicitely raised* by your code, then I agree. But with any generic enough code, documenting any possible exception that could be raised by lower layers, objects passed in as arguments etc is just plain impossible. Like, if you have a function that takes a file-like object as arg, you just cannot know in advance what exceptions this object might raise. This is one of the main concerns with which I started this c.l.py thread ! I think it's a pity that we have no way of anticipating and constraining the exceptions thrown by our code, Java's checked exception system has proven to be a total disaster. Could you elaborate on that? I'm not disagreeing with you (or agreeing, for that matter); I'd just really like to know what you mean by a total disaster. One of the most (in)famous Java coding pattern is the empty catchall clause. Read Chris Mellon and Richard Levasseur posts in this thread for more details - they already covered the whole point. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 20, 10:13 pm, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: It might not be enjoyable to have a sarcastic remark directed your way, but it isn't a personal attack. Just because a comment is about something you do doesn't make it a personal attack. Personal attacks are about who you are rather than what you do. If you type in Personal Attack in Wikipedia (not an authoritative source, I know) it takes you to the page on ad hominem arguments. There you can find the following example of the fallacious ad hominem argument: Person A makes claim X There is something objectionable about Person A Therefore claim X is false It is, ultimately, a matter of opinion, but going out of one's way to make it hard for other programmers sounds objectionable to me. I mean, I wouldn't want to work with anyone like that! There's an apparent contradiction in your argument. You seem to be arguing against EAFP and in favour of LBYL, but now you're suggesting that you don't use type-checking. As near as I can tell, you don't do type-checking, you don't do duck typing, you don't like catching exceptions. So what do you actually do to deal with invalid data? Here is an example from a Django web app: when there is a bug, a generic Exception is thrown and Django catches it and reports a beautifully formatted stack trace. When something must be reported to the user, a MyAppException is thrown (not the real name). The HTTP request handler for the application is wrapped in a single try:... except MyAppException: the big idea is that there should be a maximum of two try/except blocks on the stack at any particular point in time [1]: at a high level (already mentioned) and for wrapping primitive execute operations against Rpy and MySQLdb. In practice, this doesn't always happen - there is one place where an EAFP-style construct is used (the operation in this case is to generate some HTML and cache it based on some source XML, but the source may have syntax errors, so if the HTML can't be generated, then cleanup is performed and an error message returned). So to summarize: try/except blocks at boundaries between system components: good try/except blocks within a single component: slightly concerning I think I may have overstated the case against EAFP. There are certainly cases where EAFP makes a lot of sense; I would object to portraying EAFP as an alternative to defensive programming. [Side note: defensive programming serves much the same purpose in Python as it does in C, but I agree that in C there is extra motivation such as avoiding buffer overruns. I think of defensive programming simply as taking proactive steps to reduce the expected time to debug a program if a programming error should arise.] By the way, if you're worried that isinstance() is too long to type, you can do this: isin = isinstance isin(123, int) Actually I'm holding out for type objects to grow __lt__ and __le__ methods so you can do something like from abstract_base_classes import sequence if type(my_obj) = sequence: ... This is borrowed from the = notation for subgroups in math (there are probably other cases too). I don't use abstract base classes, so I don't even know if this is right, but hopefully you get the idea. No no no, exceptions are not necessarily bugs!!! A bug is an exceptional circumstance, but not all exceptional circumstances are bugs. I tend to agree, but I have found that thinking about these issues makes me question the wisdom of Python's built-ins throwing exceptions in non-exceptional circumstances (for instance you try to open a file that doesn't exist - IMHO this is about as exceptional as trying a no- wait acquire on a busy lock, in other words it isn't exceptional at all). As long as we are in fantasy realm, one could argue that open() should return a pair like this: f, open_ok = open(...) where open_ok is a status object whose __nonzero__ (in 3.0 __bool__ is used) is true on success and false on an error, and also has an error code and error message field. The idea is from Django's get_or_create method in the db API. [1] That I have to think about. I don't particularly care about try/ except blocks in Django's, Rpy's, or MySQLdb's activation records. David -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Here is an example from a Django web app: when there is a bug, a generic Exception is thrown and Django catches it and reports a beautifully formatted stack trace. When something must be reported to the user, a MyAppException is thrown (not the real name). The HTTP request handler for the application is wrapped in a single try:... except MyAppException: the big idea is that there should be a maximum of two try/except blocks on the stack at any particular point in time [1]: at a high level (already mentioned) and for wrapping primitive execute operations against Rpy and MySQLdb. This actually makes lots of sense as is exactly what Ned Batchelder wrote here: http://nedbatchelder.com/text/exceptions-in-the-rainforest.html * A-layer generates exceptions, * B-layer can often ignore the whole issue, and * C-layer decides what to do It's worth a read. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 19, 7:34 pm, Chris Mellon [EMAIL PROTECTED] wrote:
On Tue, Aug 19, 2008 at 12:19 PM, eliben [EMAIL PROTECTED] wrote:
Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.
This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.
But by 'handling', do you also mean rethrow with better
information ?
I feel there's an inherent clash between two 'good practices' in
exception handling:
1) Using EAFP over LBYL
2) Hiding implementation details
Consider this code, which I wrote just yesterday:
elif type in ('LinearStartAddr', 'SegmentStartAddr'):
if len(data) != 4:
line_error('expecting a 4-byte data field for this record type,
got %s' % len(data))
self.data.start_address = unpack('L', data)
This is part of a method in a class that parses a data file. I've
ended up using LBYL here, to hide an implementation detail. I could've
let the Exception from unpack propagate, but that doesn't make much
sense with hiding implementation. So I'm throwing a more useful
exception myself.
Was wrapping the call to unpack with try/except that throws my
exception a better idea, in your opinion ? Because that makes the code
somewhat more convoluted.
Eli
--
http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote: You want to look up Easier to Ask Forgivness than Permission (EAFP) which is touted as the canonical error-handling paradigm for Python. Any (semi)complete guides on this canonical paradigm online ? I've only found some references in maillist discussions. There's the glossary in the documentation: http://docs.python.org/tut/node18.html Look under 'duck-typing', 'EAFP', and 'LBYL'. Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote: between file() and open() in Python 2 and 3, a NameError is thrown with open() in Python 3 and an IOError is thrown in the other three cases bashes head against keyboard. I'm curious about the claim that open() will raise NameError in Python3. I find it hard to credit that claim, but if it is correct, what's the justification for that? This is *exactly* my concern with Python exceptions. You just never know what can be thrown at you. It's true that documentation of exceptions is relatively weak in Python. And some functions can raise a bewildering array of exceptions. See for example this thread where somebody notes that urllib2.urlopen() can raise any of six different exceptions: http://mail.python.org/pipermail/baypiggies/2008-April/003187.html And I've had it raise socket.error, which makes seven. And the documentation only mentions one of those exceptions. However, as Gregory Smith describes, some of those seven exceptions are subclasses of others, so it is possible to reduce it down to three cases -- and arguably one of those cases (ValueError) is a bug that needs fixing, not an exception that needs catching. That's probably as bad as it gets in Python, at least for the standard library. Most functions don't raise arbitrary exceptions for sensible data, and if you pass non-sensible data then you should treat the exception as a bug in your code and fix it. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote: def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... You're running on a multitasking modern machine, right? What happens when some other process deletes filename, or changes its permissions, in the time after you check for access but before you actually open it? This is a good point - if you want to use the correct way of opening files, and you don't want to worry about tracking down exception types, then we can probably agree that the following is the simplest, easiest-to-remember way: def do_something(filename): try: f = open(filename) except: handle exception ... Opening files is a special case where EAFP is the only correct solution (AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...) and other similar assertions in routines. The point is that EAFP conflicts with the interest of reporting errors as soon as possible (on which much has been written see, for instance Ch. 8 - Defensive Programming in Code Complete), but LBYL conflicts with correctness when objects can be shared. Also, look at the man page for access. I have found at least two (one on my Linux box, another online) that essentially say never use it. I completely forgot about this in my last post... David -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
[EMAIL PROTECTED] a écrit : On Aug 19, 10:19 am, eliben [EMAIL PROTECTED] wrote: P.S. There's a common case where a method is passed a filename, to do something with a file (say, read data). Should the method catch the errors possibly thrown by open(), or leave it to the caller ? You want to look up Easier to Ask Forgivness than Permission (EAFP) which is touted as the canonical error-handling paradigm for Python. This would give rise to the following function: def do_something(filename): try: f = open(filename) except IOError: return err(File %s not found % filename) ... where err is a function that generates an error object that your application understands. Sorry but that's IMHO totally broken. This error object is useless (heck, we *do* have exceptions, don't we ???), *returning* it ruins the whole point of structured exception handling and take us back to infamous C error code checking (which are almost never checked), and - icing on the cake - the error message is very possibly wrong and misleading (IOError dont necessarily mean 'file not found'). This kind of exception handling manages to be worse than no exception handling at all. I personally think this is sloppy because you have to couple the exception type with the function --- between file() and open() in Python 2 and 3, a NameError is thrown with open() in Python 3 ??? I suspect this has nothing to do with any error happening while opening the file. NameError means the name doesn't exists in the current namespace nor it's enclosing namespaces. Could it be possible that open() has been removed from Py3k ? and an IOError is thrown in the other three cases bashes head against keyboard. The alternative is def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... This gets even worse. race condition... Things can change between the call to os.access and the call to open. Well-known antipattern. or, (and this last one I actually used for a web application) def do_something(filename): if not os.access(filename,os.R_OK): raise MyApplicationsExceptionType(File not found...) You loose all the useful information you'd have from an IOError raised by a direct call to open... f = open(filename) ... ... IOError that you're still likely to see happen anyway. The last one has the advantage that you can write a request handler like this def handle_http_request(...): func = specific_handler_func(...) try: response = func(...) return response except MyApplicationsExceptionType as exc: #3.0 syntax return error_response(exc,...) If you want to raise a different exception type - which can indeed be a sensible thing to do, depending on the context -, you can do it safely and keep accurate informations: def do_something(filename): try: f = open(filename) except IOError, e raise MyApplicationsExceptionType(e.msg) # could even pass whole traceback etc # etc... Exceptions you don't expect (i.e. bugs) An exception you don't expect is not necessarily a bug. Try unplugging your lan cable while writing to a socket connected to another computer... (snip) If you are writing a library (for instance using a file for persistent storage), then the answer to your question is don't catch the exception. Clients will expect the usual exception to be thrown when a bad file name is passed. Indeed. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
eliben a écrit : This is *exactly* my concern with Python exceptions. You just never know what can be thrown at you. This rarely happen to be a problem in real life. At least not in mine. Exception that can be expected (ie : IOError when dealing with files) are usually obvious and more or less documented - or easy to figure out (like TypeError and ValueError when trying to build an int from an arbitrary object, KeyError when working with dicts, AttributeError when inspecting an object, etc) from concrete use. IOW, it's usually easy to know which exceptions you're able to deal with at the lower level. Any other exception is either a programming error - which needs to be fixed, not hidden - or nothing you can deal with at the lower level - in which case just let it propagate until some other layer above deal with it (eventually just logging the error, displaying a user-friendly message, and crashing if nothing else is possible). def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... But does os.access cover absolutely all the errors that can happen during open() ? What guarantees it, and how can I know without you teaching me, just from the docs ? The above code is a perfect antipattern. It's useless (if you can't access the file, you'll get an IOError when trying to open it anyway), it's wrong (things may change between the call to os.access and the call to open), and it defeats the whole point of exception handling (by returning some kind of error object instead of using exception handling). -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 20, 12:47 am, Fredrik Lundh [EMAIL PROTECTED] wrote: Rafe wrote: Again, this is probably too simple to help, but the only way to ignore certain types of exceptions, as far as I know, is to catch them and pass. e.g. this ignores type errors... try: somethingBad() except TypeError, err: pass except Exception, err: raise TypeError(err) so what kind of code are you writing where *type errors* are not considered programming errors? (catching them and proceeding is one thing, but catching them and ignoring them?) I'd be really worried if I found that in a piece of source code I had to maintain. /F I'm not it was just the first exception that came to mind... It is pretty rare that I would pass an exception in fact. Maybe as a last- resort test in some cases. - Rafe -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Wed, 20 Aug 2008 09:23:22 -0700, [EMAIL PROTECTED] wrote:
On Aug 19, 4:12 pm, Steven D'Aprano [EMAIL PROTECTED]
cybersource.com.au wrote:
On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote:
def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:
def do_something(filename):
try:
f = open(filename)
except:
handle exception
No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.
If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.
And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.
Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style assert isinstance(...)
Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.
BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:
def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.
# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.
y = some_function(x)
# y should now be between -1 and 1.
assert -1 y 1
do_something_with(y)
and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible
Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:
def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...
How could you report the exception any earlier than immediately?
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Wed, 20 Aug 2008 18:37:02 +0200, Bruno Desthuilliers wrote: I personally think this is sloppy because you have to couple the exception type with the function --- between file() and open() in Python 2 and 3, a NameError is thrown with open() in Python 3 ??? I suspect this has nothing to do with any error happening while opening the file. NameError means the name doesn't exists in the current namespace nor it's enclosing namespaces. Could it be possible that open() has been removed from Py3k ? No it's moved/changed but there's still a name for it in the builtin namespace. `file` on the other hand is gone: Python 3.0b2 (r30b2:65080, Aug 20 2008, 20:41:17) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2 Type help, copyright, credits or license for more information. open class 'io.OpenWrapper' file Traceback (most recent call last): File stdin, line 1, in module NameError: name 'file' is not defined Ciao, Marc 'BlackJack' Rintsch -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 20, 10:59 am, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: Oh goodie. Another programmer who goes out of his way to make it hard for other programmers, by destroying duck-typing. Remember kids: personal attacks are cruise control for cool. So this was a simplification - most of the asserts I've written don't actually use isinstance, partly because typing isinstance takes too long. The point is to create a barricade so that when something goes wrong, you get an assertion error against the code you wrote, not an exception against doing something like print(blah blah %s % message) where message turns out to be None. This is simply a way to make debugging a more pleasant experience (quite valuable IMHO since debugging is inherently difficult and can be quite aggravating). Here is a sampling: assert statelt.tag == 'stat' assert len(path) 0 and path[0] == '/' assert self.__expr != None So here asserts are used to made distinctions that are more fine- grained than type. and other similar assertions in routines. The point is that EAFP conflicts with the interest of reporting errors as soon as possible Not necessarily. Tell me how this conflicts with reporting errors as soon as possible: def do_something(filename): try: f = open(filename) except IOError, e: report_exception(e) # use a GUI, log to a file, whatever... How could you report the exception any earlier than immediately? Here is an example: a simple query tool for a tiny mock SQL relational database. With a method (called select_all) you can perform the equivalent of a select query on the database. The contents of the query are specified with triples of the form [field, comparison_operator, value] for instance ['name', operator.equals, cmd_name]. You can also specify an order by field which is None by default. In the code written, there is an assertion that the order-by field is either None or a valid field name (we can't order by a nonexistent field!). If the assertion isn't there, then I will get an error on this line: key_extractor = KeyExtractor(q_column_names.index(order_by_column)) In this particular case, I will get a ValueError (what does ValueError mean again? And what is this KeyExtractor?) since the index method will fail. I wrote the tiny relational database a long time ago, and I really don't want to put pressure on my mental cache by thinking about the internal logic of this chunk of code. After scratching my head for a while, I'll probably figure it out. Now imagine that you instead get an error on this line: assert order_by_column in q_column_names Now the programming error slaps me with a fish and yells STOP! YOU CAN'T ORDER BY A FIELD THAT DOESN'T EXIST!!!. It will take about 2 seconds to figure out what went wrong. I just saved a minute figuring out what the problem is. Multiply that by ten, and you've just eliminated work in a potentially laborious debugging session. If you look at the history of the EAFP concept in Python, then you see that it comes from Alex Martelli's Python in a Nutshell around pages 113-114. I don't think the code examples make the case for EAFP very well (not that I know what EAFP is in the first place, given that it is barely explained. I interpret it as wrap questionable stuff in try/ except blocks), and in any case there is practically no support for using EAFP as the dominant error-handling paradigm. If you look at Code Complete, then you'll see the opposite suggestion, namely that exceptions should only be used for truly exceptional circumstances (i.e. bugs). McConnell argues that try/except is an inherently complex control structure so it should be used sparingly (just like balsamic vinegar!). I happen to think that truth lies between these extremes, but I'd err on using fewer try/except structures, not more. Using several try/except blocks across multiple activation records sounds like unreadable code to me. If shared objects are used pervasively, then I would predict that EAFP will not provide adequate abstractions to control program complexity (see http://research.microsoft.com/Users/simonpj/papers/stm/stm.pdf and the Wikipedia article on software transactional memory). These days you can switch to Stackless and use tasklets and atomic operations (See http://www.stackless.com/wiki/Tasklets). There is a debate between EAFP and LBYL here: http://mail.python.org/pipermail/python-list/2003-May/205182.html. Martelli's posts in support of EAFP are heavily skewed towards a multithreaded scenario and avoiding race conditions. IMHO, letting locking and race condition concerns dictate your error-handling paradigm is a case of the tail wagging the dog, especially when there are alternatives to this particular tar pit: pipes or a shared nothing architecture. David -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Wed, 20 Aug 2008 17:49:14 -0700, [EMAIL PROTECTED] wrote: On Aug 20, 10:59 am, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: Oh goodie. Another programmer who goes out of his way to make it hard for other programmers, by destroying duck-typing. Remember kids: personal attacks are cruise control for cool. It might not be enjoyable to have a sarcastic remark directed your way, but it isn't a personal attack. Just because a comment is about something you do doesn't make it a personal attack. Personal attacks are about who you are rather than what you do. So this was a simplification - most of the asserts I've written don't actually use isinstance, partly because typing isinstance takes too long. You say that you liberally sprinkle isinstance() checks through your code, then you say that you don't. That confuses me. There's an apparent contradiction in your argument. You seem to be arguing against EAFP and in favour of LBYL, but now you're suggesting that you don't use type-checking. As near as I can tell, you don't do type-checking, you don't do duck typing, you don't like catching exceptions. So what do you actually do to deal with invalid data? By the way, if you're worried that isinstance() is too long to type, you can do this: isin = isinstance isin(123, int) The point is to create a barricade so that when something goes wrong, you get an assertion error against the code you wrote ... assert statelt.tag == 'stat' assert len(path) 0 and path[0] == '/' assert self.__expr != None All of those examples seem to be reasonably straight forward tests of program logic, which would make them good cases for assertions. Assuming that statelt, path and self.__expr are internally generated and not user- supplied arguments. I'll note that testing for (non-)equality against None is possibly a mistake. It won't catch the case where self.__expr is an object that, for some reason, compares equal to None but isn't None. If that's your intention, then it's fine, but given that you don't seem to be a big fan of duck typing I guess you'd probably be better off with: assert self.__expr is not None It runs faster too, although a micro-optimization of that tiny size isn't in itself sufficient reason for preferring is not over !=. The real reason is to avoid accidental matches. So here asserts are used to made distinctions that are more fine- grained than type. The problem with your earlier example isn't that isinstance() is too coarse-grained (although I try to avoid it as much as possible), but that assert is not meant for argument testing. In principle, the end user should never see an AssertionError. As I said earlier, assert is for testing program logic. and other similar assertions in routines. The point is that EAFP conflicts with the interest of reporting errors as soon as possible Not necessarily. Tell me how this conflicts with reporting errors as soon as possible: def do_something(filename): try: f = open(filename) except IOError, e: report_exception(e) # use a GUI, log to a file, whatever... How could you report the exception any earlier than immediately? Here is an example: I gather by your lack of answer to my question that you now accept that exceptions don't necessarily delay reporting errors as early as possible. a simple query tool for a tiny mock SQL relational database. With a method (called select_all) you can perform the equivalent of a select query on the database. The contents of the query are specified with triples of the form [field, comparison_operator, value] for instance ['name', operator.equals, cmd_name]. You can also specify an order by field which is None by default. In the code written, there is an assertion that the order-by field is either None or a valid field name (we can't order by a nonexistent field!). If the assertion isn't there, then I will get an error on this line: key_extractor = KeyExtractor(q_column_names.index(order_by_column)) In this particular case, I will get a ValueError (what does ValueError mean again? It means you've supplied an invalid value. What did you think it meant? In fact, what you get is: ValueError: list.index(x): x not in list which tells you exactly what went wrong and why. It's concise and self- explanatory. And what is this KeyExtractor?) Irrelevant. That's not part of the exception. It just happens to be on the same line of source code as the expression that raises an exception. since the index method will fail. I wrote the tiny relational database a long time ago, and I really don't want to put pressure on my mental cache by thinking about the internal logic of this chunk of code. After scratching my head for a while, I'll probably figure it out. Now imagine that you instead get an error on this line: assert order_by_column in q_column_names Now the programming error slaps me with a
Re: exception handling in complex Python programs
En Wed, 20 Aug 2008 21:49:14 -0300, [EMAIL PROTECTED] [EMAIL PROTECTED] escribió: On Aug 20, 10:59 am, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: Oh goodie. Another programmer who goes out of his way to make it hard for other programmers, by destroying duck-typing. Remember kids: personal attacks are cruise control for cool. So this was a simplification - most of the asserts I've written don't actually use isinstance, partly because typing isinstance takes too long. The point is to create a barricade so that when something goes wrong, you get an assertion error against the code you wrote, not an exception against doing something like print(blah blah %s % message) where message turns out to be None. This is simply a way to make debugging a more pleasant experience (quite valuable IMHO since debugging is inherently difficult and can be quite aggravating). Here is a sampling: assert statelt.tag == 'stat' assert len(path) 0 and path[0] == '/' assert self.__expr != None So here asserts are used to made distinctions that are more fine- grained than type. I think you missed the point. All of those look like program logic verification, and that's fine. But using assert to check user-supplied data is wrong (here user may be another programmer if you're developing a library). Assertions may be turned off at runtime. If you look at the history of the EAFP concept in Python, then you see that it comes from Alex Martelli's Python in a Nutshell around pages 113-114. Mmm, I think it's older than that. I don't think the code examples make the case for EAFP very well (not that I know what EAFP is in the first place, given that it is barely explained. Ok, so you don't know what it is, but dislike it anyway? I interpret it as wrap questionable stuff in try/ except blocks), and in any case there is practically no support for using EAFP as the dominant error-handling paradigm. Uh? I think that using try/except IS the dominant error-handling paradigm and that's just EAFP. Martelli's posts in support of EAFP are heavily skewed towards a multithreaded scenario and avoiding race conditions. IMHO, letting locking and race condition concerns dictate your error-handling paradigm is a case of the tail wagging the dog, especially when there are alternatives to this particular tar pit: pipes or a shared nothing architecture. There are race conditions since multiprogramming existed, around '70, and I'm afraid they'll stay for a long time... -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
exception handling in complex Python programs
Python provides a quite good and feature-complete exception handling mechanism for its programmers. This is good. But exceptions, like any complex construct, are difficult to use correctly, especially as programs get large. Most of the issues of exceptions are not specific to Python, but I sometimes feel that Python makes them more acute because of the free-n- easy manner in which it employs exceptions for its own uses and allows users to do the same. Now, what do I mean more specifically... When a program starts growing large, I find myself a bit scared of all the exceptions that might be thrown: Python's exceptions as a result of runtime-detection of errors (Python's dynamic typing also comes into play here), exceptions from libraries used by the code, and exceptions from my lower-level classes. Python doesn't allow to specify which exceptions are thrown (C++'s feature adding 'throw' after a function/method declaration specifying the exceptions that can be thrown), and this leaves me at loss - what should be caught and where ? Which errors should be left to propagate ? I've tried looking around the Python blogosphere, but there doesn't seem to be much concern with this topic. Apologies for the not-too-coherent post, but I suspect you feel the pain too and can understand my meaning. Eli P.S. There's a common case where a method is passed a filename, to do something with a file (say, read data). Should the method catch the errors possibly thrown by open(), or leave it to the caller ? P.P.S. There's a great post on conditions (Common Lisp's exceptions) here: http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/ Not really CL specific, and can apply to Python's exceptions. -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Tue, Aug 19, 2008 at 12:19 PM, eliben [EMAIL PROTECTED] wrote: Python provides a quite good and feature-complete exception handling mechanism for its programmers. This is good. But exceptions, like any complex construct, are difficult to use correctly, especially as programs get large. Most of the issues of exceptions are not specific to Python, but I sometimes feel that Python makes them more acute because of the free-n- easy manner in which it employs exceptions for its own uses and allows users to do the same. Lots of people seem to have this fear. They treat exceptions like they would treat error codes, trying to handle any possible case around any particular call. This is the wrong thing to do, and it only leads to more fragile code. There are only 2 reasonable things to do with an exception: 1) handle it, by which I mean catch the exception knowing what error condition it signifies, and take an appropriate action to correct the error and 2) pass it up so something else has a chance at it. Catching an exception when you don't know exactly what to do to fix it is an error. At best, it will make debugging a program harder (because you're losing context information about the error) and at worst it adds bugs to your program. The way Javas checked exceptions encourage empty or otherwise useless exception handlers is a major problem with them. There's some fear about presenting exceptions to the end user. That's a user interface issues, not a software quality or engineering issue, and it's resolvable with top-level handlers that log tracebacks somewhere a user can't see them if desired. Now, what do I mean more specifically... When a program starts growing large, I find myself a bit scared of all the exceptions that might be thrown: Python's exceptions as a result of runtime-detection of errors (Python's dynamic typing also comes into play here), exceptions from libraries used by the code, and exceptions from my lower-level classes. Python doesn't allow to specify which exceptions are thrown (C++'s feature adding 'throw' after a function/method declaration specifying the exceptions that can be thrown), and this leaves me at loss - what should be caught and where ? Which errors should be left to propagate ? You should catch anything that you can correct. If you don't have a specific answer for a specific exception, don't catch it. I've tried looking around the Python blogosphere, but there doesn't seem to be much concern with this topic. Apologies for the not-too-coherent post, but I suspect you feel the pain too and can understand my meaning. Eli P.S. There's a common case where a method is passed a filename, to do something with a file (say, read data). Should the method catch the errors possibly thrown by open(), or leave it to the caller ? Same rules apply. The only sort-of exception (no pun intended) is that sometimes you want to re-raise as a different type of exception. Make sure that you preserve all of the original information (including the original traceback) if you do this. P.P.S. There's a great post on conditions (Common Lisp's exceptions) here: http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/ Not really CL specific, and can apply to Python's exceptions. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 20, 12:19 am, eliben [EMAIL PROTECTED] wrote: Python provides a quite good and feature-complete exception handling mechanism for its programmers. This is good. But exceptions, like any complex construct, are difficult to use correctly, especially as programs get large. Most of the issues of exceptions are not specific to Python, but I sometimes feel that Python makes them more acute because of the free-n- easy manner in which it employs exceptions for its own uses and allows users to do the same. Now, what do I mean more specifically... When a program starts growing large, I find myself a bit scared of all the exceptions that might be thrown: Python's exceptions as a result of runtime-detection of errors (Python's dynamic typing also comes into play here), exceptions from libraries used by the code, and exceptions from my lower-level classes. Python doesn't allow to specify which exceptions are thrown (C++'s feature adding 'throw' after a function/method declaration specifying the exceptions that can be thrown), and this leaves me at loss - what should be caught and where ? Which errors should be left to propagate ? I've tried looking around the Python blogosphere, but there doesn't seem to be much concern with this topic. Apologies for the not-too-coherent post, but I suspect you feel the pain too and can understand my meaning. Eli P.S. There's a common case where a method is passed a filename, to do something with a file (say, read data). Should the method catch the errors possibly thrown by open(), or leave it to the caller ? P.P.S. There's a great post on conditions (Common Lisp's exceptions) here:http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-... Not really CL specific, and can apply to Python's exceptions. Maybe I am oversimplifying (and I am here to learn), but I catch all exceptions which otherwise would be hard to understand as a user. In other words, when a better error message is useful. Again, this is probably too simple to help, but the only way to ignore certain types of exceptions, as far as I know, is to catch them and pass. e.g. this ignores type errors... try: somethingBad() except TypeError, err: pass except Exception, err: raise TypeError(err) I suppose you could write a decorator to do this if you want it at the function level, but that seems a bit to broad. Shouldn't exceptions be on a case-by-case basis to add protection and return information exactly where it is needed? - Rafe -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
Rafe wrote: Again, this is probably too simple to help, but the only way to ignore certain types of exceptions, as far as I know, is to catch them and pass. e.g. this ignores type errors... try: somethingBad() except TypeError, err: pass except Exception, err: raise TypeError(err) so what kind of code are you writing where *type errors* are not considered programming errors? (catching them and proceeding is one thing, but catching them and ignoring them?) I'd be really worried if I found that in a piece of source code I had to maintain. /F -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Aug 19, 10:19 am, eliben [EMAIL PROTECTED] wrote: P.S. There's a common case where a method is passed a filename, to do something with a file (say, read data). Should the method catch the errors possibly thrown by open(), or leave it to the caller ? You want to look up Easier to Ask Forgivness than Permission (EAFP) which is touted as the canonical error-handling paradigm for Python. This would give rise to the following function: def do_something(filename): try: f = open(filename) except IOError: return err(File %s not found % filename) ... where err is a function that generates an error object that your application understands. I personally think this is sloppy because you have to couple the exception type with the function --- between file() and open() in Python 2 and 3, a NameError is thrown with open() in Python 3 and an IOError is thrown in the other three cases bashes head against keyboard. The alternative is def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... or, (and this last one I actually used for a web application) def do_something(filename): if not os.access(filename,os.R_OK): raise MyApplicationsExceptionType(File not found...) f = open(filename) ... The last one has the advantage that you can write a request handler like this def handle_http_request(...): func = specific_handler_func(...) try: response = func(...) return response except MyApplicationsExceptionType as exc: #3.0 syntax return error_response(exc,...) Exceptions you don't expect (i.e. bugs) will get handled by the web app framework, but you get to handle your own exceptions. Raising your own exception type can also be employed with the EAFP approach like this: def do_something(filename): try: f = open(filename) except IOError: raise MyApplicationsExceptionType(File %s not found % filename) ... If you are writing a library (for instance using a file for persistent storage), then the answer to your question is don't catch the exception. Clients will expect the usual exception to be thrown when a bad file name is passed. David -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
On Tue, 19 Aug 2008 11:07:39 -0700, [EMAIL PROTECTED] wrote: def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... You're running on a multitasking modern machine, right? What happens when some other process deletes filename, or changes its permissions, in the time after you check for access but before you actually open it? This isn't just a theoretical risk. There's a whole class of errors and security holes based on similar race conditions. I find it amusing that you consider it sloppy to deal with errors raised when actually opening a file, but then recommend a technique that has a well-known failure mode. That's not to say that I never use such techniques myself. For quick and dirty scripts, where I can tolerate the risk of some other process moving a file behind my back, I've been known to do something similar. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: exception handling in complex Python programs
between file() and open() in Python 2 and 3, a NameError is thrown with open() in Python 3 and an IOError is thrown in the other three cases bashes head against keyboard. This is *exactly* my concern with Python exceptions. You just never know what can be thrown at you. You want to look up Easier to Ask Forgivness than Permission (EAFP) which is touted as the canonical error-handling paradigm for Python. Any (semi)complete guides on this canonical paradigm online ? I've only found some references in maillist discussions. def do_something(filename): if not os.access(filename,os.R_OK): return err(...) f = open(filename) ... But does os.access cover absolutely all the errors that can happen during open() ? What guarantees it, and how can I know without you teaching me, just from the docs ? -- http://mail.python.org/mailman/listinfo/python-list
