Re: [Python-ideas] Improving Catching Exceptions

[Jan Kaliszewski]

2017-06-25 Greg Ewing  dixit:

> > (2) There's a *specific* problem with property where a bug in your 
> > getter or setter that raises AttributeError will be masked,
> > appearing as if the property itself doesn't exist.
[...]
> Case 2 needs to be addressed within the method concerned on a
> case-by-case basis. If there's a general principle there, it's
> something like this: If you're writing a method that uses
> an exception as part of it's protocol, you should catch any
> incidental occurrences of the same exception and reraise it
> as a different exception.
> 
> I don't think there's anything more the Python language could do
> to help with either of those.

In "case 2", maybe some auto-zeroing flag (or even a decrementing
counter?) could be useful?  Please, consider the following draft:

class ExtAttributeError(AttributeError):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._propagate_beyond = True

# of course this should be a class and it should support not only
# getters but also setters and deleters -- but you get the idea...
def new_property(func):
def wrapper(self):
try:
return func(self)
except AttributeError as exc:
if getattr(exc, '_propagate_beyond', False):
exc._propagate_beyond = False
raise
raise RuntimeError(
f'Unexpected {exc.__class__.__name__}') from exc 
return wrapper

Then we could have:

class Spam:
@new_property
def target(self):
if len(self.targgets) == 1:
return self.targets[0]
raise ExtAttributeError(
'only exists when this has exactly one target')

Cheers.
*j
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Sven R. Kunze]

On 28.06.2017 21:14, Nathaniel Smith wrote:
With PEP 479 there was a different and better way to generate a 
StopIteration if you wanted one (just 'return'). Here I'm afraid 
existing projects might actually be relying on the implicit exception 
leakage in significant numbers :-/


My concern as well.

More generally, my impression is that this is one of the reasons why 
exceptions have fallen out of favor in more recent languages. They're 
certainly workable, and python's certainly not going to change now, 
but they do have these awkward aspects that weren't as clear 20 years 
ago and that now we have to live with.


I am quite satisfied with the ability of exceptions to expose bugs as 
fast and clear as possible. I still think we can improve on the catching 
side a little bit to narrow down the relevant exceptions.


Other than that, I would be interested to hear what system you have in 
mind. What alternative (borrowed from more recent languages) can you 
imagine?
Checking return values like C or golang? No ability to catch them at 
all? How to handle bugs in the context of UI applications where a crash 
in front of the user should be avoided?


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Sven R. Kunze]

On 28.06.2017 15:26, Nick Coghlan wrote:

1. In 3.3+ you can just catch ImportError, check "exc.name", and
re-raise if it's not for the module you care about


I see, didn't know that one. I gave it a try and it's not 100% the 
behavior I have expected, but one could workaround if the valid package 
structure is known. Not sure for certain.


"from foo.bar.baz import abc" can yield to "exc.name" being one of 
"foo", "foo.bar" or "foo.bar.baz". Not perfect but sort of doable.



2. There's a reasonable case to be made that importlib should include
an ImportError -> RuntimeError conversion around the call to
loader.exec_module (in the same spirit as PEP 479). That way, in:

 try:
 import myspeciallib
 except ImportError:
 import fallbacks.MySpecialLib as myspeciallib

any caught ImportError would relate to "myspeciallib", while uncaught
ImportErrors arising from *executing* "myspeciallib" will be converted
to RuntimeError, with the original ImportError as their __cause__.


Generally changing the behavior for ImportError doesn't sound like it 
would work for all projects out there.


For fallback imports, I am on your side, that's a real use case which 
can be solved by changing the behavior of ImportErrors. But for all 
imports? I don't know if that's good idea.



[People should use tools, guard against bugs and try to avoid mistakes.]


Sure, but I don't see who this can help, if I use third-party code. The 
cases, which I described in the original post, were simple cases, where 
we catch too many exception. So, I don't even have the chance to see the 
error, to file a bug report, to issue a pull request, etc. etc.



The cases I'm interested in are the ones where you're either
developing some kind of framework and you need to code that framework
defensively to guard against unexpected failures in the components
you're executing (e.g. exec_module() in the PEP 451 import protocol),
or else you're needing to adapt between two different kinds of
exception reporting protocol (e.g. KeyError to AttributeError and
vice-versa).


I am unsure what you mean by those abstract words "framework" and 
"components". But let me state it in different words: there are 
*raisers* and *catchers* which do the respective thing with exceptions.


If you control the code on both sides, things are easy to change. 
Pre-condition: you know the bug in the first place, which is hard when 
you catch too much.
If you control the raiser only, it doesn't help to say: "don't make 
mistakes, configure systems right, code better, etc." People will make 
mistakes, systems will be misconfigured, linters don't find everything, etc.
If you control the catcher only, you definitely want to narrow down the 
amount of caught exceptions as far as possible. This was the original 
intend of this thread IIRC. This way you help to discover bugs in 
raising code. Addition benefit, you catching code reacts only to the 
right exceptions.



One word about frameworks here. Django, for instance, is on both sides. 
The template engine is mostly on the catchers side, whereas the database 
layer is on the raisers side.



I get the feeling that the solutions presented here are way too 
complicated and error-prone. My opinion on this topic still is that 
catching exceptions is not mandatory. Nobody is forced to do it and it's 
even better to let exceptions bubble up to visualize bugs. If one still 
needs to catch them, he should only catch those he really, really needs 
to catch and nothing more. If this cannot be improved sensibly, well, so 
be it. Although I still don't find the argument presented against 
"catching shallow exception" a little bit too abstract compared to the 
practical benefit. Maybe, there's a better solution, maybe not.



Regards,
Sven

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nathaniel Smith]

On Jun 28, 2017 6:26 AM, "Nick Coghlan"  wrote:

On 28 June 2017 at 21:48, Sven R. Kunze  wrote:
> As it was snipped away, let me ask again:
>
> I don't see how this helps differentiating shallow and nested exceptions
> such as:
>
> try:
> with exception_guard(ImportError):
> import myspeciallib
> except RuntimeError: # catches shallow and nested ones
> import fallbacks.MySpecialLib as myspeciallib

There are two answers to that:

1. In 3.3+ you can just catch ImportError, check "exc.name", and
re-raise if it's not for the module you care about

2. There's a reasonable case to be made that importlib should include
an ImportError -> RuntimeError conversion around the call to
loader.exec_module (in the same spirit as PEP 479). That way, in:

try:
import myspeciallib
except ImportError:
import fallbacks.MySpecialLib as myspeciallib

any caught ImportError would relate to "myspeciallib", while uncaught
ImportErrors arising from *executing* "myspeciallib" will be converted
to RuntimeError, with the original ImportError as their __cause__.

So it would make sense to file an RFE against 3.7 proposing that
behavioural change (we couldn't reasonably do anything like that with
the old `load_module()` importer API, as raising ImportError was how
that API signalled "I can't load that". We don't have that problem
with `exec_module()`).


What about modules that want to raise ImportError to indicate that they
aren't available on the current system, perhaps because some of their
dependencies are missing? For example, 'import ssl' should raise an
ImportError if 'ssl.py' is present but '_ssl.so' is missing; the existence
of '_ssl.so' is an internal implementation detail. And perhaps 'import
trio.ssl' should raise ImportError if 'ssl' is missing. (Historically not
all systems have openssl available, so this is a common situation where
existing libraries contain ImportError guards.)

With PEP 479 there was a different and better way to generate a
StopIteration if you wanted one (just 'return'). Here I'm afraid existing
projects might actually be relying on the implicit exception leakage in
significant numbers :-/

More generally, my impression is that this is one of the reasons why
exceptions have fallen out of favor in more recent languages. They're
certainly workable, and python's certainly not going to change now, but
they do have these awkward aspects that weren't as clear 20 years ago and
that now we have to live with.

-n
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 28 June 2017 at 21:48, Sven R. Kunze  wrote:
> As it was snipped away, let me ask again:
>
> I don't see how this helps differentiating shallow and nested exceptions
> such as:
>
> try:
> with exception_guard(ImportError):
> import myspeciallib
> except RuntimeError: # catches shallow and nested ones
> import fallbacks.MySpecialLib as myspeciallib

There are two answers to that:

1. In 3.3+ you can just catch ImportError, check "exc.name", and
re-raise if it's not for the module you care about

2. There's a reasonable case to be made that importlib should include
an ImportError -> RuntimeError conversion around the call to
loader.exec_module (in the same spirit as PEP 479). That way, in:

try:
import myspeciallib
except ImportError:
import fallbacks.MySpecialLib as myspeciallib

any caught ImportError would relate to "myspeciallib", while uncaught
ImportErrors arising from *executing* "myspeciallib" will be converted
to RuntimeError, with the original ImportError as their __cause__.

So it would make sense to file an RFE against 3.7 proposing that
behavioural change (we couldn't reasonably do anything like that with
the old `load_module()` importer API, as raising ImportError was how
that API signalled "I can't load that". We don't have that problem
with `exec_module()`).

> At least in my tests, exception_guard works this way and I don't see any
> improvements to current behavior. Moreover, I am somewhat skeptical that
> using this recipe will really improve the situation. It's a lot of code
> where users don't have any stdlib support. I furthermore doubt that all
> Python coders will now wrap their properties using the guard. So, using
> these properties we will have almost no improvement. I still don't see it as
> the responsibility of coder of the property to guard against anything.
> Nobody is forced to catch exceptions when using a property.

Honestly, if folks are trying to write complex Python code without
using at least "pylint -E" as a static check for typos in attribute
names (regardless of whether those lines get executed or not), then
inadvertently hiding AttributeError in property and __getattr__
implementations is likely to be the least of their problems. So
pylint's structural checks, type analysis tools like MyPy, or more
advanced IDEs like PyCharm are typically going to be a better option
for folks wanting to guard against bugs in their *own* code than
adding defensive code purely as a cross-check on their own work.

The cases I'm interested in are the ones where you're either
developing some kind of framework and you need to code that framework
defensively to guard against unexpected failures in the components
you're executing (e.g. exec_module() in the PEP 451 import protocol),
or else you're needing to adapt between two different kinds of
exception reporting protocol (e.g. KeyError to AttributeError and
vice-versa).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Sven R. Kunze]

On 28.06.2017 08:00, Nick Coghlan wrote:

Right, and I'd like us to keep in mind the KeyError -> AttributeError
(and vice-versa) use case as well. Similar to ExitStack, it would be
appropriate to make some additions to the "recipes" section in the
docs that covered things like "Keep AttributeError from being
suppressed in a property implementation".


As it was snipped away, let me ask again:

I don't see how this helps differentiating shallow and nested exceptions 
such as:


try:
with exception_guard(ImportError):
import myspeciallib
except RuntimeError: # catches shallow and nested ones
import fallbacks.MySpecialLib as myspeciallib


At least in my tests, exception_guard works this way and I don't see any 
improvements to current behavior. Moreover, I am somewhat skeptical that 
using this recipe will really improve the situation. It's a lot of code 
where users don't have any stdlib support. I furthermore doubt that all 
Python coders will now wrap their properties using the guard. So, using 
these properties we will have almost no improvement. I still don't see 
it as the responsibility of coder of the property to guard against 
anything. Nobody is forced to catch exceptions when using a property.


If that's the "best" outcome, I will stick to

https://stackoverflow.com/questions/20459166/how-to-catch-an-importerror-non-recursively

because 1) Google finds it for me and 2) we don't have to maintain 100 
lines of code ourself.


Regards,
Sven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 28 June 2017 at 13:16, Chris Angelico  wrote:
> On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan  wrote:
>> While generator functions now do that implicitly for StopIteration,
>> and "raise X from Y" lets people write suitable exception handlers
>> themselves, we don't offer an easy way to do it with a context manager
>> (with statement as stack boundary), asynchronous context manager
>> (async with statement as stack boundary), or a function decorator
>> (execution frame as stack boundary).
>>
>> So while I prefer "contextlib.convert_exception" as the name (rather
>> than the "exception_guard" Steven used in his recipe), I'd definitely
>> be open to a bugs.python.org RFE and a PR against contextlib to add
>> such a construct to Python 3.7.
>>
>> We'd have a few specific API details to work out (e.g. whether or not
>> to accept arbitrary conversion functions as conversion targets in
>> addition to accepting exception types and iterables of exception
>> types, whether or not to allow "None" as the conversion target to get
>> the same behaviour as "contextlib.suppress"), but I'm already sold on
>> the general concept.
>
> I agree broadly, but I'm sure there'll be the usual ton of
> bikeshedding about the details. The idea behind this decorator, AIUI,
> is a declaration that "a FooException coming out of here is a bug",
> and if I were looking for that, I'd look for something about the
> function leaking an exception, or preventing exceptions. So maybe
> convert_exception will work, but definitely have a docs reference from
> contextlib.suppress to this ("if exceptions of this type would
> indicate code bugs, consider convert_exception instead"). In my
> testing, I've called it "no_leak" or some variant thereon, though
> that's a shorthand that wouldn't suit the stdlib.

Right, and I'd like us to keep in mind the KeyError -> AttributeError
(and vice-versa) use case as well. Similar to ExitStack, it would be
appropriate to make some additions to the "recipes" section in the
docs that covered things like "Keep AttributeError from being
suppressed in a property implementation".

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Chris Angelico]

On Wed, Jun 28, 2017 at 12:25 PM, Nick Coghlan  wrote:
> While generator functions now do that implicitly for StopIteration,
> and "raise X from Y" lets people write suitable exception handlers
> themselves, we don't offer an easy way to do it with a context manager
> (with statement as stack boundary), asynchronous context manager
> (async with statement as stack boundary), or a function decorator
> (execution frame as stack boundary).
>
> So while I prefer "contextlib.convert_exception" as the name (rather
> than the "exception_guard" Steven used in his recipe), I'd definitely
> be open to a bugs.python.org RFE and a PR against contextlib to add
> such a construct to Python 3.7.
>
> We'd have a few specific API details to work out (e.g. whether or not
> to accept arbitrary conversion functions as conversion targets in
> addition to accepting exception types and iterables of exception
> types, whether or not to allow "None" as the conversion target to get
> the same behaviour as "contextlib.suppress"), but I'm already sold on
> the general concept.

I agree broadly, but I'm sure there'll be the usual ton of
bikeshedding about the details. The idea behind this decorator, AIUI,
is a declaration that "a FooException coming out of here is a bug",
and if I were looking for that, I'd look for something about the
function leaking an exception, or preventing exceptions. So maybe
convert_exception will work, but definitely have a docs reference from
contextlib.suppress to this ("if exceptions of this type would
indicate code bugs, consider convert_exception instead"). In my
testing, I've called it "no_leak" or some variant thereon, though
that's a shorthand that wouldn't suit the stdlib.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 28 June 2017 at 06:03, Chris Angelico  wrote:
> On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze  wrote:
>> I would agree with you here but this "refactoring principle in Python"
>> doesn't work for control flow.
>>
>> Just look at "return", "break", "continue" etc. Exceptions are another way
>> of handling control flow. So, this doesn't apply here IMO.
>
> The ability to safely refactor control flow is part of why 'yield
> from' exists, and why PEP 479 changed how StopIteration bubbles. Local
> control flow is hard to refactor, but exceptions are global control
> flow, and most certainly CAN be refactored safely.

And PEP 479 establishes a precedent for how we handle the cases where
we decide we *don't* want a particular exception type to propagate
normally: create a boundary on the stack that converts the otherwise
ambiguous exception type to RuntimeError.

While generator functions now do that implicitly for StopIteration,
and "raise X from Y" lets people write suitable exception handlers
themselves, we don't offer an easy way to do it with a context manager
(with statement as stack boundary), asynchronous context manager
(async with statement as stack boundary), or a function decorator
(execution frame as stack boundary).

So while I prefer "contextlib.convert_exception" as the name (rather
than the "exception_guard" Steven used in his recipe), I'd definitely
be open to a bugs.python.org RFE and a PR against contextlib to add
such a construct to Python 3.7.

We'd have a few specific API details to work out (e.g. whether or not
to accept arbitrary conversion functions as conversion targets in
addition to accepting exception types and iterables of exception
types, whether or not to allow "None" as the conversion target to get
the same behaviour as "contextlib.suppress"), but I'm already sold on
the general concept.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Chris Angelico]

On Wed, Jun 28, 2017 at 5:49 AM, Sven R. Kunze  wrote:
> On 27.06.2017 13:41, Nick Coghlan wrote:
>>
>> The shallow exception notion breaks a fairly fundamental refactoring
>> principle in Python: you should be able to replace an arbitrary
>> expression with a subfunction or subgenerator that produces the same
>> result without any of the surrounding code being able to tell the
>> difference.
>
>
> I would agree with you here but this "refactoring principle in Python"
> doesn't work for control flow.
>
> Just look at "return", "break", "continue" etc. Exceptions are another way
> of handling control flow. So, this doesn't apply here IMO.

The ability to safely refactor control flow is part of why 'yield
from' exists, and why PEP 479 changed how StopIteration bubbles. Local
control flow is hard to refactor, but exceptions are global control
flow, and most certainly CAN be refactored safely.

ChrisA
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Sven R. Kunze]

On 27.06.2017 13:41, Nick Coghlan wrote:

The shallow exception notion breaks a fairly fundamental refactoring
principle in Python: you should be able to replace an arbitrary
expression with a subfunction or subgenerator that produces the same
result without any of the surrounding code being able to tell the
difference.


I would agree with you here but this "refactoring principle in Python" 
doesn't work for control flow.


Just look at "return", "break", "continue" etc. Exceptions are another 
way of handling control flow. So, this doesn't apply here IMO.



By contrast, Steven's exception_guard recipe just takes the existing
"raise X from Y" feature, and makes it available as a context manager
and function decorator.


I don't see how this helps differentiating shallow and nested exceptions 
such as:


try:
with exception_guard(ImportError):
import myspeciallib
except RuntimeError: # catches shallow and nested ones
import fallbacks.MySpecialLib as myspeciallib


Regards,
Sven

PS: this has nothing to do with cyclic imports. It can be a 
misconfiguration of the system which fails nested imports. In those 
cases, we fallback silently.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 27 June 2017 at 02:29, Sven R. Kunze  wrote:
> On 24.06.2017 01:37, MRAB wrote:
>>
>> I think a "shallow exception" would be one that's part of a defined API,
>> as distinct from one that is an artifact of the implementation, a leak in
>> the abstraction.
>
> I like the "shallow exception" idea most. It's simple and it covers most if
> not all issues. You also hit the nail with pointing to leaking abstractions.

The shallow exception notion breaks a fairly fundamental refactoring
principle in Python: you should be able to replace an arbitrary
expression with a subfunction or subgenerator that produces the same
result without any of the surrounding code being able to tell the
difference.

By contrast, Steven's exception_guard recipe just takes the existing
"raise X from Y" feature, and makes it available as a context manager
and function decorator.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Rob Cliffe]

On 24/06/2017 11:03, Steven D'Aprano wrote:

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:


In any case, this doesn't address the issue raised by the OP,
which in this example is that if the implementation of
bah.__getitem__ calls something else that raises an IndexError,
there's no easy way to distinguish that from one raised by
bah.__getitem__ itself.

I'm not convinced that's a meaningful distinction to make in general.
Consider the difference between these two classes:

class X:
def __getitem__(self, n):
if n < 0:
 n += len(self)
if not 0 <= n < len(self):
 raise IndexError
...

class Y:
def __getitem__(self, n):
self._validate(n)
...
 def _validate(self, n):
if n < 0:
 n += len(self)
if not 0 <= n < len(self):
 raise IndexError


The difference is a mere difference of refactoring. Why should one of
them be treated as "bah.__getitem__ raises itself" versus
"bah.__getitem__ calls something which raises"? That's just an
implementation detail.

I think we're over-generalizing this problem. There's two actual issues
here, and we shouldn't conflate them as the same problem:

(1) People write buggy code based on invalid assumptions of what can and
can't raise. E.g.:

 try:
 foo(baz[5])
 except IndexError:
 ... # assume baz[5] failed (but maybe foo can raise too?)


(2) There's a *specific* problem with property where a bug in your
getter or setter that raises AttributeError will be masked, appearing as
if the property itself doesn't exist.


In the case of (1), there's nothing Python the language can do to fix
that. The solution is to write better code. Question your assumptions.
Think carefully about your pre-conditions and post-conditions and
invariants. Plan ahead. Read the documentation of foo before assuming
it won't raise. In other words, be a better programmer.

If only it were that easy :-(

(Aside: I've been thinking for a long time that design by contract is a
very useful feature to have. It should be possibly to set a contract
that states that this function won't raise a certain exception, and if
it does, treat it as a runtime error. But this is still at a very early
point in my thinking.)

Python libraries rarely give a definitive list of what exceptions
functions can raise, so unless you wrote it yourself and know exactly
what it can and cannot do, defensive coding suggests that you assume any
function call might raise any exception at all:

 try:
 item = baz[5]
 except IndexError:
 ... # assume baz[5] failed
 else:
 foo(item)


Can we fix that? Well, maybe we should re-consider the rejection of PEP
463 (exception-catching expressions).

https://www.python.org/dev/peps/pep-0463/

I'm all in favour of that :-) but I don't see how it helps in this example:

try:
item = (baz[5] except IndexError: SomeSentinelValue)
if item == SomeSentinelValue:
... # assume baz[5] failed
else:
foo(item)

is clunkier than the original version.  Or am I missing something? Only 
if the normal and exceptional cases could be handled the same way would 
it help:


foo(baz[5] except IndexError: 0)

Rob Cliffe




Maybe we need a better way to assert that a certain function won't raise
a particular exception:

 try:
 item = bah[5]
 without IndexError:
 foo(item)
 except IndexError:
 ... # assume baz[5] failed

(But how is that different from try...except...else?)



In the case of (2), the masking of bugs inside property getters if they
happen to raise AttributeError, I think the std lib can help with that.
Perhaps a context manager or decorator (or both) that converts one
exception to another?

@property
@bounce_exception(AttributeError, RuntimeError)
def spam(self):
  ...


Now spam.__get__ cannot raise AttributeError, if it does, it will be
converted to RuntimeError. If you need finer control over the code that
is guarded use the context manager form:

@property
def spam(self):
 with bounce_exception(AttributeError, RuntimeError):
 # guarded
 if condition:
 ...
 # not guarded
 raise AttributeError('property spam doesn't exist yet')





___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Steven D'Aprano]

On Sat, Jun 24, 2017 at 11:45:25PM +1000, Nick Coghlan wrote:

> While I used to think that, I'm no longer sure it's true, as it seems
> to me that a `contextlib.convert_exception` context manager could help
> with both of them. 

Here is a recipe for such a context manager which is also useable as 
a decorator:

https://code.activestate.com/recipes/580808-guard-against-an-exception-in-the-wrong-place/

or just

https://code.activestate.com/recipes/580808


It should work with Python 2.6 through 3.6 and later.

try:
with exception_guard(ZeroDivisionError):
1/0  # raises ZeroDivisionError
except RuntimeError:
print ('ZeroDivisionError replaced by RuntimeError')



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 24 June 2017 at 22:31, Greg Ewing  wrote:
> Steven D'Aprano wrote:
>> I think we're over-generalizing this problem. There's two actual issues
>> here, and we shouldn't conflate them as the same problem:
>>
>> (1) People write buggy code based on invalid assumptions of what can and
>> can't raise. E.g.:
>>
>> (2) There's a *specific* problem with property where a bug in your getter
>> or setter that raises AttributeError will be masked, appearing as if the
>> property itself doesn't exist.
>
> Agreed.
>
> Case 1 can usually be handled by rewriting the code so as to
> make the scope of exception catching as narrow as possible.
>
> Case 2 needs to be addressed within the method concerned on a
> case-by-case basis. If there's a general principle there, it's
> something like this: If you're writing a method that uses
> an exception as part of it's protocol, you should catch any
> incidental occurrences of the same exception and reraise it
> as a different exception.
>
> I don't think there's anything more the Python language could do
> to help with either of those.

While I used to think that, I'm no longer sure it's true, as it seems
to me that a `contextlib.convert_exception` context manager could help
with both of them. (That's technically the standard library helping,
rather than the language per se, but it's still taking a currently
obscure implementation pattern and making it more obvious by giving it
a name)

So if we assume that existed, and converted the given exception to
RuntimeError (the same way PEP 479 does for StopIteration), we'd be
able to write magic methods and properties in the following style:

   def __getitem__(self, n):
  self.validate(n)
  with contextlib.convert_exception(IndexError):
  # If we get an IndexError in here, it's a bug
  return self._getitem(n)

That idiom then works the same way regardless of how far away your
code is from the exception handler you're trying to bypass - you could
just as easily put it inside the `self._getitem(n)` helper method
instead.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Greg Ewing]

Steven D'Aprano wrote:

class X:
   def __getitem__(self, n):
   if n < 0:
n += len(self)
   if not 0 <= n < len(self):
raise IndexError
   ...

class Y:
   def __getitem__(self, n):
   self._validate(n)
   ...
def _validate(self, n):
   if n < 0:
n += len(self)
   if not 0 <= n < len(self):
raise IndexError


Why should one of 
them be treated as "bah.__getitem__ raises itself" versus 
"bah.__getitem__ calls something which raises"?


They shouldn't be treated differently -- they're both legitimate
ways for __getitem__ to signal that the item doesn't exist.

What *should* be treated differently is if an IndexError occurs
incidentally from something else that __getitem__ does. In
other words, Y's __getitem__ should be written something like

   def __getitem__(self, n):
  self.validate(n)
  # If we get an IndexError from here on, it's a bug
  try:
 # work out the result and return it
  except IndexError as e:
 raise RuntimeError from e

I think we're over-generalizing this problem. There's two actual issues 
here, and we shouldn't conflate them as the same problem:


(1) People write buggy code based on invalid assumptions of what can and 
can't raise. E.g.:


(2) There's a *specific* problem with property where a bug in your 
getter or setter that raises AttributeError will be masked, appearing as 
if the property itself doesn't exist.


Agreed.

Case 1 can usually be handled by rewriting the code so as to
make the scope of exception catching as narrow as possible.

Case 2 needs to be addressed within the method concerned on a
case-by-case basis. If there's a general principle there, it's
something like this: If you're writing a method that uses
an exception as part of it's protocol, you should catch any
incidental occurrences of the same exception and reraise it
as a different exception.

I don't think there's anything more the Python language could do
to help with either of those.

(Aside: I've been thinking for a long time that design by contract is a 
very useful feature to have. It should be possibly to set a contract 
that states that this function won't raise a certain exception, and if 
it does, treat it as a runtime error. But this is still at a very early 
point in my thinking.)


That sounds dangerously similar to Java's checked exceptions,
which has turned out to be a huge nuisance and not very
helpful.

Maybe we need a better way to assert that a certain function won't raise 
a particular exception:


try:
item = bah[5]
without IndexError:
foo(item)
except IndexError:
... # assume baz[5] failed

(But how is that different from try...except...else?)


It's no different, if I understand what it's supposed to mean
correctly.


@property
@bounce_exception(AttributeError, RuntimeError)
def spam(self):
 ...


In the case of property getters, it seems to me you're almost
always going to want that functionality, so maybe it should
be incorporated into the property decorator itself.

The only cases where you wouldn't want it would be if your
property dynamically figures out whether it exists or not,
and in those rare cases you would just have to write your
own descriptor class.

--
Greg

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Steven D'Aprano]

On Sat, Jun 24, 2017 at 01:02:55PM +1200, Greg Ewing wrote:

> In any case, this doesn't address the issue raised by the OP,
> which in this example is that if the implementation of
> bah.__getitem__ calls something else that raises an IndexError,
> there's no easy way to distinguish that from one raised by
> bah.__getitem__ itself.

I'm not convinced that's a meaningful distinction to make in general. 
Consider the difference between these two classes:

class X:
   def __getitem__(self, n):
   if n < 0:
n += len(self)
   if not 0 <= n < len(self):
raise IndexError
   ...

class Y:
   def __getitem__(self, n):
   self._validate(n)
   ...
def _validate(self, n):
   if n < 0:
n += len(self)
   if not 0 <= n < len(self):
raise IndexError


The difference is a mere difference of refactoring. Why should one of 
them be treated as "bah.__getitem__ raises itself" versus 
"bah.__getitem__ calls something which raises"? That's just an 
implementation detail.

I think we're over-generalizing this problem. There's two actual issues 
here, and we shouldn't conflate them as the same problem:

(1) People write buggy code based on invalid assumptions of what can and 
can't raise. E.g.:

try:
foo(baz[5])
except IndexError:
... # assume baz[5] failed (but maybe foo can raise too?)


(2) There's a *specific* problem with property where a bug in your 
getter or setter that raises AttributeError will be masked, appearing as 
if the property itself doesn't exist.


In the case of (1), there's nothing Python the language can do to fix 
that. The solution is to write better code. Question your assumptions. 
Think carefully about your pre-conditions and post-conditions and 
invariants. Plan ahead. Read the documentation of foo before assuming 
it won't raise. In other words, be a better programmer.

If only it were that easy :-(

(Aside: I've been thinking for a long time that design by contract is a 
very useful feature to have. It should be possibly to set a contract 
that states that this function won't raise a certain exception, and if 
it does, treat it as a runtime error. But this is still at a very early 
point in my thinking.)

Python libraries rarely give a definitive list of what exceptions 
functions can raise, so unless you wrote it yourself and know exactly 
what it can and cannot do, defensive coding suggests that you assume any 
function call might raise any exception at all:

try:
item = baz[5]
except IndexError:
... # assume baz[5] failed
else:
foo(item)


Can we fix that? Well, maybe we should re-consider the rejection of PEP 
463 (exception-catching expressions).

https://www.python.org/dev/peps/pep-0463/


Maybe we need a better way to assert that a certain function won't raise 
a particular exception:

try:
item = bah[5]
without IndexError:
foo(item)
except IndexError:
... # assume baz[5] failed

(But how is that different from try...except...else?)



In the case of (2), the masking of bugs inside property getters if they 
happen to raise AttributeError, I think the std lib can help with that. 
Perhaps a context manager or decorator (or both) that converts one 
exception to another?

@property
@bounce_exception(AttributeError, RuntimeError)
def spam(self):
 ...


Now spam.__get__ cannot raise AttributeError, if it does, it will be 
converted to RuntimeError. If you need finer control over the code that 
is guarded use the context manager form:

@property
def spam(self):
with bounce_exception(AttributeError, RuntimeError):
# guarded
if condition:
...
# not guarded
raise AttributeError('property spam doesn't exist yet')



-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Greg Ewing]

Cameron Simpson wrote:


   try:
   foo(bah[5])
   except IndexError as e:
   ... infer that there is no bah[5] ...

One can easily want, instead, some kind of "shallow except", which would 
catch exceptions only if they were directly raised from the surface 
code;


The problem I see with that is how to define what counts as
"surface code". If the __getitem__ method of bah is written in
Python, I don't see how you could tell that an IndexError raised
by it should be caught, but one raised by foo() shouldn't.

In any case, this doesn't address the issue raised by the OP,
which in this example is that if the implementation of
bah.__getitem__ calls something else that raises an IndexError,
there's no easy way to distinguish that from one raised by
bah.__getitem__ itself.

I don't see any way to solve that by messing around with
different try-except constructs. It can only be addressed from
within bah.__getitem__ itself, by having it catch any
incidental IndexErrors and turn them into a different
exception.

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Cameron Simpson]

On 23Jun2017 20:30, Stephan Houben  wrote:

2017-06-23 17:09 GMT+02:00 Andy Dirnberger :

It's not really a proposal. It's existing syntax.


Wow! I have been using Python since 1.5.2 and I never knew this.
This is not Guido's famous time machine in action, by any chance?
Guess there's some code to refactor using this construct now...


Alas, no. It is existing syntax in Standard ML, not in Python.

Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Cameron Simpson]

On 23Jun2017 15:59, Paul Moore  wrote:

On 23 June 2017 at 15:20, Sven R. Kunze  wrote:

On 23.06.2017 03:02, Cameron Simpson wrote:
How about something like this?

   try:
   val = bah[5]
   except IndexError:
   # handle your expected exception here
   else:
   foo(val)

That is the kind of refactor to which I alluded in the paragraph above.
Doing that a lot tends to obscure the core logic of the code, hence the
desire for something more succinct requiring less internal code fiddling.

And depending on how complex bha.__getitem__ is, it can raise IndexError
unintentionally as well. So, rewriting the outer code doesn't even help
then. :-(


At this point, it becomes unclear to me what constitutes an
"intentional" IndexError, as opposed to an "unintentional" one, at
least in any sense that can actually be implemented.


While I agree that in object with its own __getitem__ would look "deep", what I 
was actually suggesting as a possibility was a "shallow" except catch, not some 
magic "intentional" semantic.


A shallow catch would effectively need to mean "the exceptions uppermost 
traceback frame referers to one of the program lines in the try/except suite".  
Which would work well for lists and other builtin types. And might be 
insufficient for a duck-type with python-coded dunder methods.


[...snip...]

On the other hand, I do see the point that insisting on finer and
finer grained exception handling ultimately ends up with unreadable
code. But it's not a problem I'd expect to see much in real life code
(where code is either not written that defensively, because either
there's context that allows the coder to make assumptions that objects
will behave reasonably sanely, or the code gets refactored to put the
exception handling in a function, or something like that).


Sure, there are many circumstances where a succinct "shallow catch" might not 
be useful. But there are also plenty of circumstances where one would like just 
this flavour of precision.


Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Cameron Simpson]

On 23Jun2017 11:48, Nick Coghlan  wrote:

On 23 June 2017 at 09:29, Cameron Simpson  wrote:

This is so common that I actually keep around a special hack:

   def prop(func):
 ''' The builtin @property decorator lets internal AttributeErrors
escape.
 While that can support properties that appear to exist
conditionally,
 in practice this is almost never what I want, and it masks deeper
errors.
 Hence this wrapper for @property that transmutes internal
AttributeErrors
 into RuntimeErrors.
 '''
 def wrapper(*a, **kw):
   try:
 return func(*a, **kw)
   except AttributeError as e:
 e2 = RuntimeError("inner function %s raised %s" % (func, e))
 if sys.version_info[0] >= 3:
   try:
 eval('raise e2 from e', globals(), locals())
   except:
 # FIXME: why does this raise a SyntaxError?
 raise e
 else:
   raise e2
 return property(wrapper)


Slight tangent, but I do sometimes wonder if adding a decorator
factory like the following to functools might be useful:

   def raise_when_returned(return_exc):
   def decorator(f):
   @wraps(f)
   def wrapper(*args, **kwds):
   try:
   result = f(*args, **kwds)
   except selective_exc as unexpected_exc:
   msg = "inner function {} raised {}".format(f,
unexpected_exc)
   raise RuntimeError(msg) from unexpected_exc
   if isinstance(result, return_exc):
   raise result
   return result

It's essentially a generalisation of PEP 479 to arbitrary exception
types, since it lets you mark a particular exception type as being
communicated back to the wrapper via the return channel rather than as
a regular exception:

   def with_traceback(exc):
   try:
   raise exc
   except BaseException as caught_exc:
   return caught_exc

   @property
   @raise_when_returned(AttributeError)
   def target(self):
   if len(self.targets) == 1:
   return self.targets[0]
   return with_traceback(AttributeError('only exists when this
has exactly one target'))


Funnily enough I have an @transmute decorator which serves just this purpose.  

It doesn't see as much use as I might imagine, but that is partially because my 
function predates "raise ... from", which meant that it loses the stack trace 
from the transmuted exception, impeding debugging. I need to revisit it with 
that in mind.


So yes, your proposed decorator has supporting real world use cases in my 
world.


Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Cameron Simpson]

On 24Jun2017 05:02, Steven D'Aprano  wrote:

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:

On 23Jun2017 06:55, Steven D'Aprano  wrote:
>On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:
>>We usually teach our newbies to catch exceptions as narrowly as
>>possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This
>>works out quite well for now but the number of examples continue to grow
>>where it's not enough.
>
>(1) Under what circumstances is it not enough?

I believe that he means that it isn't precise enough. In particular,
"nested exceptions" to me, from his use cases, means exceptions thrown from
within functions called at the top level. I want this control too sometimes.


But why teach it to newbies? Sven explicitly mentions teaching
beginners. If we are talking about advanced features for experts, that's
one thing, but it's another if we're talking about Python 101 taught to
beginners and newbies.


It depends. Explaining that exceptions from called code can be mishandled by a 
naive try/except is something newbies need to learn to avoid common pitfalls 
with exceptions, and a real world situation that must be kept in mind when 
acting on caught exceptions.



Do we really need to be teaching beginners how to deal with circular
imports beyond "don't do it"?


Sven's example is with import. The situation is more general.

[... snip basic example of simple code where IndexError can arise from
 multiple causes
...]

Naturally one can rearrange this code to call foo() outside that
try/except, but that degree of control often leads to quite fiddly looking
code with the core flow obscured by many tiny try/excepts.


Sadly, that is often the nature of real code, as opposed to "toy" or
textbook code that demonstrates an algorithm as cleanly as possible.
It's been said that for every line of code in the textbook, the
function needs ten lines in production.


But not always so. And the reason for many language constructs and idioms is 
explicitly to combat what would otherwise need 10 lines of code (or 100 to do 
it "right" with corner cases), obscuring the core task and reducing 
readability/maintainability.


So "Sadly, that is often the nature of real code" is not itself an argument 
against this idea.



One can easily want, instead, some kind of "shallow except", which would
catch exceptions only if they were directly raised from the surface code;
such a construct would catch the IndexError from a missing bah[5] in the
example above, but _not_ catch an IndexError raised from deeper code such
within the foo() function.


I think the concept of a "shallow exception" is ill-defined, and to the
degree that it is defined, it is *dangerous*: a bug magnet waiting to
strike.

What do you mean by "directly raised from the surface code"? Why is
bah[5] "surface code" but foo(x) is not? But call a function (or
method).

[...]

I've replied to Paul Moore and suggested this definition as implementable and 
often useful:


 A shallow catch would effectively need to mean "the exception's
 uppermost traceback frame refers to one of the program lines
 in the try/except suite".  Which would work well for lists and
 other builtin types. And might be insufficient for a duck-type
 with python-coded dunder methods.

The target here is not perform magic but to have a useful tool to identify 
exceptions that arise fairly directly from the adjacent clode and not what it 
calls. Without writing cumbersome and fragile boilerplate to dig into an 
exception's traceback.


Cheers,
Cameron Simpson 
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Steven D'Aprano]

On Fri, Jun 23, 2017 at 09:29:23AM +1000, Cameron Simpson wrote:
> On 23Jun2017 06:55, Steven D'Aprano  wrote:
> >On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:
> >>We usually teach our newbies to catch exceptions as narrowly as
> >>possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This
> >>works out quite well for now but the number of examples continue to grow
> >>where it's not enough.
> >
> >(1) Under what circumstances is it not enough?
> 
> I believe that he means that it isn't precise enough. In particular, 
> "nested exceptions" to me, from his use cases, means exceptions thrown from 
> within functions called at the top level. I want this control too sometimes.

But why teach it to newbies? Sven explicitly mentions teaching 
beginners. If we are talking about advanced features for experts, that's 
one thing, but it's another if we're talking about Python 101 taught to 
beginners and newbies.

Do we really need to be teaching beginners how to deal with circular 
imports beyond "don't do it"?



> Consider:
> 
>try:
>foo(bah[5])
>except IndexError as e:
>... infer that there is no bah[5] ...
> 
> Of course, it is possible that bah[5] existed and that foo() raised an 
> IndexError of its own. One might intend some sane handling of a missing 
> bah[5] but instead silently conceal the IndexError from foo() by 
> mishandling it as a missing bah[5].

Indeed -- if both foo and bah[5] can raise IndexError when the coder 
believes that only bah[5] can, then the above code is simply buggy.

On the other hand, if the author is correct that foo cannot raise 
IndexError, then the code as given is fine.


> Naturally one can rearrange this code to call foo() outside that 
> try/except, but that degree of control often leads to quite fiddly looking 
> code with the core flow obscured by many tiny try/excepts.

Sadly, that is often the nature of real code, as opposed to "toy" or 
textbook code that demonstrates an algorithm as cleanly as possible. 
It's been said that for every line of code in the textbook, the 
function needs ten lines in production.


> One can easily want, instead, some kind of "shallow except", which would 
> catch exceptions only if they were directly raised from the surface code; 
> such a construct would catch the IndexError from a missing bah[5] in the 
> example above, but _not_ catch an IndexError raised from deeper code such 
> within the foo() function.

I think the concept of a "shallow exception" is ill-defined, and to the 
degree that it is defined, it is *dangerous*: a bug magnet waiting to 
strike.

What do you mean by "directly raised from the surface code"? Why is 
bah[5] "surface code" but foo(x) is not? But call a function (or 
method).

But worse, it seems that the idea of "shallow" or "deep" depends on 
*implementation details* of where the exception comes from.

For example, changing from a recursive function to a while loop might 
change the exception from "50 function calls deep" to "1 function deep".

What makes bah[5] "shallow"? For all you know, it calls a chain of a 
dozen __getitem__ methods, due to inheritance or proxying, before the 
exception is actually raised. Or it might call just a single __getitem__ 
method, but the method's implementation puts the error checking into a 
helper method:

def __getitem__(self, n):
self._validate(n)  # may raise IndexError
...

How many function calls are shallow, versus deep?

This concept of a shallow exception is, it seems to me, a bug magnet. It 
is superficially attractive, but then you realise that:

try:
spam[5]
except shallow IndexError:
...

will behave differently depending on how spam is implemented, even if 
the interface (raises IndexError) is identical.

It seems to me that this concept is trying to let us substitute some 
sort of undefined but mechanical measurement of "shallowness" for 
actually understanding what our code does. I don't think this can work.

It would be awesome if there was some way for our language to Do What We
Mean instead of What We Say. And then we can grow a money tree, and have 
a magic plum-pudding that stays the same size no matter how many slices 
we eat, and electricity so cheap the power company pays you to use it...

*wink*


> The nested exception issue actually bites me regularly, almost always with 
> properties.
[...]
> However, more commonly I end up hiding coding errors with @property, 
> particularly nasty when the coding error is deep in some nested call. Here 
> is a nondeep example based on the above:
> 
>@property
>def target(self):
>if len(self.targgets) == 1:
>return self.targets[0]
>raise AttributeError('only exists when this has exactly one target')

The obvious solution to this is to learn to spell correctly :-)

Actually, a linter probably would have picked up that typo. But I do 
see that the issue if more than just typos.

Re: [Python-ideas] Improving Catching Exceptions

[Stephan Houben]

2017-06-23 17:09 GMT+02:00 Andy Dirnberger :

> It's not really a proposal. It's existing syntax.

Wow! I have been using Python since 1.5.2 and I never knew this.

This is not Guido's famous time machine in action, by any chance?

Guess there's some code to refactor using this construct now...

Stephan


2017-06-23 17:09 GMT+02:00 Andy Dirnberger :
> Hi Stephan,
>
> On Fri, Jun 23, 2017 at 6:23 AM, Stephan Houben 
> wrote:
>>
>> Hi Andy,
>>
>> What you propose is essentially the "try .. catch .. in" construct as
>> described for Standard ML in:
>
>
> It's not really a proposal. It's existing syntax. I was suggesting a way to
> implement the example that would catch an IndexError raised by accessing
> elements in bah but not those raised by foo.
>
>
>>
>>
>>
>> https://pdfs.semanticscholar.org/b24a/60f84b296482769bb6752feeb3d93ba6aee8.pdf
>>
>> Something similar for Clojure is at:
>> https://github.com/rufoa/try-let
>>
>> So clearly this is something more people have struggled with.
>> The paper above goes into deep detail on the practical and
>> (proof-)theoretical
>> advantages of such a construct.
>>
>> Stephan
>
>
> Andy
>
>
>>
>>
>> 2017-06-23 1:47 GMT+02:00 Andy Dirnberger :
>> >
>> >
>> >> On Jun 22, 2017, at 7:29 PM, Cameron Simpson  wrote:
>> >>
>> >>> On 23Jun2017 06:55, Steven D'Aprano  wrote:
>>  On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:
>>  We usually teach our newbies to catch exceptions as narrowly as
>>  possible, i.e. MyModel.DoesNotExist instead of a plain Exception.
>>  This
>>  works out quite well for now but the number of examples continue to
>>  grow
>>  where it's not enough.
>> >>>
>> >>> (1) Under what circumstances is it not enough?
>> >>
>> >> I believe that he means that it isn't precise enough. In particular,
>> >> "nested exceptions" to me, from his use cases, means exceptions thrown 
>> >> from
>> >> within functions called at the top level. I want this control too 
>> >> sometimes.
>> >>
>> >> Consider:
>> >>
>> >>   try:
>> >>   foo(bah[5])
>> >>   except IndexError as e:
>> >>   ... infer that there is no bah[5] ...
>> >>
>> >> Of course, it is possible that bah[5] existed and that foo() raised an
>> >> IndexError of its own. One might intend some sane handling of a missing
>> >> bah[5] but instead silently conceal the IndexError from foo() by 
>> >> mishandling
>> >> it as a missing bah[5].
>> >>
>> >> Naturally one can rearrange this code to call foo() outside that
>> >> try/except, but that degree of control often leads to quite fiddly looking
>> >> code with the core flow obscured by many tiny try/excepts.
>> >>
>> >> One can easily want, instead, some kind of "shallow except", which
>> >> would catch exceptions only if they were directly raised from the surface
>> >> code; such a construct would catch the IndexError from a missing bah[5] in
>> >> the example above, but _not_ catch an IndexError raised from deeper code
>> >> such within the foo() function.
>> >>
>> >> Something equivalent to:
>> >>
>> >>   try:
>> >>   foo(bah[5])
>> >>   except IndexError as e:
>> >>   if e.__traceback__ not directly from the try..except lines:
>> >>   raise
>> >>   ... infer that there is no bah[5] ...
>> >>
>> >> There doesn't seem to be a concise way to write that. It might not even
>> >> be feasible at all, as one doesn't have a way to identify the line(s) 
>> >> within
>> >> the try/except in a form that one can recognise in a traceback.
>> >
>> > How about something like this?
>> >
>> > try:
>> > val = bah[5]
>> > except IndexError:
>> > # handle your expected exception here
>> > else:
>> > foo(val)
>> >>
>> >>
>> >> Cheers,
>> >> Cameron Simpson 
>> >> ___
>> >> Python-ideas mailing list
>> >> Python-ideas@python.org
>> >> https://mail.python.org/mailman/listinfo/python-ideas
>> >> Code of Conduct: http://python.org/psf/codeofconduct/
>> > ___
>> > Python-ideas mailing list
>> > Python-ideas@python.org
>> > https://mail.python.org/mailman/listinfo/python-ideas
>> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Paul Moore]

On 23 June 2017 at 15:20, Sven R. Kunze  wrote:
> On 23.06.2017 03:02, Cameron Simpson wrote:
>
>
> How about something like this?
>
>try:
>val = bah[5]
>except IndexError:
># handle your expected exception here
>else:
>foo(val)
>
>
> That is the kind of refactor to which I alluded in the paragraph above.
> Doing that a lot tends to obscure the core logic of the code, hence the
> desire for something more succinct requiring less internal code fiddling.
>
>
> And depending on how complex bha.__getitem__ is, it can raise IndexError
> unintentionally as well. So, rewriting the outer code doesn't even help
> then. :-(

At this point, it becomes unclear to me what constitutes an
"intentional" IndexError, as opposed to an "unintentional" one, at
least in any sense that can actually be implemented.

I appreciate that you want IndexError to mean "there is no 5th element
in bah". But if bah has a __getitem__ that raises IndexError for any
reason other than that, then the __getitem__ implementation has a bug.
And while it might be nice to be able to continue working properly
even when the code you're executing has bugs, I think it's a bit
optimistic to hope for :-)

On the other hand, I do see the point that insisting on finer and
finer grained exception handling ultimately ends up with unreadable
code. But it's not a problem I'd expect to see much in real life code
(where code is either not written that defensively, because either
there's context that allows the coder to make assumptions that objects
will behave reasonably sanely, or the code gets refactored to put the
exception handling in a function, or something like that).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[MRAB]

On 2017-06-23 00:29, Cameron Simpson wrote:

On 23Jun2017 06:55, Steven D'Aprano  wrote:

On Thu, Jun 22, 2017 at 10:30:57PM +0200, Sven R. Kunze wrote:

We usually teach our newbies to catch exceptions as narrowly as
possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This
works out quite well for now but the number of examples continue to grow
where it's not enough.


(1) Under what circumstances is it not enough?


I believe that he means that it isn't precise enough. In particular, "nested
exceptions" to me, from his use cases, means exceptions thrown from within
functions called at the top level. I want this control too sometimes.

Consider:

 try:
 foo(bah[5])
 except IndexError as e:
 ... infer that there is no bah[5] ...

Of course, it is possible that bah[5] existed and that foo() raised an
IndexError of its own. One might intend some sane handling of a missing bah[5]
but instead silently conceal the IndexError from foo() by mishandling it as a
missing bah[5].

Naturally one can rearrange this code to call foo() outside that try/except,
but that degree of control often leads to quite fiddly looking code with the
core flow obscured by many tiny try/excepts.

One can easily want, instead, some kind of "shallow except", which would catch
exceptions only if they were directly raised from the surface code; such a
construct would catch the IndexError from a missing bah[5] in the example
above, but _not_ catch an IndexError raised from deeper code such within the
foo() function.

Something equivalent to:

 try:
 foo(bah[5])
 except IndexError as e:
 if e.__traceback__ not directly from the try..except lines:
 raise
 ... infer that there is no bah[5] ...

There doesn't seem to be a concise way to write that. It might not even be
feasible at all, as one doesn't have a way to identify the line(s) within the
try/except in a form that one can recognise in a traceback.


[snip]

Increment a counter on every function call and record it on the 
exception, perhaps?


If the exception's call count == the current call count, it was raised 
in this function.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Improving Catching Exceptions

[Nick Coghlan]

On 23 June 2017 at 09:29, Cameron Simpson  wrote:
> This is so common that I actually keep around a special hack:
>
>def prop(func):
>  ''' The builtin @property decorator lets internal AttributeErrors
> escape.
>  While that can support properties that appear to exist
> conditionally,
>  in practice this is almost never what I want, and it masks deeper
> errors.
>  Hence this wrapper for @property that transmutes internal
> AttributeErrors
>  into RuntimeErrors.
>  '''
>  def wrapper(*a, **kw):
>try:
>  return func(*a, **kw)
>except AttributeError as e:
>  e2 = RuntimeError("inner function %s raised %s" % (func, e))
>  if sys.version_info[0] >= 3:
>try:
>  eval('raise e2 from e', globals(), locals())
>except:
>  # FIXME: why does this raise a SyntaxError?
>  raise e
>  else:
>raise e2
>  return property(wrapper)

Slight tangent, but I do sometimes wonder if adding a decorator
factory like the following to functools might be useful:

def raise_when_returned(return_exc):
def decorator(f):
@wraps(f)
def wrapper(*args, **kwds):
try:
result = f(*args, **kwds)
except selective_exc as unexpected_exc:
msg = "inner function {} raised {}".format(f,
unexpected_exc)
raise RuntimeError(msg) from unexpected_exc
if isinstance(result, return_exc):
raise result
return result

It's essentially a generalisation of PEP 479 to arbitrary exception
types, since it lets you mark a particular exception type as being
communicated back to the wrapper via the return channel rather than as
a regular exception:

def with_traceback(exc):
try:
raise exc
except BaseException as caught_exc:
return caught_exc

@property
@raise_when_returned(AttributeError)
def target(self):
if len(self.targets) == 1:
return self.targets[0]
return with_traceback(AttributeError('only exists when this
has exactly one target'))

The part I don't like about that approach is the fact that you need to
mess about with the exception internals to get a halfway decent
traceback on the AttributeError.

The main alternative would be to add a "convert_exception" context
manager in contextlib, so you could write the example property as:

@property
def target(self):
with convert_exception(AttributeError):
if len(self.targets) == 1:
return self.targets[0]
raise AttributeError('only exists when this has exactly one target')

Where "convert_exception" would be something like:

def convert_exception(exc_type):
"""Prevents the given exception type from escaping a region of
code by converting it to RuntimeError"""
if not issubclass(exc_type, Exception):
raise TypeError("Only Exception subclasses can be flagged
as unexpected")
try:
 yield
except exc_type as unexpected_exc:
new_exc = RuntimeError("Unexpected exception")
raise new_exc from unexpected_exc

The API for this could potentially be made more flexible to allow easy
substition of lookup errors with attribute errors and vice-versa (e.g.
via additional keyword-only parameters)

To bring the tangent back closer to Sven's original point, there are
probably also some parts of the import system (such as executing the
body of a found module) where the case can be made that we should be
converting ImportError to RuntimeError, rather than letting the
ImportError escape (with essentially the same rationale as PEP 479).


Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/