[Python-ideas] Re: Suggestion: annotated exceptions

[sascha.schlemmer--- via Python-ideas]

> From a cultural perspective (with python traditionally not being written in 
> this style)
> I think writing code like this at the level of internal library code (that 
> the user of the
> library will not see when using the library) is perfectly fine.

>From a cultural perspective (with python traditionally not being written in a 
>functional style)
I think writing code using functional-style error handling at the level of 
internal library code (that the user of the library will not see when using the 
library) is perfectly fine.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/L5WR6CEP4T4IRMT43CWFZEIUZDRJ3MNY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[sascha.schlemmer--- via Python-ideas]

> Alas, this still doesn't work because type annotations are advisory and 
> not mandatory. Regardless of the type hints given, I can still call 
> div(1, "surprise") and get a TypeError.

I'd assume that everyone is a consenting adult and if someone really wanted to 
shoot themselves in the foot there is no way to really prevent that (although 
mypy does a good job at that). The runtime `TypeError` at the call site of the 
offending code is actually what I'd prefer over code that mostly works (e.g. 
`div(1, 2)`) but crashes in the rare event that someone passes `b=0` to `div.`

> Worse, because types in Python use subtyping, I can pass an int that 
> does something you don't expect:
> class MyWeirdInt(int):
> def __truediv__(self, other):
> if other == 0:
> import loging
> logging.logg("tried to divide by " + other)
> raise ValueError('divided by zero')

Again there is no general way to prevent people from writing buggy code, the 
most can do is try and guide people towards using a library correctly (by good 
naming conventions, suitable levels of abstractions and yes also by making use 
of type hints)

> So your "total function" is still not 
> total: as soon as we move beyond the builtins written in C, literally 
> anything can happen and any promise you make in div can be only 
> advisory, not guaranteed.
> To get your "total function" you have to catch anything, but that cure 
> is worse than the disease. That means that obvious bugs get suppressed 
> and turned into Nothing() when they should in fact raise.

I fully agree with you that catching any possible exception just to make the 
compiler happy is the worst of all possible ways to handle errors. This is 
however not what I suggest; the point is not to make a function *total* in 
terms of anything that can happen in the interpreter/compile/runtime/hardware - 
that is what exceptions are for - but in terms of the functions api e.g. 
dividing by zero being a natural part of division that has to be handled 
somehow.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/J6TDJXDHSUNOJSL6SXMWRQWK34HK4YZS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[sascha.schlemmer--- via Python-ideas]

I think on this point at lot of people will have to agree to disagree. The 
clear advantage for me is that the `ZeroDivisionError` that is naturally part 
of `div` is made explicit and handled in a type safe way (given than one is 
using a type checker like mypy). Especially in this example I can't for the 
life of me think of a situation where I would not have a `try`/`except` 
directly surrounding the regular `div` with exceptions and instead let the 
exception bubble up the stack and have something like

```
try:
do_lots_of_arithmetic() # code that calls div and any other similar function
except EveryArithmeticErrorYouCanThinkOf: # very bad design imho.
print('ooops something went wrong')
``` 

>From a cultural perspective (with python traditionally not being written in 
>this style) I think writing code like this at the level of internal library 
>code (that the user of the library will not see when using the library) is 
>perfectly fine.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/NXUC2NFJR2XTVEJ4FKDKTLHDQLT4HXF5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Steven D'Aprano]

On Sat, Sep 26, 2020 at 09:59:27AM +0200, Sascha Schlemmer via Python-ideas 
wrote:

> I think a clear distinction has to be made between errors that belong 
> to the api surface of a function e.g. some `ValueError` or 
> `ZeroDivisionError` and *exceptional* errors e,g.`KeyboardInterrupt` 
> or `DatabaseExplodedError`.

One problem is that while we can make a distinction between "expected" 
errors and "unexpected" errors, it's subjective and anything but clear. 
Whether you handle it, or just let it go, depends in part on the 
software you are writing.


> For the first case I think a better way to model a function, that has 
> some *expected* failure mode is to try and write a *total* function 
> that doesn’t use exceptions at all. Let me show what I mean:
> 
> ```
> def div(a: int, b: int) -> float:
>   return a / b
> ```
> 
> This function tries to divide two integers, trouble is that `div` is 
> not defined for every input we may pass it, i.e. passing `b=0` will 
> lead to `ZeroDivisionError` being raised. In other words `div` is a 
> *partial* function because it is not defined for every member of its 
> domain (type of its arguments).

> There are two ways to amend this issue and make `div` a *total* 
> function that is honest about its domain and co-domain (types of input 
> arguments and return type):

You're making a mighty effort to bring some functional programming 
concepts to Python, but it doesn't really work. For example:

> 1. Extending the type of the functions co-domain (return type) and 
> making the error case explicit
[...]

> ```
> def div(a: int, b: int) -> Maybe[float]:
>   try:
>   return Some(a / b)
>   except ZeroDivisionError:
>   return Nothing()
> ```

Alas, this still doesn't work because type annotations are advisory and 
not mandatory. Regardless of the type hints given, I can still call 
`div(1, "surprise")` and get a TypeError.

Worse, because types in Python use subtyping, I can pass an int that 
does something you don't expect:

class MyWeirdInt(int):
def __truediv__(self, other):
if other == 0:
import loging
logging.logg("tried to divide by " + other)
raise ValueError('divided by zero')
...


I've intentionally included at least three bugs in that method. 
Obviously in practice only one is likely to occur at a time, but this 
illustrates that even if the arguments to your div function are ints, in 
this case a subclass of int, you might expect a ZeroDivisionError but 
actually get:

* ImportError  (`import loging`)
* AttributeError  (`logging.logg`)
* TypeError  (concatenate a string with a number)
* ValueError  (explicit raise)

or any other exception at all. So your "total function" is still not 
total: as soon as we move beyond the builtins written in C, literally 
anything can happen and any promise you make in `div` can be only 
advisory, not guaranteed.

To get your "total function" you have to catch *anything*, but that cure 
is worse than the disease. That means that obvious bugs get suppressed 
and turned into `Nothing()` when they should in fact raise.

And this brings us to where Java discovered that checked exceptions are 
counter-productive. To satisfy the compiler and have a "total function", 
anything which uses CheckedExceptions eventually ends up having the Java 
equivalent of:

# Declared to only raise SomeError
try:
actual implementation
except:
# Catch anything and everything.
raise SomeError("something bad happened")


because that's the only way to end up with a total function in your 
sense, given that subclasses can raise anything.

And that's why many of us are quite resistant to attempts to bring 
checked exceptions to Python.


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/UADOFVBPRMECPDLHQVRBOZVHGYQRC7O7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 10:47 PM Sascha Schlemmer via Python-ideas
 wrote:
> ```
> def div(a: int, b: int) -> Result[float, ZeroDivisionError]:
>   try:
>   return Success(a / b)
>   except ZeroDivisionError as error:
>   return Failure(error)
> ```
>
> Now `div` does return a valid instance of `Maybe` (or `Result` if more a more 
> detailed failure case is wanted) which is either something like  
> `Some(3.1415) ` or `Nothing` (analogous to `None` ). The caller of the 
> function then has to deal with this and mypy will correctly warn if the user 
> fails to do so properly e.g.
>

This is strictly worse than exception handling, because you're just
forcing every level to do a try/except. It's otherwise identical. Very
very bad idea to inflict this on Python programmers. Please do not
write any code like this in any function that you expect anyone else
to use.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/YVREVFRCLGVJM5MTCQGHFJZYQ23JRMGF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

On Sat, Sep 26, 2020 at 8:50 AM Sascha Schlemmer via Python-ideas <
python-ideas@python.org> wrote:

> I think a clear distinction has to be made between errors that belong to
> the api surface of a function e.g. some `ValueError` or `ZeroDivisionError`
> and *exceptional* errors e,g.`KeyboardInterrupt` or
> `DatabaseExplodedError`. For the latter case exceptions work perfectly well
> as they are and it is both unhelpful and infeasible to annotate/document
> every exception - just let the exception bubble up and someone will handle
> it (might even be a process supervisor like supervisord)
>
> For the first case I think a better way to model a function, that has some
> *expected* failure mode is to try and write a *total* function that doesn’t
> use exceptions at all. Let me show what I mean:
>
> ```
> def div(a: int, b: int) -> float:
>   return a / b
> ```
>
> This function tries to divide two integers, trouble is that `div` is not
> defined for every input we may pass it, i.e. passing `b=0` will lead to
> `ZeroDivisionError` being raised. In other words `div` is a *partial*
> function because it is not defined for every member of its domain (type of
> its arguments).
>
> There are two ways to amend this issue and make `div` a *total* function
> that is honest about its domain and co-domain (types of input arguments and
> return type):
>
> 1. Extending the type of the functions co-domain (return type) and making
> the error case explicit (see
> https://returns.readthedocs.io/en/latest/index.html for a library that
> implements this style of error handling)
>
> ```
> def div(a: int, b: int) -> Maybe[float]:
>   try:
>   return Some(a / b)
>   except ZeroDivisionError:
>   return Nothing()
> ```
>
> or
>
> ```
> def div(a: int, b: int) -> Result[float, ZeroDivisionError]:
>   try:
>   return Success(a / b)
>   except ZeroDivisionError as error:
>   return Failure(error)
> ```
>
> Now `div` does return a valid instance of `Maybe` (or `Result` if more a
> more detailed failure case is wanted) which is either something like
>  `Some(3.1415) ` or `Nothing` (analogous to `None` ). The caller of the
> function then has to deal with this and mypy will correctly warn if the
> user fails to do so properly e.g.
>
> ´´´
> div(1,1) + 1  # mypy will give a type error
> ´´´
>

https://en.wikipedia.org/wiki/Partial_function
https://en.wikipedia.org/wiki/Partial_application#Implementations
- https://docs.python.org/3/library/functools.html#functools.partial
- https://docs.python.org/3/library/functools.html#functools.singledispatch

PEP 0622: Structural Pattern Matching
https://www.python.org/dev/peps/pep-0622/


>
> 2. Restricting the functions domain to values with defined behavior
>
> ```
> def div(a: int, b: NonZeroInteger) -> float:
>   return a / b
> ```
>
> In this case the burden is put on the caller to supply valid inputs to the
> function and this effectively pushes the error handling/ input validation
> out of `div`
>

https://pypi.org/project/icontract/ :

```python
>>> import icontract
>>> @icontract.require(lambda x: x > 3)
... def some_func(x: int, y: int = 5)->None:
... pass
...
>>> some_func(x=5)
# Pre-condition violation
>>> some_func(x=1)
Traceback (most recent call last):
  ...
icontract.errors.ViolationError: File , line 1 in
:
x > 3: x was 1
```

There exist a couple of contract libraries. However, at the time of this
> writing (September 2018), they all required the programmer either to learn
> a new syntax (PyContracts) or to write redundant condition descriptions (
> e.g., contracts, covenant, dpcontracts, pyadbc and pcd).


https://andreacensi.github.io/contracts/ supports runtime argument and
return constraints as Python 3 annotations, with the @contract decorator,
and in docstrings:

```python
@contract
def my_function(a : 'int,>0', b : 'list[N],N>0') -> 'list[N]':
 # Requires b to be a nonempty list, and the return
 # value to have the same length.
 ...
```
icontract supports inheritance.

Defining the domain and range of a partial callable is relevant to this
discussion about callables that don't return a value or None, per se, but
instead raise Exceptions.

Results looks neat. I think that just reading about all of these tools
could make me a better programmer. What an ironic conflation of None with
null void pointer references here (see also Pandas' handling of
None/np.NaN) https://returns.readthedocs.io/en/latest/index.html#id1 :

> Or you can use Maybe container! It consists of Some and Nothing types,
representing existing state and empty (instead of None) state respectively.



>
> A language that does all this really well is F# (which like python is a
> multi-paradigm language that offers both object-oriented and
> functional-programming-oriented constructs). The trouble in python at the
> moment is that using something like `Maybe` and `Result` is not as nice to
> use as exceptions due to lack of a nice syntax i.e. for exceptions we have
> `try`/`except`

[Python-ideas] Re: Suggestion: annotated exceptions

[Stephen J. Turnbull]

On Fri, Sep 25, 2020 at 3:38 PM Steven D'Aprano  wrote:

 > The last thing I want to see is people being encouraged to write code
 > like this:
 >
 > def demo(arg)->Something:
[... elided ...]
 > except:
 > # Any other unexpected error.
 > raise ValueError('something unexpected')
 >
 > just to ensure that the declared exception is correct.

Chris Angelico follows up:

 > Another thing I don't want to see is:
[... example which only catches the subset named in raise elided ...]
 > where people feel they HAVE to catch everything that a function could
 > raise. (Which, btw, still ignores the fact that basically any code
 > could raise MemoryError, KeyboardInterrupt, etc.)

I'm -1 on this because I don't see how it could possibly be more
useful than a tool that greps all the class and def statements (to
identify functions) and all the raises (and presumably caches at least
the builtin and stdlib raise information), and then tells your IDE
about it.

But "encouraging people to write bad code" seems a bit unfair, and
probably invalidated in practice by the experience with mypy
annotations (and certainly mitigated by the fact that a lot of
annotations end up in stub files that only mypy users care much
about).

 > Only catch those exceptions that you can actually handle. Otherwise,
 > it's irrelevant what the function might raise.

This is a good rule.  It seems to me that this proposal is intended to
address the question: how do you know what subset of exceptions you
can handle (and care to do so) are raised?

This is why I don't see this proposal as useful to me.  It leaves it
up to the writer of the function to decide what exceptions interest
me.  Will the list be sufficiently complete from my point of view?
Will it be correctly updated in new versions?
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/R7QTUQCQQLAK27PSOAVXUUDNCORMM55Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sascha Schlemmer via Python-ideas]

I think a clear distinction has to be made between errors that belong to the 
api surface of a function e.g. some `ValueError` or `ZeroDivisionError` and 
*exceptional* errors e,g.`KeyboardInterrupt` or `DatabaseExplodedError`. For 
the latter case exceptions work perfectly well as they are and it is both 
unhelpful and infeasible to annotate/document every exception - just let the 
exception bubble up and someone will handle it (might even be a process 
supervisor like supervisord)

For the first case I think a better way to model a function, that has some 
*expected* failure mode is to try and write a *total* function that doesn’t use 
exceptions at all. Let me show what I mean:

```
def div(a: int, b: int) -> float:
  return a / b
```

This function tries to divide two integers, trouble is that `div` is not 
defined for every input we may pass it, i.e. passing `b=0` will lead to 
`ZeroDivisionError` being raised. In other words `div` is a *partial* function 
because it is not defined for every member of its domain (type of its 
arguments). 

There are two ways to amend this issue and make `div` a *total* function that 
is honest about its domain and co-domain (types of input arguments and return 
type):

1. Extending the type of the functions co-domain (return type) and making the 
error case explicit (see https://returns.readthedocs.io/en/latest/index.html 
 for a library that 
implements this style of error handling)

```
def div(a: int, b: int) -> Maybe[float]:
  try:
  return Some(a / b)
  except ZeroDivisionError:
  return Nothing()
```

or

```
def div(a: int, b: int) -> Result[float, ZeroDivisionError]:
  try:
  return Success(a / b)
  except ZeroDivisionError as error:
  return Failure(error)
```

Now `div` does return a valid instance of `Maybe` (or `Result` if more a more 
detailed failure case is wanted) which is either something like  `Some(3.1415) 
` or `Nothing` (analogous to `None` ). The caller of the function then has to 
deal with this and mypy will correctly warn if the user fails to do so properly 
e.g.

´´´
div(1,1) + 1  # mypy will give a type error
´´´

2. Restricting the functions domain to values with defined behavior

```
def div(a: int, b: NonZeroInteger) -> float:
  return a / b
```

In this case the burden is put on the caller to supply valid inputs to the 
function and this effectively pushes the error handling/ input validation out 
of `div`



A language that does all this really well is F# (which like python is a 
multi-paradigm language that offers both object-oriented and 
functional-programming-oriented constructs). The trouble in python at the 
moment is that using something like `Maybe` and `Result` is not as nice to use 
as exceptions due to lack of a nice syntax i.e. for exceptions we have 
`try`/`except` and for functional error handling we’d need pattern matching.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/KBBXERGVQ4YMBFGJL2TLGP6YWN3VR3AJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Greg Ewing]

On 26/09/20 7:26 am, Oscar Benjamin wrote:

The suggestion ... was not that it would be a
type-checking error to call the inverse function without catching the
exception. It only becomes a type-checking error if a function calls
the inverse function and claims not to raise the exception.


This isn't very coherent. We've agreed that the declared exceptions
is not an exhaustive list of the exceptions that may be raised. So
not declaring an exception is *not* claiming not to raise it!

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6GJ5ZW3K52SYUT3RDDMYAW6WQUSQ5SAM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[MRAB]

On 2020-09-26 01:23, Greg Ewing wrote:

On 26/09/20 4:32 am, Oscar Benjamin wrote:

annotations could be used to document in a statically
analysable way what the "expected" exceptions are. A type checker
could use those to check whether a caller is handling the *expected*
exceptions


But that would be inappropriate. Even if an exception is
"expected" (assuming for a moment we agree on what that means),
the immediate caller is *not* obliged to handle it. Just as
with any other exception, it's perfectly fine to let it propagate
up to a level where something sensible can be done with it.

Treating this as an error would be more annoying than helpful in
most situations, I think.

There are a few cases where it *might* make sense, such as
StopIteration, which is pretty much part of the function's API
and letting it escape is probably a mistake. But such cases
are very rare and probably not worth adding a new language
mechanism for.

An "expected" exception would be one that could be regarded as part of 
the interface/API, e.g. json.load raising JSONDecodeError on malformed 
JSON. Other exceptions could also be raised, if you, for example, you 
didn't pass in a file, or there's an I/O error, or you run out of memory.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TQBQMSHAZDM5YVI4HQGLVJUQXZ47QQS7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Greg Ewing]

On 26/09/20 5:27 am, Wes Turner wrote:

Safe coding styles (in other languages) do specify that *there may not 
be any unhandled exceptions*.


I don't think a blind rule like that actually makes a program any safer.
It encourages a coding style that covers up problems and carries on to
produce garbage, rather than bringing the problem to someone's
attention.

The Ariane V incident is sometimes cited as an example of what can
happen if you fail to handle exceptions. But "handling" the exception
in that case wouldn't have helped -- the program was being fed
invalid inputs, and whatever it did, some kind of disaster would have
occurred.

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ZT6F7RMTENC7KXOH4NUOK6K3N5MLO7M2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Steven D'Aprano]

On Fri, Sep 25, 2020 at 04:58:00PM -0400, Wes Turner wrote:
> On Fri, Sep 25, 2020 at 1:43 PM Steven D'Aprano  wrote:

> > Of course rare or unusual exceptions probably won't be discovered
> > without a lot of testing, including stress testing, or not until the
> > code goes out into production. That's okay.
> >
> 
> While there are plenty of ways to debug in production,
> debugging in production is a bad idea and is not allowed (because: __, __,
> __) :
> log the exception with necessary details (traceback, exception attrs,  stack frame>) but exclude sensitive information that shouldn't be leaking
> into the logging system.

I hear you about sensitive information leaking into the logging system. 
But production code is not necessarily used by a third party where such 
leaks are a concern.

In a perfect world, our code would be 100% bug-free when we release it 
into production. And we have some strategies aimed to help approach 
that: TDD, fuzzing, test coverage metrics, etc.

If we are writing software that could kill people if we get it wrong:

https://hackaday.com/2015/10/26/killed-by-a-machine-the-therac-25/

this level of care is justified.

But under normal circumstances, don't let the perfect get in the way 
of the good. "Good enough" depends on what the code does and the 
consequences of a failure.

https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good

In practice, we have a number of competing desires. We want the software 
for a reason, and we might not want to wait for it to be absolutely 100% 
bug free before actually using it in production. (Especially since we 
can never be sure that it is bug-free.)

So we have to expect some bugs will be reported from production. This is 
fine. We shouldn't let purists bully or shame us into thinking that this 
is necessarily the wrong thing. Sometimes "good enough, right now" is 
better than "perfect, in three years".



-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/T6A7EIBC4RTUPE4V33JELUBT5HKYNPUT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Steven D'Aprano]

On Fri, Sep 25, 2020 at 08:01:39PM +0100, Paul Moore wrote:
> On Fri, 25 Sep 2020 at 18:42, Steven D'Aprano  wrote:
> > There may be practical difficulties in sticking exceptions into
> > annotations. Annotations already can be pretty long and bulky. But if
> > you are okay with functions documenting that they might raise a certain
> > exception, then *in principle* you should be okay with moving that into
> > an annotation rather than the docstring.
> >
> > Annotations are just a form of documentation in a consistent standard
> > format to make it easy for IDEs to read them.
> 
> Expressed that way, it's a reasonable argument. And to the extent that
> we're talking about a way for functions to note what exceptions the
> developer feels are worth mentioning, having a language construct to
> do this would be OK.

Indeed. But the practical difficulties aren't trivial.

There's the issue of mere ackwardness: there's already a tonne of 
information in a complex function signature, adding annotations pretty 
much doubles that, and adding exceptions could easily increase that 
again by fifty percent.

More importantly, attempts in other languages to declare exceptions have 
been unsuccessful, if not counter-productive, for reasons that Chris 
explains.

On the other hand, for many years before mypy came on the 
scene, IDEs parsed docstrings for parameter information, with at least 
four different standards for declaring parameters:

- Epydoc
- ReST format
- Google's format
- Numpydoc format

https://stackoverflow.com/questions/3898572/what-is-the-standard-python-docstring-format

There may be others, and all of these seem to have support for declaring 
the raised exception types, so presumably IDEs have been able to make 
use of this information for a decade or more and the world hasn't ended.


> But in reality, people would treat the annotation as more definitive
> than it is. And people would feel pressure to "stick to the contract"
> when writing the annotated code. And style guides (which like it or
> not are sometimes written by people who are *not* experienced
> developers) will appear which state rules rather than allowing for
> nuanced interpretation. All of which might not be how the feature was
> intended to be used. I know, why assume the worst? But in contrast,
> what's the *best* outcome?

Yes, Chris does a good job of explaning the problems, and I'm not going 
to repeat them here.

And the benefit seems small: so your IDE will autocomplete the 
exceptions for you.

I think this may come down to this:

- people who use IDEs think that autocompletion in general is a
  great feature and may want want to extend that to exceptions;

- people who don't use IDEs think autocomplete is a very weak
  reason for adding a feature to the language.

My own personal feeling is "so I don't have to look up the docs" is a 
weak feature, especially given that an abbreviated version of the docs 
are available from the interactive interpreter:

help(some_function)


The bottom line is that I think any impetus for this feature would have 
to come from the mypy and other static tools community. Can they use it? 
I don't think it's far wrong to say that if Guido and the mypy devs want 
this, they will probably get it, and if they don't, you probably won't 
convince them to add it just for the sake of autocomplete.

-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7N5VR66P5AH4LHEMETVNHC5LI4D7CDZN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

Search upwards for what (if anything) is going to catch an Exception would
be a mighty useful static/dynamic analysis thing. Is there anything that
can do this?

I'd hazard to guess that *most* apps will just crash and expect the process
spawner (e.g. systemd, supervisord, not sysV init) to re-spawn the crashed
process after the unhandled fault?

https://github.com/analysis-tools-dev/static-analysis#python
https://github.com/analysis-tools-dev/dynamic-analysis#python
https://github.com/vinta/awesome-python#code-analysis

On Fri, Sep 25, 2020 at 8:35 PM Greg Ewing 
wrote:

> On 26/09/20 4:32 am, Oscar Benjamin wrote:
> > annotations could be used to document in a statically
> > analysable way what the "expected" exceptions are. A type checker
> > could use those to check whether a caller is handling the *expected*
> > exceptions
>
> But that would be inappropriate. Even if an exception is
> "expected" (assuming for a moment we agree on what that means),
> the immediate caller is *not* obliged to handle it. Just as
> with any other exception, it's perfectly fine to let it propagate
> up to a level where something sensible can be done with it.
>
> Treating this as an error would be more annoying than helpful in
> most situations, I think.
>
> There are a few cases where it *might* make sense, such as
> StopIteration, which is pretty much part of the function's API
> and letting it escape is probably a mistake. But such cases
> are very rare and probably not worth adding a new language
> mechanism for.
>
> --
> Greg
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/DYD26ANQUYTIAO4ABCJJGQIXN7LX77ID/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/H3TIFOAVG536N2QU4QTZ7F3V4UYC3LDN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Greg Ewing]

On 26/09/20 4:32 am, Oscar Benjamin wrote:

annotations could be used to document in a statically
analysable way what the "expected" exceptions are. A type checker
could use those to check whether a caller is handling the *expected*
exceptions


But that would be inappropriate. Even if an exception is
"expected" (assuming for a moment we agree on what that means),
the immediate caller is *not* obliged to handle it. Just as
with any other exception, it's perfectly fine to let it propagate
up to a level where something sensible can be done with it.

Treating this as an error would be more annoying than helpful in
most situations, I think.

There are a few cases where it *might* make sense, such as
StopIteration, which is pretty much part of the function's API
and letting it escape is probably a mistake. But such cases
are very rare and probably not worth adding a new language
mechanism for.

--
Greg
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DYD26ANQUYTIAO4ABCJJGQIXN7LX77ID/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

Depth-first-search for every callable, [property] descriptor, operator,
__getitem__

On Fri, Sep 25, 2020 at 4:36 PM Wes Turner  wrote:

>
> LSP servers and clients: https://langserver.org/
> - https://github.com/microsoft/python-language-server
>   -
> https://code.visualstudio.com/api/language-extensions/programmatic-language-features
>

What static and/or dynamic analysis tools can find all handled and
unhandled exceptions in every transitive {callable, [property] descriptor,
operator, __getitem__} given a callable?

def find_all_exceptions(callable):
"""
Args:
callable (callable): callable to traverse from in search of handled
and unhandled exceptions
Returns:
 list: nested lists of items [exception, [exception_locations],
[exception_handler_locations]] (?)
"""

What subset of these exceptions would a code annotation and/or docstring
happen to helpfully contain?

- AFAIU, there's not yet anything in any LSP Language Extensions for
> statically or dynamically discovering exceptions from the DFS of all
> references to types and callables of a callable
>
> (Type) Annotation type inference tools:
>
> pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram)
>> [3] all do dynamic / runtime PEP-484 type annotation type inference [4]
>> [1] https://github.com/google/pytype
>> [2] https://github.com/dropbox/pyannotate
>> [3] https://github.com/Instagram/MonkeyType
>> [4] https://news.ycombinator.com/item?id=19454411
>
>
I haven't realized that I had need for a tool that does this either; but I
guess I've effectively just manually searched for `raise` and `except` (and
failed to sufficiently fuzz with every possible combination of
user-supplied inputs)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GG7C246C2YGOBI4OMWBGO7DZYC5H7IYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 10:19 AM Wes Turner  wrote:
>
>
>
> On Fri, Sep 25, 2020 at 8:09 PM Chris Angelico  wrote:
>>
>> On Sat, Sep 26, 2020 at 8:17 AM Wes Turner  wrote:
>> >
>> > What a worthless semantic distinction.
>> >
>> > You don't want to be executing code to determine why an Exception occurred 
>> > because you do not trust support devs to access all of the data in the 
>> > production system.
>> >
>> > "Exceptions happen" is true, but that's not satisfactory in an 
>> > organization focused on quality.
>> >
>>
>>  wut?
>>
>> I don't understand what you mean here. Sometimes a traceback isn't all
>> the information, and you need to add code to something to figure out
>> the cause. Are you implying that, because I actually put effort into
>> tracking bugs down, I am clearly not focused on quality??
>>
>> I'll just go ahead and repeat myself: "what?"
>
>
> - The context of this tangent was ~"[Exceptions happen in production] and 
> that's okay".
>   - Looseness with exception handling is entirely unacceptable for 
> safety-critical and high availability applications.
> - *Interactive debugging* is fine for many non-production applications.
> - Yup, the process spawner is going to reap and restart when the process 
> fails due to an unhandled Exception,
>but data loss due to unhandled exceptions (and non-transactional data 
> persistence) is a real problem
>- "It's just going to raise whatever sometimes and you can just pdb it 
> every time"
> - You can add information to exceptions and preserve the traceback with 
> `raise _ from _`
>
> I have nothing further to add about this non-personal tangent.

Okay, I can see what's happening. You're assuming that "debugging"
means "interactive debugging in a specialized harness". I consider
"debugging" to be "anything done with a view to removing a bug". So by
my definition, debugging in production is normal, and by yours, it's
impossible/unwise. I think we're just arguing terminology here, and
either way, checked exceptions won't really help.

There's nothing further to debate - we just use the word "debugging"
differently. :)

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/A7W3CISZRXMAZTA23UCPS5GIE4FBQZXP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

On Fri, Sep 25, 2020 at 8:09 PM Chris Angelico  wrote:

> On Sat, Sep 26, 2020 at 8:17 AM Wes Turner  wrote:
> >
> > What a worthless semantic distinction.
> >
> > You don't want to be executing code to determine why an Exception
> occurred because you do not trust support devs to access all of the data in
> the production system.
> >
> > "Exceptions happen" is true, but that's not satisfactory in an
> organization focused on quality.
> >
>
>  wut?
>
> I don't understand what you mean here. Sometimes a traceback isn't all
> the information, and you need to add code to something to figure out
> the cause. Are you implying that, because I actually put effort into
> tracking bugs down, I am clearly not focused on quality??
>
> I'll just go ahead and repeat myself: "what?"
>

- The context of this tangent was ~"[Exceptions happen in production] and
that's okay".
  - Looseness with exception handling is entirely unacceptable for
safety-critical and high availability applications.
- *Interactive debugging* is fine for many non-production applications.
- Yup, the process spawner is going to reap and restart when the
process fails due to an unhandled Exception,
   but data loss due to unhandled exceptions (and non-transactional
data persistence) is a real problem
   - "It's just going to raise whatever sometimes and you can just pdb
it every time"
- You can add information to exceptions and preserve the traceback with
`raise _ from _`

I have nothing further to add about this non-personal tangent.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JSUBOGJLQTIIJWYMLP2DZFSKMM4AQYDG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 8:48 AM Oscar Benjamin
 wrote:
>
> On Fri, 25 Sep 2020 at 22:05, Chris Angelico  wrote:
> >
> > On Sat, Sep 26, 2020 at 6:56 AM Oscar Benjamin
> >  wrote:
> > >
> > > There are many common bugs that arise as a result of exceptions that
> > > are overlooked. There are also often simple ways to rewrite the code
> > > so that it becomes exception free. For example in sympy:
> > >
> > > if x.is_positive:  # never raises (excepting bugs) - False if 
> > > indeterminate
> > > if x > 0: # raises if indeterminate
> >
> > The novice believes that his ultimate goal is to stop the program from
> > crashing. If "exception free" is your goal, then sure, this feature
> > will certainly help you. I don't want Python to have language support
> > for something that is primarily aimed at helping novices to remain
> > forever novices.
>
> Yeah, because real professionals write applications that spew out
> tracebacks to their users for perfectly valid inputs and real users
> love that :)
>
> There are good reasons for writing your code in such a way that it
> handles all cases rather than blowing up. Of course if you can't
> handle all cases then that's one good reason for raising an exception.

That's something you do at a single location, usually - an error
boundary. There are points in the code where it's correct to absorb
and log all errors, but what that really means is that you're handling
Exception (or BaseException) - you still don't need to enumerate every
possible exception and you don't want to.

I still don't know of ANY situation in Python where I have wanted a
full list of every exception something might raise.

> The way that I would propose is this:
>
> Yes, g can possibly raise any exception class. As you say it's not
> generally possible to verify that arbitrary Python code can not raise
> any exception outside of a nontrivial set (BaseException).
>
> However g can not raise any exception from a function that was
> explicitly annotated as raising that exception. If some function h is
> annotated as raising FooError then a FooError from h will not arise
> from calling g.
>

I'm not sure I follow. Maybe some example code? Are you saying that,
if g calls h and h is declared as raising FooError, g must be declared
as raising FooError or must catch it, but any other exception is fair
game?

Very confused and still don't see any benefit here.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PCLEVTHFBKQEDNSPXDTIHAKITD6UVF6H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 8:17 AM Wes Turner  wrote:
>
> What a worthless semantic distinction.
>
> You don't want to be executing code to determine why an Exception occurred 
> because you do not trust support devs to access all of the data in the 
> production system.
>
> "Exceptions happen" is true, but that's not satisfactory in an organization 
> focused on quality.
>

 wut?

I don't understand what you mean here. Sometimes a traceback isn't all
the information, and you need to add code to something to figure out
the cause. Are you implying that, because I actually put effort into
tracking bugs down, I am clearly not focused on quality??

I'll just go ahead and repeat myself: "what?"

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XVNFVANNIJUBIHTYD7LJRRGH72X5OKAV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Oscar Benjamin]

On Fri, 25 Sep 2020 at 22:05, Chris Angelico  wrote:
>
> On Sat, Sep 26, 2020 at 6:56 AM Oscar Benjamin
>  wrote:
> >
> > There are many common bugs that arise as a result of exceptions that
> > are overlooked. There are also often simple ways to rewrite the code
> > so that it becomes exception free. For example in sympy:
> >
> > if x.is_positive:  # never raises (excepting bugs) - False if 
> > indeterminate
> > if x > 0: # raises if indeterminate
>
> The novice believes that his ultimate goal is to stop the program from
> crashing. If "exception free" is your goal, then sure, this feature
> will certainly help you. I don't want Python to have language support
> for something that is primarily aimed at helping novices to remain
> forever novices.

Yeah, because real professionals write applications that spew out
tracebacks to their users for perfectly valid inputs and real users
love that :)

There are good reasons for writing your code in such a way that it
handles all cases rather than blowing up. Of course if you can't
handle all cases then that's one good reason for raising an exception.
Raising an exception just because you wrote your code in slightly the
wrong way is obviously something to avoid though. A tool that helps
you to see non-obvious ways that exceptions can be raised can be very
useful.

> > That is *not* what I am suggesting. Rather only an explicit raises
> > hint could give an error so the above is fine. The type checker knows
> > that g might raise FooError but there's nothing wrong with raising an
> > error without any declaration. However the type checker *would*
> > complain about this:
> >
> > def f(): raises FooError
> > ...
> >
> > def g(): raises None
> > f()   # <--- typecheck error because g claims that it does not raise
> >
>
> Okay. So what exceptions can a "raises None" function raise? Or if it
> "raises FooError", then what, other than FooError, can it raise? If
> this is to have the blessing of the language, there needs to be an
> absolutely 100% consistent answer to that question. And there's only
> one valid answer that will always be correct, and it's the useless
> one: BaseException.

The way that I would propose is this:

Yes, g can possibly raise any exception class. As you say it's not
generally possible to verify that arbitrary Python code can not raise
any exception outside of a nontrivial set (BaseException).

However g can not raise any exception from a function that was
explicitly annotated as raising that exception. If some function h is
annotated as raising FooError then a FooError from h will not arise
from calling g.


Oscar
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/7O5ITX23PIFDM3L3CUKB657224GULXPA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

What a worthless semantic distinction.

You don't want to be executing code to determine why an Exception
occurred because you do not trust support devs to access all of the data in
the production system.

"Exceptions happen" is true, but that's not satisfactory in an organization
focused on quality.

On Fri, Sep 25, 2020 at 5:38 PM Chris Angelico  wrote:

> On Sat, Sep 26, 2020 at 7:25 AM Wes Turner  wrote:
> >
> > (Executing arbitrary code on a production server is debugging in
> production.
> >
> > Logging (additional information added to exceptions with 'raise _ from
> _`) may assist with root cause analysis and debugging on a different
> instance but IMHO logging is not debugging.)
> >
>
> Uhh... okay sure. What if you add instrumentation to the live
> server specifically so that it can log useful exceptions as you're
> trying to probe a bug that shows up only in prod and only once every
> two weeks? Is that debugging in prod, or is that logging?
>
> Or... does the distinction really even matter, and "debugging in prod"
> is just part of life?
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/THD4Y4RKERBSFUTU36EQHBEPDGTLIVQM/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/G3K55L2GFHP2R3PSTK5POM4E4BPZZH6P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 7:25 AM Wes Turner  wrote:
>
> (Executing arbitrary code on a production server is debugging in production.
>
> Logging (additional information added to exceptions with 'raise _ from _`) 
> may assist with root cause analysis and debugging on a different instance but 
> IMHO logging is not debugging.)
>

Uhh... okay sure. What if you add instrumentation to the live
server specifically so that it can log useful exceptions as you're
trying to probe a bug that shows up only in prod and only once every
two weeks? Is that debugging in prod, or is that logging?

Or... does the distinction really even matter, and "debugging in prod"
is just part of life?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/THD4Y4RKERBSFUTU36EQHBEPDGTLIVQM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

(Executing arbitrary code on a production server is debugging in production.

Logging (additional information added to exceptions with 'raise _ from _`)
may assist with root cause analysis and debugging on a different instance
but IMHO logging is not debugging.)

On Fri, Sep 25, 2020, 5:18 PM Chris Angelico  wrote:

> On Sat, Sep 26, 2020 at 7:05 AM Wes Turner  wrote:
> >> "Well, if I get an import error, I can add some more directories to
> >> sys.path and try adding the values again, that might fix it..."
> >>
> >> Who does that? Not me. And I bet you don't either.
> >
> >
> > "Defensive programming" / "Offensive programming"
> > https://en.wikipedia.org/wiki/Defensive_programming
>
> Do you plan for every possible exception from every possible line of
> code? Really? Okay, show me something that catches ImportError from
> "import math" then. What's in your except clause?
>
> > While there are plenty of ways to debug in production,
> > debugging in production is a bad idea and is not allowed (because: __,
> __, __) :
> > log the exception with necessary details (traceback, exception attrs,
> ) but exclude sensitive information that shouldn't be
> leaking into the logging system.
> >
>
> That IS debugging in production. I don't understand why you say that
> that's a bad thing.
>
> ChrisA
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/PYQ6KWUGIVRH4BSDGAA2JLBCVBSTLKSJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Y46CYIII6YMMHNLGCV4MTO3Q7OANMTVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 7:05 AM Wes Turner  wrote:
>> "Well, if I get an import error, I can add some more directories to
>> sys.path and try adding the values again, that might fix it..."
>>
>> Who does that? Not me. And I bet you don't either.
>
>
> "Defensive programming" / "Offensive programming"
> https://en.wikipedia.org/wiki/Defensive_programming

Do you plan for every possible exception from every possible line of
code? Really? Okay, show me something that catches ImportError from
"import math" then. What's in your except clause?

> While there are plenty of ways to debug in production,
> debugging in production is a bad idea and is not allowed (because: __, __, 
> __) :
> log the exception with necessary details (traceback, exception attrs,  stack frame>) but exclude sensitive information that shouldn't be leaking 
> into the logging system.
>

That IS debugging in production. I don't understand why you say that
that's a bad thing.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/PYQ6KWUGIVRH4BSDGAA2JLBCVBSTLKSJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 6:56 AM Oscar Benjamin
 wrote:
>
> There are many common bugs that arise as a result of exceptions that
> are overlooked. There are also often simple ways to rewrite the code
> so that it becomes exception free. For example in sympy:
>
> if x.is_positive:  # never raises (excepting bugs) - False if 
> indeterminate
> if x > 0: # raises if indeterminate

The novice believes that his ultimate goal is to stop the program from
crashing. If "exception free" is your goal, then sure, this feature
will certainly help you. I don't want Python to have language support
for something that is primarily aimed at helping novices to remain
forever novices.

> That is *not* what I am suggesting. Rather only an explicit raises
> hint could give an error so the above is fine. The type checker knows
> that g might raise FooError but there's nothing wrong with raising an
> error without any declaration. However the type checker *would*
> complain about this:
>
> def f(): raises FooError
> ...
>
> def g(): raises None
> f()   # <--- typecheck error because g claims that it does not raise
>

Okay. So what exceptions can a "raises None" function raise? Or if it
"raises FooError", then what, other than FooError, can it raise? If
this is to have the blessing of the language, there needs to be an
absolutely 100% consistent answer to that question. And there's only
one valid answer that will always be correct, and it's the useless
one: BaseException.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RYCEOPKWRLRGSWAE5BB5FRC7PMFBSF4K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

On Fri, Sep 25, 2020 at 1:43 PM Steven D'Aprano  wrote:

> On Fri, Sep 25, 2020 at 11:14:01PM +1000, Chris Angelico wrote:
> > On Fri, Sep 25, 2020 at 7:59 PM Sergio Fenoll  wrote:
>
> > > Surely there has to be a better way of programming than running stuff,
> > > watching it fail, and then keeping track of how it fails so you can
> > > later handle that failure?
> >
> > Why? Do you really think you can enumerate EVERY possible way that
> > something might fail?
>
> Nobody is demanding that "EVERY possible way" is handled -- but if you
> need to, then Python lets you do so:
>
> # Please don't do this.
> try:
> something()
> except:
> # handle EVERY error
>
> Of course this is nearly always the wrong choice. "What can I cope with"
> is easy: it's always *everything*, if you define "cope with" as just
> suppressing the error.
>
> > Think, instead, about all the possible problems that you can actually
> > cope with. That way, you have a finite - and usually small - set of
> > things to deal with, instead of an infinite field of "well this could
> > go wrong, but we can't do anything about that".
>
> The problem here is that you can't decide what you can deal with in
> isolation. I can deal with UnicodeEncodeError easily: try again with a
> different encoding, or with a different error handler, easy-peasy.
>
> But if I'm doing `y = x + 1` and it somehow raised UnicodeEncodeError,
> what do I do? I'm stuck.
>
> In order to tell what you can deal with, you need to know the
> circumstances of the error, and why it occurred. In other words, you
> need to understand the operation being called, in particular, what
> exceptions it might raise under normal circumstances.
>
> I'm pretty confident that most people, unless they are TDD zealots,
> start by using either their pre-existing knowledge of the operation, or
> reading the documentation, to find out what exceptions are likely under
> normal circumstances, and *only then* start to think about how to deal
> with such exceptions.
>
> The alternative is to waste time and mental energy thinking about how to
> deal with exceptions that you will probably never get in real life:
>
> "Well, if I get an import error, I can add some more directories to
> sys.path and try adding the values again, that might fix it..."
>
> Who does that? Not me. And I bet you don't either.
>

"Defensive programming" / "Offensive programming"
https://en.wikipedia.org/wiki/Defensive_programming

... Tacking additional information onto the exception and re-raising may be
the helpful thing to do; though that's still not handling the situation.

Recently I learned about the `raise _ from _` syntax (when writing an
example implementation for "[Python-ideas] f-strings as assignment targets":

"""
def cast_match_groupdict(matchobj, typemap):
matchdict = matchobj.groupdict()
if not typemap:
return matchdict
for attr, castfunc in typemap.items():
try:
matchdict[attr] = castfunc(matchdict[attr])
except ValueError as e:
raise ValueError(("attr", attr), ("rgx", matchobj.re)) from e
return matchdict
"""


>
> So I think that most of us:
>
> - start with documented or well-known exceptions;
>
> - and only then decide whether or not we can deal with them.
>
> Of course rare or unusual exceptions probably won't be discovered
> without a lot of testing, including stress testing, or not until the
> code goes out into production. That's okay.
>

While there are plenty of ways to debug in production,
debugging in production is a bad idea and is not allowed (because: __, __,
__) :
log the exception with necessary details (traceback, exception attrs, ) but exclude sensitive information that shouldn't be leaking
into the logging system.

Catching exceptions early is easier when:
- TDD / test coverage are emphasized
- fuzzing is incorporated into the release process (fuzzing is easier with
parameterized test cases)
- unit/functional/integration testing in a copy of production (sufficient
DevOps/DevSecOps)

- the coding safety guide says that all exceptions must be handled


> And I think that is Sergio's point: it would be good to have a standard,
> consistent place for functions to document which exceptions they are
> likely to raise under normal circumstances, and one which is available
> to IDEs and runtime inspection. Annotations.
>

Annotation: (type)
Docstring: (type, docstr)


>
> Of course we can inspect the docstring of the function, but it's hard
> for an automated tool to distinguish:
>
> This will raise WidgetExplosionError if the widget explodes.
>
> from:
>
> This is guaranteed to never raise WidgetExplosionError even
> if the widget explodes.
>
>
> There may be practical difficulties in sticking exceptions into
> annotations. Annotations already can be pretty long and bulky. But if
> you are okay with functions documenting that they might raise a certain
> exception, then *in principle* you should b

[Python-ideas] Re: Suggestion: annotated exceptions

[Oscar Benjamin]

On Fri, 25 Sep 2020 at 20:36, Chris Angelico  wrote:
>
> On Sat, Sep 26, 2020 at 5:27 AM Oscar Benjamin
>  wrote:
> >
> > On Fri, 25 Sep 2020 at 18:59, Chris Angelico  wrote:
> > >
> > > On Sat, Sep 26, 2020 at 2:32 AM Oscar Benjamin
> > >  wrote:
> > > > I do agree but maybe that suggests a different role for annotated
> > > > exceptions in Python. Rather than attempting to enumerate all possible
> > > > exceptions annotations could be used to document in a statically
> > > > analysable way what the "expected" exceptions are. A type checker
> > > > could use those to check whether a caller is handling the *expected*
> > > > exceptions rather than to verify that the list of *all* exceptions
> > > > possibly raised is exhaustive.
> > > >
> > > > Consider an example:
> > > >
> > > > def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
> > > > if determinant(M) == 0:
> > > > raise NotInvertibleError
> > > > rows, cols = M.shape
> > > > for i in range(rows):
> > > > for j in range(cols):
> > > > ...
> > > >
> > > > Here the function is expected to raise NotInvertibleError for some
> > > > inputs.
> > >
> > > Please no. Once again, this will encourage "handling" of errors by
> > > suppressing them. Don't have anything that forces people to handle
> > > exceptions they can't handle.
> >
> > Why would anyone be forced to "handle" the exceptions?
> >
> > The suggestion (which you've snipped away) was not that it would be a
> > type-checking error to call the inverse function without catching the
> > exception. It only becomes a type-checking error if a function calls
> > the inverse function and claims not to raise the exception. The
> > checker just verifies the consistency of the different claims about
> > what is raised.
> >
>
> Forcing people to declare it is exactly the same as forcing them to
> handle it every other way. There are many MANY bad ways this can go,
> most of which have been mentioned already:

I wouldn't suggest that a declaration be forced but rather that a
false declaration be disallowed. More below...

> 1) Catching exceptions and doing nothing, just so your linter shuts up
> 2) Catching exceptions and raising a generic "something went wrong"
> error so that's the only thing you have to declare
> 3) Declaring that you could raise BaseException, thus adding useless
> and meaningless boilerplate to your functions
> 4) Redeclaring every exception all the way up the chain, and having to
> edit them every time something changes
> 5) Copying and pasting a gigantic list of exceptions onto every
> function's signature

I'm sure those things would happen sometimes. At least all of these
things are very visible in the diff so a reviewer can at least see
what's going on. The tricky thing with exceptions is the action at a
distance effect so that it's not always obvious what the effect of a
change is so when you see a diff like:

   -   x = f()
   +  x = g()

you would potentially need to trawl a lot of code to see how that
affects what exceptions might be raised.

> And what are the possible *good* ways to use this? What do you
> actually gain? So far, the only use-case I've heard is "IDEs might
> help you tab-complete an except clause". That's only going to be
> useful if the exception tracking is accurate all the way up and down
> the tree, and that's only going to happen if it's done by analysis,
> NOT by forcing people to declare them.

I can tell you what I would use a feature like this for (if it is
implemented in a useful way) which is cleaning up the exception
handling in the sympy codebase. It's a large codebase with thousands
of raises and excepts:

$ git grep raise | wc -l
8186
$ git grep except | wc -l
1657

There are many common bugs that arise as a result of exceptions that
are overlooked. There are also often simple ways to rewrite the code
so that it becomes exception free. For example in sympy:

if x.is_positive:  # never raises (excepting bugs) - False if indeterminate
if x > 0: # raises if indeterminate

That distinction is not understood by many contributors to sympy but
is easy to explain during code review. That's an easy example but
others are not so easy.

> I've worked with Java's checked exceptions, which is what you're
> getting at here. They're terrible. The best way to handle them is to
> ensure that all your exceptions are RuntimeErrors so they don't get
> checked by the compiler.

My understanding of the situation in Java although I don't have much
experience of it as that checked exceptions must *always* be declared
so e.g. (in hypothetical Python syntax):

def f(): raises FooError
...

def g():
f()   # <--- typecheck error because g does not declare FooError

That is *not* what I am suggesting. Rather only an explicit raises
hint could give an error so the above is fine. The type checker knows
that g might raise FooError but there's nothing wrong with raising an
error without any declaration

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

On Fri, Sep 25, 2020 at 3:03 PM Paul Moore  wrote:

> On Fri, 25 Sep 2020 at 18:42, Steven D'Aprano  wrote:
> > There may be practical difficulties in sticking exceptions into
> > annotations. Annotations already can be pretty long and bulky. But if
> > you are okay with functions documenting that they might raise a certain
> > exception, then *in principle* you should be okay with moving that into
> > an annotation rather than the docstring.
> >
> > Annotations are just a form of documentation in a consistent standard
> > format to make it easy for IDEs to read them.
>
> Expressed that way, it's a reasonable argument. And to the extent that
> we're talking about a way for functions to note what exceptions the
> developer feels are worth mentioning, having a language construct to
> do this would be OK.
>
> But in reality, people would treat the annotation as more definitive
> than it is. And people would feel pressure to "stick to the contract"
> when writing the annotated code. And style guides (which like it or
> not are sometimes written by people who are *not* experienced
> developers) will appear which state rules rather than allowing for
> nuanced interpretation. All of which might not be how the feature was
> intended to be used. I know, why assume the worst? But in contrast,
> what's the *best* outcome?
>

Exception docstrings contain the Exception type and the explanation str:
(type, docstr)

Exception annotations as proposed contain the Exception type: (type)
and presumably then docstrings would also be expected to contain (type,
docstr);
which isn't DRY.

But annotations are parseable and more commonly linted.


>
> My problem with the proposal is that it's about adding a mechanism,
> without any clear statement of how that mechanism is intended to be
> used. And there's so much scope for misguided use, and so few clear
> benefits, that it feels like a bad trade-off at the moment. (For the
> record, I'm not convinced by the IDE argument. I use Vim, which is
> relentlessly primitive in its basic form, and wouldn't use anything
> like this, and VS Code, which is so clever that it seems like magic
> most of the time, and I'm pretty certain that if people wanted it, VS
> Code could include "exception autocompletion" even without explicit
> annotation like this).
>

LSP servers and clients: https://langserver.org/
- https://github.com/microsoft/python-language-server
  -
https://code.visualstudio.com/api/language-extensions/programmatic-language-features
- AFAIU, there's not yet anything in any LSP Language Extensions for
statically or dynamically discovering exceptions from the DFS of all
references to types and callables of a callable

(Type) Annotation type inference tools:

pytype (Google) [1], PyAnnotate (Dropbox) [2], and MonkeyType (Instagram)
> [3] all do dynamic / runtime PEP-484 type annotation type inference [4]
> [1] https://github.com/google/pytype
> [2] https://github.com/dropbox/pyannotate
> [3] https://github.com/Instagram/MonkeyType
> [4] https://news.ycombinator.com/item?id=19454411
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/DBYCXFI3ZQDNHZJGQ6BV7QKX7STHWUDQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 5:27 AM Oscar Benjamin
 wrote:
>
> On Fri, 25 Sep 2020 at 18:59, Chris Angelico  wrote:
> >
> > On Sat, Sep 26, 2020 at 2:32 AM Oscar Benjamin
> >  wrote:
> > > I do agree but maybe that suggests a different role for annotated
> > > exceptions in Python. Rather than attempting to enumerate all possible
> > > exceptions annotations could be used to document in a statically
> > > analysable way what the "expected" exceptions are. A type checker
> > > could use those to check whether a caller is handling the *expected*
> > > exceptions rather than to verify that the list of *all* exceptions
> > > possibly raised is exhaustive.
> > >
> > > Consider an example:
> > >
> > > def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
> > > if determinant(M) == 0:
> > > raise NotInvertibleError
> > > rows, cols = M.shape
> > > for i in range(rows):
> > > for j in range(cols):
> > > ...
> > >
> > > Here the function is expected to raise NotInvertibleError for some
> > > inputs.
> >
> > Please no. Once again, this will encourage "handling" of errors by
> > suppressing them. Don't have anything that forces people to handle
> > exceptions they can't handle.
>
> Why would anyone be forced to "handle" the exceptions?
>
> The suggestion (which you've snipped away) was not that it would be a
> type-checking error to call the inverse function without catching the
> exception. It only becomes a type-checking error if a function calls
> the inverse function and claims not to raise the exception. The
> checker just verifies the consistency of the different claims about
> what is raised.
>

Forcing people to declare it is exactly the same as forcing them to
handle it every other way. There are many MANY bad ways this can go,
most of which have been mentioned already:

1) Catching exceptions and doing nothing, just so your linter shuts up
2) Catching exceptions and raising a generic "something went wrong"
error so that's the only thing you have to declare
3) Declaring that you could raise BaseException, thus adding useless
and meaningless boilerplate to your functions
4) Redeclaring every exception all the way up the chain, and having to
edit them every time something changes
5) Copying and pasting a gigantic list of exceptions onto every
function's signature

And what are the possible *good* ways to use this? What do you
actually gain? So far, the only use-case I've heard is "IDEs might
help you tab-complete an except clause". That's only going to be
useful if the exception tracking is accurate all the way up and down
the tree, and that's only going to happen if it's done by analysis,
NOT by forcing people to declare them.

I've worked with Java's checked exceptions, which is what you're
getting at here. They're terrible. The best way to handle them is to
ensure that all your exceptions are RuntimeErrors so they don't get
checked by the compiler.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SVTC3HU5NXX62FII37WI5BVGPHQUOO32/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Oscar Benjamin]

On Fri, 25 Sep 2020 at 18:59, Chris Angelico  wrote:
>
> On Sat, Sep 26, 2020 at 2:32 AM Oscar Benjamin
>  wrote:
> > I do agree but maybe that suggests a different role for annotated
> > exceptions in Python. Rather than attempting to enumerate all possible
> > exceptions annotations could be used to document in a statically
> > analysable way what the "expected" exceptions are. A type checker
> > could use those to check whether a caller is handling the *expected*
> > exceptions rather than to verify that the list of *all* exceptions
> > possibly raised is exhaustive.
> >
> > Consider an example:
> >
> > def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
> > if determinant(M) == 0:
> > raise NotInvertibleError
> > rows, cols = M.shape
> > for i in range(rows):
> > for j in range(cols):
> > ...
> >
> > Here the function is expected to raise NotInvertibleError for some
> > inputs.
>
> Please no. Once again, this will encourage "handling" of errors by
> suppressing them. Don't have anything that forces people to handle
> exceptions they can't handle.

Why would anyone be forced to "handle" the exceptions?

The suggestion (which you've snipped away) was not that it would be a
type-checking error to call the inverse function without catching the
exception. It only becomes a type-checking error if a function calls
the inverse function and claims not to raise the exception. The
checker just verifies the consistency of the different claims about
what is raised.


Oscar
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TPGC6DJUSOHJN32UFYYRJRS53CZFXFPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Paul Moore]

On Fri, 25 Sep 2020 at 18:42, Steven D'Aprano  wrote:
> There may be practical difficulties in sticking exceptions into
> annotations. Annotations already can be pretty long and bulky. But if
> you are okay with functions documenting that they might raise a certain
> exception, then *in principle* you should be okay with moving that into
> an annotation rather than the docstring.
>
> Annotations are just a form of documentation in a consistent standard
> format to make it easy for IDEs to read them.

Expressed that way, it's a reasonable argument. And to the extent that
we're talking about a way for functions to note what exceptions the
developer feels are worth mentioning, having a language construct to
do this would be OK.

But in reality, people would treat the annotation as more definitive
than it is. And people would feel pressure to "stick to the contract"
when writing the annotated code. And style guides (which like it or
not are sometimes written by people who are *not* experienced
developers) will appear which state rules rather than allowing for
nuanced interpretation. All of which might not be how the feature was
intended to be used. I know, why assume the worst? But in contrast,
what's the *best* outcome?

My problem with the proposal is that it's about adding a mechanism,
without any clear statement of how that mechanism is intended to be
used. And there's so much scope for misguided use, and so few clear
benefits, that it feels like a bad trade-off at the moment. (For the
record, I'm not convinced by the IDE argument. I use Vim, which is
relentlessly primitive in its basic form, and wouldn't use anything
like this, and VS Code, which is so clever that it seems like magic
most of the time, and I'm pretty certain that if people wanted it, VS
Code could include "exception autocompletion" even without explicit
annotation like this).

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LT6LFEHTUZY7SMFHEQU3K25ZQJIEUBIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Paul Moore]

On Fri, 25 Sep 2020 at 18:28, Samuel Colvin  wrote:
>> There's also the problem that you've explicitly acknowledged, that
>> exception hints are *always* going to be inaccurate, in stark contrast
>> to type hints which are expected to be correct when used.
>
> We should draw a distinction between two kinds of "inaccurate":
> * inaccuracies because data about what errors could be raised is
>   missing some exceptions
> * errors that could happen at almost any time:
>   MemoryError, OverflowError, SystemError, SystemExit
>
> The first set of errors is not in stark contrast to type hints,
> in fact they're very similar to errors in type hints which come up
> from time to time.

I get what you are saying, but I think you're being optimistic in
assuming there's a clean distinction. It may be that you;'re thinking
mostly about application code, where you are in control of what types
exist in the system. But in library code, you simply cannot assume
that. Consider a library that manipulates strings. There have been
many attempts to write libraries that represent filesystem paths using
a subclass of the string type. Suppose your library was passed one of
those, and concatenated some text onto it. Possible exceptions that
could in theory be raised include:

* UnicodeError (if the path class checks that the value can be encoded
in the filesystem encoding)
* OSError (if the class enforces maximum path length or allowed
character restrictions)
* ValueError (if the programmer was lazy and used these rather than
one of the above)

as well as pretty much anything else, if the code that implemented
__add__ for the path class had a bug.

And of course, if libraries don't declare "expected" exceptions,
application code using those libraries don't have anything to work
with. Maybe you could get away with just declaring use of exception
types that "you" created. But that's going to be very messy to get
right.

> Granted the "exceptions raised" database would be much more
> immature than the type hints database to start with. But that's
> not a reason not to start the error.

Agreed. What *is* a reason to not start is if we can't even agree what
we're building.

> The second set of exceptions are a different matter, I think
> in general they should not be included in the
> "exceptions raised" database. If that makes exceptions
> raised data invalid, then virtually all python logic in the world
> that catches some exceptions but doesn't catch those
> exceptions is also invalid!

Precisely. So it's critical to clearly define exactly what exceptions
can happen "at any time", and given that operations like indexing,
basic operators, and even attribute access can call arbitrary Python
code, essentially any exception can happen at any time. Unless you
start mandating exception behaviour on basic operations, and then you
preclude creative but unusual designs such as the path class I
described above. Maybe that's not unreasonable (over-clever path
classes never really became that popular) but Python has a tradition
of *not* arbitrarily constraining how features can be used.

> In other words: right now when we write code and catch
> exceptions we generally (but not always) catch
> exceptions that are likely to happen in the general
> course of calling the code in the try, except block. But
> not all the strange things that could conceivably happen.
>
> I think we should maintain that convention but provide a
> way to remind developers when they forget to catch an exception.

How do you distinguish between a developer "forgetting" to catch an
exception and deliberately letting it bubble up. I'm a strong -1 on
any proposal that makes people explicitly say "I meant to let this
through" (it's essentially leading to checked exceptions, as well as
violating the principle I stated above about not constraining how
people use features - in this case the feature of exceptions bubbling
up). And it's obviously impossible to make people declare when they
forgot to catch an exception.

> I guess overall, the question is: would it be useful to have a
> way to check what exceptions a block of code might raise?
>
> Yes, massively.

You state that without any justification, and it's the key point. My
view is that it's (barely) possible that it might be occasionally
convenient, but no-one has really demonstrated a benefit that I care
about at this point. And to be honest, claims like "tools might be
able to warn about certain types of error" are too vague to carry much
weight. I've been developing Python for many years, and work on pip,
which is a major codebase that's full of legacy weirdness - and I
don't think I've *ever* hit an issue where I've felt that having a
checker warn me that I haven't caught a type of exception would be
useful. (Many, many times I've had to fix issues where uncaught
exceptions were raised, but they have basically always been logic
errors where fixing the underlying issue is the correct action, not
catching the exception.

A

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 3:42 AM Steven D'Aprano  wrote:
> > In the list of all possible failures, will you include MemoryError?
>
> Sure, if I'm writing some sort of ultra-high availability server
> application that is expected to run 24/7 for months at a time.
>
> while memory_in_reserve():
> try:
> something()
> break
> except MemoryError:
> release_rainy_day_fund()  # Frees some emergency memory.

But if you DON'T have code like that, will you declare that your
function might raise MemoryError?

Just because it might be raised doesn't mean it's part of what needs
to be documented. Nor because it might be caught.

The problem with a proposal like this is that it is impossible to
determine which exceptions make sense to document. My alarm script
attempts to query upcoming events, and it catches ssl.SSLError,
OSError, IOError, socket.error, and googleapiclient.http.HttpError.
Should my upcoming_events function (a call to which is the only line
in the 'try' block) be declared as raising all of those? Are there any
other exceptions that I should declare? How would I know what set of
exceptions is relevant to my caller? Only my caller knows what's
relevant, so that's where the list of exceptions belongs: in the
try/except block itself.

Oh, I just discovered that that function can raise OverflowError if
the script is run with --days=9 - should I
add that to the annotated list of exceptions? Is it relevant? How
would I know?

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MSI5KHXWBLMMG5BUSM2JIHJIJUJIN7O7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Sat, Sep 26, 2020 at 2:32 AM Oscar Benjamin
 wrote:
> I do agree but maybe that suggests a different role for annotated
> exceptions in Python. Rather than attempting to enumerate all possible
> exceptions annotations could be used to document in a statically
> analysable way what the "expected" exceptions are. A type checker
> could use those to check whether a caller is handling the *expected*
> exceptions rather than to verify that the list of *all* exceptions
> possibly raised is exhaustive.
>
> Consider an example:
>
> def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
> if determinant(M) == 0:
> raise NotInvertibleError
> rows, cols = M.shape
> for i in range(rows):
> for j in range(cols):
> ...
>
> Here the function is expected to raise NotInvertibleError for some
> inputs.

Please no. Once again, this will encourage "handling" of errors by
suppressing them. Don't have anything that forces people to handle
exceptions they can't handle.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2CHBAJGWGO7S6P4CKWYG7ZZLTBZB2W35/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Steven D'Aprano]

On Fri, Sep 25, 2020 at 11:14:01PM +1000, Chris Angelico wrote:
> On Fri, Sep 25, 2020 at 7:59 PM Sergio Fenoll  wrote:

> > Surely there has to be a better way of programming than running stuff,
> > watching it fail, and then keeping track of how it fails so you can
> > later handle that failure?
> 
> Why? Do you really think you can enumerate EVERY possible way that
> something might fail?

Nobody is demanding that "EVERY possible way" is handled -- but if you 
need to, then Python lets you do so:

# Please don't do this.
try:
something()
except:
# handle EVERY error

Of course this is nearly always the wrong choice. "What can I cope with" 
is easy: it's always *everything*, if you define "cope with" as just 
suppressing the error.

> Think, instead, about all the possible problems that you can actually
> cope with. That way, you have a finite - and usually small - set of
> things to deal with, instead of an infinite field of "well this could
> go wrong, but we can't do anything about that".

The problem here is that you can't decide what you can deal with in 
isolation. I can deal with UnicodeEncodeError easily: try again with a 
different encoding, or with a different error handler, easy-peasy.

But if I'm doing `y = x + 1` and it somehow raised UnicodeEncodeError, 
what do I do? I'm stuck.

In order to tell what you can deal with, you need to know the 
circumstances of the error, and why it occurred. In other words, you 
need to understand the operation being called, in particular, what 
exceptions it might raise under normal circumstances.

I'm pretty confident that most people, unless they are TDD zealots, 
start by using either their pre-existing knowledge of the operation, or 
reading the documentation, to find out what exceptions are likely under 
normal circumstances, and *only then* start to think about how to deal 
with such exceptions.

The alternative is to waste time and mental energy thinking about how to 
deal with exceptions that you will probably never get in real life:

"Well, if I get an import error, I can add some more directories to 
sys.path and try adding the values again, that might fix it..."

Who does that? Not me. And I bet you don't either.

So I think that most of us:

- start with documented or well-known exceptions;

- and only then decide whether or not we can deal with them.

Of course rare or unusual exceptions probably won't be discovered 
without a lot of testing, including stress testing, or not until the 
code goes out into production. That's okay.

And I think that is Sergio's point: it would be good to have a standard, 
consistent place for functions to document which exceptions they are 
likely to raise under normal circumstances, and one which is available 
to IDEs and runtime inspection. Annotations.

Of course we can inspect the docstring of the function, but it's hard 
for an automated tool to distinguish:

This will raise WidgetExplosionError if the widget explodes.

from:

This is guaranteed to never raise WidgetExplosionError even
if the widget explodes.


There may be practical difficulties in sticking exceptions into 
annotations. Annotations already can be pretty long and bulky. But if 
you are okay with functions documenting that they might raise a certain 
exception, then *in principle* you should be okay with moving that into 
an annotation rather than the docstring.

Annotations are just a form of documentation in a consistent standard 
format to make it easy for IDEs to read them.


> In the list of all possible failures, will you include MemoryError?

Sure, if I'm writing some sort of ultra-high availability server 
application that is expected to run 24/7 for months at a time.

while memory_in_reserve():
try:
something()
break
except MemoryError:
release_rainy_day_fund()  # Frees some emergency memory.

Or perhaps I'll catch the exception and try to bail out safely after 
writing out data files etc. Or maybe:

try:
fast_but_fat()
except MemoryError:
slow_but_lean()


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CBS5XGTLH7567TDLFFNBPCI7R2IWLGKT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Wes Turner]

I strongly disagree that it's useless to document which Exceptions a
function could raise; even in Python (which, for a few reasons, is not a
language that's considered for safety-critical application).

In Python, it is common practice to - at a high level in the call stack -
trap Exceptions that can occur anywhere like KeyboardInterrupt and
MemoryError (and separately specify signal handler callbacks).

A high-level catchall (except for KeyboardInterrupt) and restart may be the
best way to handle exceptions in Python.

Safe coding styles (in other languages) do specify that *there may not be
any unhandled exceptions*.
Other languages made the specific decision to omit exceptions entirely:
developers should return `retval, err := func(arg)` and handle every value
of err.
Python has Exceptions and it's helpful to document what exceptions a
function might `raise` (even though it is possible to parse the AST to find
the `raise` statements within a callable and any callables it may or may
not handle). There are a few useful ideas for checking Exception
annotations at compile-time in this thread.

https://en.wikipedia.org/wiki/Exception_handling#Static_checking_of_exceptions
https://en.wikipedia.org/wiki/Exception_handling#Dynamic_checking_of_exceptions

We could pick one or more of the software safety standards listed here and
quote and cite our favs:
https://awesome-safety-critical.readthedocs.io/en/latest/#software-safety-standards

## Exception docstrings
You can specify Exceptions in all formats of sphinx docstrings:

### Sphinx-style docstrings:

```python
"""
:raises: AttributeError: The ``Raises`` section is a list of all exceptions
that are relevant to the interface.
:raises: ValueError: If `param2` is equal to `param1`.
"""
```

### Google-style docstrings:
```python
"""
Raises:
AttributeError: The ``Raises`` section is a list of all exceptions
that are relevant to the interface.
ValueError: If `param2` is equal to `param1`.
"""
```

###Numpy-style docstrings:
```python
"""
Raises
--
AttributeError
The ``Raises`` section is a list of all exceptions
that are relevant to the interface.
ValueError
If `param2` is equal to `param1`.
"""
```

https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html#example-google
https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy

## Design-by-contracts
FWICS, neither e.g. icontract nor zope.interface support Exception
contracts. How could that work.


## Awesome-safety-critical
https://awesome-safety-critical.readthedocs.io/en/latest/#software-safety-standards


On Fri, Sep 25, 2020 at 12:34 PM Oscar Benjamin 
wrote:

> On Fri, 25 Sep 2020 at 15:57, Paul Moore  wrote:
> >
> > On Fri, 25 Sep 2020 at 14:15, Chris Angelico  wrote:
> >
> > > Why? Do you really think you can enumerate EVERY possible way that
> > > something might fail?
> >
> > Rust does a surprisingly good job of that, actually. But the point is
> > that Python is not Rust, and the infrastructure Rust uses to allow it
> > to manage code safety is baked into the language core at a very
> > fundamental level.
> >
> > Enumerating the exceptions that a piece of code can raise is
> > impractical and unhelpful in Python. But that may not be immediately
> > obvious to someone coming from a different language. That's why it's
> > important to understand Python properly before proposing new features
> > that work in other languages. (I don't think that's what the OP is
> > doing here, to be clear, but the discussion is drifting in that
> > direction, with Rust's Result type having been mentioned).
> >
> > **In Python**, writing code from the perspective of "what can I handle
> > at this point" is the right approach. Deferring unexpected exceptions
> > to your caller is the default behaviour, and results in a clean,
> > natural style *for Python*. The proposal here is basically in direct
> > contradiction to that style.
>
> I do agree but maybe that suggests a different role for annotated
> exceptions in Python. Rather than attempting to enumerate all possible
> exceptions annotations could be used to document in a statically
> analysable way what the "expected" exceptions are. A type checker
> could use those to check whether a caller is handling the *expected*
> exceptions rather than to verify that the list of *all* exceptions
> possibly raised is exhaustive.
>
> Consider an example:
>
> def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
> if determinant(M) == 0:
> raise NotInvertibleError
> rows, cols = M.shape
> for i in range(rows):
> for j in range(cols):
> ...
>
> Here the function is expected to raise NotInvertibleError for some
> inputs. It is also possible that the subsequent code could raise an
> exception e.g. AttributeError, TypeError etc and it's not necessarily
> possible to enumerate or exhaustively rule out wha

[Python-ideas] Re: Suggestion: annotated exceptions

[Oscar Benjamin]

On Fri, 25 Sep 2020 at 15:57, Paul Moore  wrote:
>
> On Fri, 25 Sep 2020 at 14:15, Chris Angelico  wrote:
>
> > Why? Do you really think you can enumerate EVERY possible way that
> > something might fail?
>
> Rust does a surprisingly good job of that, actually. But the point is
> that Python is not Rust, and the infrastructure Rust uses to allow it
> to manage code safety is baked into the language core at a very
> fundamental level.
>
> Enumerating the exceptions that a piece of code can raise is
> impractical and unhelpful in Python. But that may not be immediately
> obvious to someone coming from a different language. That's why it's
> important to understand Python properly before proposing new features
> that work in other languages. (I don't think that's what the OP is
> doing here, to be clear, but the discussion is drifting in that
> direction, with Rust's Result type having been mentioned).
>
> **In Python**, writing code from the perspective of "what can I handle
> at this point" is the right approach. Deferring unexpected exceptions
> to your caller is the default behaviour, and results in a clean,
> natural style *for Python*. The proposal here is basically in direct
> contradiction to that style.

I do agree but maybe that suggests a different role for annotated
exceptions in Python. Rather than attempting to enumerate all possible
exceptions annotations could be used to document in a statically
analysable way what the "expected" exceptions are. A type checker
could use those to check whether a caller is handling the *expected*
exceptions rather than to verify that the list of *all* exceptions
possibly raised is exhaustive.

Consider an example:

def inverse(M: Matrix) -> Matrix: raises(NotInvertibleError)
if determinant(M) == 0:
raise NotInvertibleError
rows, cols = M.shape
for i in range(rows):
for j in range(cols):
...

Here the function is expected to raise NotInvertibleError for some
inputs. It is also possible that the subsequent code could raise an
exception e.g. AttributeError, TypeError etc and it's not necessarily
possible to enumerate or exhaustively rule out what those
possibilities might be. If we wanted to annotate this with
raises(NotInvertibleError) then it would be very hard or perhaps
entirely impossible for a type checker to verify that no other
exception can be raised. Or maybe even the type checker could easily
come up with a large list of possibilities that you would never want
to annotate your code with. Maybe that's not what the purpose of the
annotation is though.

What the type checker can do is check whether a caller of this
function handles NotInvertibleError after seeing the *explicit* type
hint. A function that calls inverse without catching the exception can
also be considered as raises(NotInvertibleError). You might want to
enforce in your codebase that the caller should catch and suppress the
expected exception or should itself have a compatible raises
annotation indicating that it can also be expected to raise the same
exception e.g. either of these is fine:

def some_calc(M: Matrix): raises(NotInvertibleError)
A = inverse(M)
...

def some_calc(M: Matrix):
try:
A = inverse(M)
except NotInvertibleError
# do something else
...

Perhaps rather than requiring all exceptions to be annotated
everywhere you could allow the raises type hints to propagate
implicitly and only verify them where there is another explicit type
hint:

def some_calc(M):
# no hint but checker infers this raises NotInvertibleError
A = inverse(M)

def other_func(M): raises(ZeroError)
# checker gives an error for this
# because the raises should include NotInvertibleError
B = some_calc(M)

You could then have an explicit hint for the type checker to say that
a function is not expected to raise any exceptions maybe like this:

def main(args): raises(None)
...

The intent of this would be that the type checker could then follow
the chain of all functions called by main to verify that any
exceptions that were expected to raise had been handled somewhere.
This wouldn't verify all of the exceptions that could possibly be
raised by any line of code. It could verify that for those exceptions
that have been explicitly annotated.


Oscar
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/L2YK75C7XSFWOJLM6ROAI3ZVAY2WE5GZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Marco Sulla]

On Fri, 25 Sep 2020 at 14:44, Samuel Colvin  wrote:
>
> Sorry I probably wasn't clear enough in what I was suggesting.
>
>>
>> The main question here is why using a hint or a decorator should be
>> better than a simple documentation.
>
>
> For the same reason type hints are better than documentation

Type hints help an IDE to check if you're potentially passing a bad
parameter to your function.
What does an "exception hint" will do in an IDE? Alerts you that you
are not catching that exception, and, if you really want it to bubble
up, silencing that warning? Again, no thanks :-)
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CFVT7ICR7A5C5AFT2U4JEO2XSCMEHS2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Paul Moore]

On Fri, 25 Sep 2020 at 14:15, Chris Angelico  wrote:

> Why? Do you really think you can enumerate EVERY possible way that
> something might fail?

Rust does a surprisingly good job of that, actually. But the point is
that Python is not Rust, and the infrastructure Rust uses to allow it
to manage code safety is baked into the language core at a very
fundamental level.

Enumerating the exceptions that a piece of code can raise is
impractical and unhelpful in Python. But that may not be immediately
obvious to someone coming from a different language. That's why it's
important to understand Python properly before proposing new features
that work in other languages. (I don't think that's what the OP is
doing here, to be clear, but the discussion is drifting in that
direction, with Rust's Result type having been mentioned).

**In Python**, writing code from the perspective of "what can I handle
at this point" is the right approach. Deferring unexpected exceptions
to your caller is the default behaviour, and results in a clean,
natural style *for Python*. The proposal here is basically in direct
contradiction to that style.

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OERGYDEBVCBA67RBZN6EJYMWPPJ4JMVN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Paul Moore]

On Fri, 25 Sep 2020 at 13:46, Samuel Colvin  wrote:
>
> Sorry I probably wasn't clear enough in what I was suggesting.
>
>>
>> The main question here is why using a hint or a decorator should be
>> better than a simple documentation.
>
>
> For the same reason type hints are better than documentation -
>
> 1. static analysis can catch a multitude of potential errors that humans 
> often miss.
> 2. type hints stay with code in a way documentation often doesn't
> 3. developers are often forced to keep type hints up to date to get tests to 
> pass
>   the same is very rarely true of documentation, this would apply to
>   "exceptions raised" too.

Type hints were largely driven by developers of type checking
software. Is there interest from the mypy developers in this sort of
exception annotation? From any other static checking tool? If there's
no-one proposing to actually use the proposed feature in a tool, it's
not clear how the cost of implementing it is justified. You;ve
discounted runtime checking (and everyone hates that anyway), and no
type checkers have spoken up saying they want it.

You mention IDEs, but if you look at the history of suggestions
intended to support IDEs in this list, you'll find that they very
rarely go anywhere. A significant proportion of Python users don't use
an IDE. And generally, IDE developers don't seem interested in pushing
new language features that they can use - I don't recall ever seeing
an IDE developer speak up to say how much a proposed feature will help
them. In practice, I think IDEs tend to work with heuristics or what
they can do with the language as it is, and aren't really interested
in shaping the language to be "IDE-friendly". But that's just my
impression.

There's also the problem that you've explicitly acknowledged, that
exception hints are *always* going to be inaccurate, in stark contrast
to type hints which are expected to be correct when used. Enumerating
all possible exceptions is impossible, and (as Java's checked
exceptions shows) typically does more harm than good. No exception
hints is where we are now. You're suggesting a feature that lets
people list "some" exceptions - somewhere in the spectrum between
"all" and "none" - but it's completely unspecified what exceptions
need to be included and what don't. "Whatever the developer feels is
appropriate" is basically useless - there's no way any tool or user
can usefully interpret what an exception hint means if things are this
wide open. I can't see how anything could make any practical use of
the data the proposal offers...

Paul
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/ERXDU7SOOILCVS4N7CD2TBSZ2WR6HKXW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 7:59 PM Sergio Fenoll  wrote:
>
> O 25/09/20 ás 11:52, Chris Angelico escribiu:
> > But requests.get() doesn't have a single raise statement anywhere in
> > it. And if you dig through the entire source code for the requests
> > package, you'll still only find a small number of the exceptions that
> > might be raised. Errors come from anywhere, and if you're not handling
> > them, you will simply let them bubble; is it your responsibility to
> > document that? No. It's just the normal thing to do.
> >
> > When you want to catch an exception, don't look at the function to see
> > what it's documented as raising. Look at its behaviour and see what it
> > does that you can cope with. You're looking at things backwards and
> > that's why you're wanting a list of possible things to catch. Instead,
> > look at your program WITHOUT any exception handling, and see what
> > exceptions are happening. Those are the ones to look at.
> >
> > ChrisA
>
>
> Surely there has to be a better way of programming than running stuff,
> watching it fail, and then keeping track of how it fails so you can
> later handle that failure?

Why? Do you really think you can enumerate EVERY possible way that
something might fail?

Think, instead, about all the possible problems that you can actually
cope with. That way, you have a finite - and usually small - set of
things to deal with, instead of an infinite field of "well this could
go wrong, but we can't do anything about that".

In the list of all possible failures, will you include MemoryError?
What would you do if one got raised? If the answer is "nothing, just
let it propagate", then it doesn't need to be in the list. Same goes
for every other exception ever invented.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/TZ5AOOEBFMD7HTP6A2JFLP25HFUBRT2Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[2QdxY4RzWzUUiLuE]

On 2020-09-25 at 13:44:36 +0100,
Samuel Colvin  wrote:

> 2. We should have some easy way to say "let this error propagate",
> rust uses very nice question mark at the end of a line syntax, python
> could use something similar one day, until then a magic comment,
> wrapper function or context function could be used.

In Python, the way to say "let this error propagate" is not to say
anything at all.  y = f(x) says (and very concisely, I might add) "if f
raises an exception, or propagates some exception from some other place,
then let that exception propagate."
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/25E6NBXYXZF5A5UGCXOOG6QZZ5NXH365/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[David Kolovratník]

On Fri, Sep 25, 2020 at 01:23:01PM +0900, Stephen J. Turnbull wrote:
> Sergio Fenoll writes:
> 
>  > In the same vein as adding type annotations to code, I think it'd
>  > be very useful to have exception "raises" annotations, i.e. a way
>  > to annotate what exceptions a function raises.
> 
> I think you need to explain the use cases in more detail.  You mention
> IDEs, but they can already grovel through the source code and work out
> exactly what exceptions each function explicitly raises, and keep a
> database for builtins and the stdlib, which could easily be updated by
> the user by running the groveler on Python itself.  3rd party imports,
> ditto.  This would allow far more accurate inference of possible
> exceptions than an optional 'raises' annotation would.
pyntch (currently ready for Python 2) can already tell us about exceptions.

Do you know about other tools?

A tiny example:

l=[0,1,2]
with open('f') as fh:
a = int(fh.read())
print(1 / l[a])

command (after fixing option hangling in tchecker.py):
tchecker.py -C show_all_exceptions=True -C raise_uncertain=True p.py

output:
loading: 'p.py' as 'p'
processing: 322 161
processing: 324 6
processing: 328 8
processing: 328 4
processing: 329 2
processing: 332 2
processing: 332 1
processing: 333 2
processing: 333 2
total files=1, lines=4 in 0.05sec
[p (p.py)]
  a = 
  fh = 
  l = []
  raises IOError: cannot open a file at p:2
  raises EOFError: end of file at p:3
  raises IndexError: index out of range at p:4

ZeroDivisionError is missing there, however, it couldbe seen as a space for
improvement.

David Kolovratník
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/GZW7NXKJMUTK54KTGBCAXMFVIWDKTTFN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Samuel Colvin]

Sorry I probably wasn't clear enough in what I was suggesting.


> The main question here is why using a hint or a decorator should be
> better than a simple documentation.


For the same reason type hints are better than documentation -

1. static analysis can catch a multitude of potential errors that humans
often miss.
2. type hints stay with code in a way documentation often doesn't
3. developers are often forced to keep type hints up to date to get tests
to pass
  the same is very rarely true of documentation, this would apply to
  "exceptions raised" too.

If the goal is to force people to manage the exception, no, thanks.


Total agree. Here we can take some lessons from rust, but not try
to make python into something it's not (a system language).

I would suggest:

1. "possible exceptions" information should be for static analysis only -
so it would only come into play if you used a static analysis tool anyway,
like type hints.

2. We should have some easy way to say "let this error propagate",
rust uses very nice question mark at the end of a line syntax, python
could use something similar one day, until then a magic comment,
wrapper function or context function could be used.

--

Samuel Colvin
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/RF27BCESZR26N5CJ4G6EKHH7IBYNCDTG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Marco Sulla]

On Fri, 25 Sep 2020 at 11:58, Samuel Colvin  wrote:
> I first found myself wanting this when I came back to python
> having been writing rust. The Result type in rust is somewhat
> similar to what's being suggested here. See
> https://doc.rust-lang.org/std/result/

I do not know Rust and I'm not sure I understood 100% the code. But,
if I'm not wrong, Rust does not use try-catch, but pattern matching.
It seems to me that Rust has a good exception handling system, but
Python does not (yet) have pattern matching.

The main question here is why using a hint or a decorator should be
better than a simple documentation.

If the goal is to force people to manage the exception, no, thanks. As
Serhiy Storchaka already said, it was historically proven as bad. And
even if I've not a great knowledge in Java (3 years), I can assure you
checked exceptions are a really bad idea.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/BRU5WUAFQWMKIJ5I6GWGKTRPSYAGTLRS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 11:52, Chris Angelico escribiu:

But requests.get() doesn't have a single raise statement anywhere in
it. And if you dig through the entire source code for the requests
package, you'll still only find a small number of the exceptions that
might be raised. Errors come from anywhere, and if you're not handling
them, you will simply let them bubble; is it your responsibility to
document that? No. It's just the normal thing to do.

When you want to catch an exception, don't look at the function to see
what it's documented as raising. Look at its behaviour and see what it
does that you can cope with. You're looking at things backwards and
that's why you're wanting a list of possible things to catch. Instead,
look at your program WITHOUT any exception handling, and see what
exceptions are happening. Those are the ones to look at.

ChrisA



Surely there has to be a better way of programming than running stuff, 
watching it fail, and then keeping track of how it fails so you can 
later handle that failure?


Don't get me wrong, it's what I've been doing with Python up until now 
because there's no way around it (other than hoping the library 
documents how and when common exceptions are raised), but I can't say it 
I've really *enjoyed* it.


I hope I'm not the only person who sees this as a suboptimal approach? 
Of course, I could be the weird one out but I frankly doubt it.


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/W7GVIYVDWK63TYHTYVADR44MFG734MDG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Samuel Colvin]

>
> I think it'd be very useful to have exception "raises"

annotations, i.e. a way to annotate what exceptions

a function raises.


I personally think this would be wonderful, I've been meaning
to suggest it here for some time, but haven't got around to it.

I first found myself wanting this when I came back to python
having been writing rust. The Result type in rust is somewhat
similar to what's being suggested here. See
https://doc.rust-lang.org/std/result/

This could be initially implemented as a decorator, without
any special syntax required, in fact to avoid excessive line
length, decorators might be the best way to describe
what exceptions can be raised by a function in the long term.

Using a decorator, this could be implemented as a separate
project (and/or extension to mypy) without any changes to the
standard library. it would obviously be a massive piece of work
to build a database of all the exceptions which could be raised
by all standard library functions, let alone 3rd party libraries.

However you could aid 90% of code by covering
10% of methods (or insert your own 80/20, 99/1 ratio). The
point is that the library could be useful long before it was complete
or 100% correct.

In this regard I see a lot of similarities with type hints and
typeshed.

Anyway, thank you Sergio for suggesting this. I really hope
it comes to fruition somehow.

--

Samuel Colvin
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/Y6WJENUAKLHVSAXJOYPMIFKWXCPCYEI7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 7:46 PM Sergio Fenoll  wrote:
> I'm really failing to see what part of my logic is faulty. I know that
> any exception that isn't handled can be raised. I know that is the
> expected behaviour. I don't expect every function to exhaustively
> document every single exception it may raise (because as you and others
> have said, that would include things like MemoryErrors and
> KeyboardInterrupts). All I'm saying is that *some* exceptions are
> "expected" by whomever implemented the function. To bring it back to my
> earlier example, requests.get() raises a ConnectionError if it can't
> connect to the url you pass it. It would be super useful as a caller of
> that function to have this information *inside your IDE*.
>
> I really don't understand why it's faulty logic to want to have at least
> *some* information about the exceptions a function *may* raise.

But requests.get() doesn't have a single raise statement anywhere in
it. And if you dig through the entire source code for the requests
package, you'll still only find a small number of the exceptions that
might be raised. Errors come from anywhere, and if you're not handling
them, you will simply let them bubble; is it your responsibility to
document that? No. It's just the normal thing to do.

When you want to catch an exception, don't look at the function to see
what it's documented as raising. Look at its behaviour and see what it
does that you can cope with. You're looking at things backwards and
that's why you're wanting a list of possible things to catch. Instead,
look at your program WITHOUT any exception handling, and see what
exceptions are happening. Those are the ones to look at.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2IN57NIPEGYLA4VBR5ODATC746BZMVBN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 11:34, Chris Angelico escribiu:

On Fri, Sep 25, 2020 at 7:19 PM Sergio Fenoll  wrote:

O 25/09/20 ás 10:56, Chris Angelico escribiu:


There's the shallow "exceptions that I expect to raise", which is
those explicitly listed within the function as raise statements; but
that's not the whole story, since exceptions can be raised by anything
that the function calls. So, no, I don't think the callee should have
a notion of "exceptions that I or anyone I call might raise".

ChrisA

I don't understand the need of talking about "exceptions that something
I call might raise" (from the POV of the callee) though, to me they are
irrelevant in this discussion. When you implement a function, you are
handling all/some/none exceptions and letting all/some/nonebubble up and
be handled by whatever called you, right? Somewhere, be it in a
docstring, some manual, external documentation, ... you write down what
exceptions you expect to raise, be it explicit or implicit (e.g. by
letting a known exception bubble up without explicitly raising it
yourself).

Look at your own code. Do you know what EVERY function you call could
raise? If not, then your code could raise those exceptions. It's not
about "letting a known exception bubble up". It's the *normal
behaviour* of letting all exceptions bubble up unless explicitly
caught. Don't think in terms of knowing every exception that could be
raised. That simply doesn't work, and it leads to faulty logic like
this.

ChrisA



I'm really failing to see what part of my logic is faulty. I know that 
any exception that isn't handled can be raised. I know that is the 
expected behaviour. I don't expect every function to exhaustively 
document every single exception it may raise (because as you and others 
have said, that would include things like MemoryErrors and 
KeyboardInterrupts). All I'm saying is that *some* exceptions are 
"expected" by whomever implemented the function. To bring it back to my 
earlier example, requests.get() raises a ConnectionError if it can't 
connect to the url you pass it. It would be super useful as a caller of 
that function to have this information *inside your IDE*.


I really don't understand why it's faulty logic to want to have at least 
*some* information about the exceptions a function *may* raise.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WIBFHB2MBQ2DFINRWCAL72TJJRXGAF52/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 7:19 PM Sergio Fenoll  wrote:
>
> O 25/09/20 ás 10:56, Chris Angelico escribiu:
>
> > There's the shallow "exceptions that I expect to raise", which is
> > those explicitly listed within the function as raise statements; but
> > that's not the whole story, since exceptions can be raised by anything
> > that the function calls. So, no, I don't think the callee should have
> > a notion of "exceptions that I or anyone I call might raise".
> >
> > ChrisA
>
> I don't understand the need of talking about "exceptions that something
> I call might raise" (from the POV of the callee) though, to me they are
> irrelevant in this discussion. When you implement a function, you are
> handling all/some/none exceptions and letting all/some/nonebubble up and
> be handled by whatever called you, right? Somewhere, be it in a
> docstring, some manual, external documentation, ... you write down what
> exceptions you expect to raise, be it explicit or implicit (e.g. by
> letting a known exception bubble up without explicitly raising it
> yourself).

Look at your own code. Do you know what EVERY function you call could
raise? If not, then your code could raise those exceptions. It's not
about "letting a known exception bubble up". It's the *normal
behaviour* of letting all exceptions bubble up unless explicitly
caught. Don't think in terms of knowing every exception that could be
raised. That simply doesn't work, and it leads to faulty logic like
this.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/EYYRYYOUDWDMXSXH2MKKJ45XL7HG4G2F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 06:23, Stephen J. Turnbull escribiu:


Sergio Fenoll writes:

  > In the same vein as adding type annotations to code, I think it'd
  > be very useful to have exception "raises" annotations, i.e. a way
  > to annotate what exceptions a function raises.

I think you need to explain the use cases in more detail.  You mention
IDEs, but they can already grovel through the source code and work out
exactly what exceptions each function explicitly raises, and keep a
database for builtins and the stdlib, which could easily be updated by
the user by running the groveler on Python itself.  3rd party imports,
ditto.  This would allow far more accurate inference of possible
exceptions than an optional 'raises' annotation would.

Even a closed-source vendor removing the .py wouldn't stop a truly
dedicated groveler, I believe.  Because of the way exceptions are
created, there'd be a trace in a namespace in the .pyc or even in a
DLL.


I'm no expert so I'm not sure how IDEs currently handle typing 
annotations, I assume they make use of the built-in language features 
for that as opposed to some proprietary solution that every IDE would 
have to re-implement? If that is the case, i.e. they use a built-in 
language feature, I think it'd be beneficial to have exception 
annotations as part of the language as well. Also, if you had a dunder 
like __raises__ for functions (like you have __annotations__) you could 
write some cool (and possibly useful?) introspective code.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/A3BPJNCFTEEYKOPZJ6RVHGEJROLTYHPJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 10:56, Chris Angelico escribiu:


There's the shallow "exceptions that I expect to raise", which is
those explicitly listed within the function as raise statements; but
that's not the whole story, since exceptions can be raised by anything
that the function calls. So, no, I don't think the callee should have
a notion of "exceptions that I or anyone I call might raise".

ChrisA


I don't understand the need of talking about "exceptions that something 
I call might raise" (from the POV of the callee) though, to me they are 
irrelevant in this discussion. When you implement a function, you are 
handling all/some/none exceptions and letting all/some/nonebubble up and 
be handled by whatever called you, right? Somewhere, be it in a 
docstring, some manual, external documentation, ... you write down what 
exceptions you expect to raise, be it explicit or implicit (e.g. by 
letting a known exception bubble up without explicitly raising it 
yourself). I'm basically only talking about those kinds of exceptions 
here, because in my experience those are the exceptions I'm most often 
having to try-except. Sure, as a caller you may want to handle more 
exceptions than what the callee sees as "expected exceptions", but 
nothing will stop you from doing that just as you do it now. I don't see 
how having these exceptions be annotated in the code makes anything 
worse. From my perspective it just makes the most common try-except 
you'll write easier to implement without having to leave the IDE and 
check external documentation.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AUESIFXSGGBAZIJBRBAHJ3UFOODR233I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 6:14 PM Sergio Fenoll  wrote:
> I see what you mean now, and I agree. But surely there's a fairly big
> overlap between
> "exceptions I expect to raise" (from the POV of the callee) and
> "exceptions I can handle"
> (from the POV of the caller). Wouldn't it still be useful to have the
> first part (exceptions
> I expect to raise) defined in the code (i.e., via some kind of
> annotation) in such a way that
> makes it easier for the caller to handle the exceptions they want to catch?
>
> Or, in your opinion, should the callee not even have a notion of
> "exceptions I expect to raise"?
>

There's the shallow "exceptions that I expect to raise", which is
those explicitly listed within the function as raise statements; but
that's not the whole story, since exceptions can be raised by anything
that the function calls. So, no, I don't think the callee should have
a notion of "exceptions that I or anyone I call might raise".

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/2IUTNJBADJDHCR7J7UNDRYL32PFY7YIC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 10:02, Chris Angelico escribiu:

On Fri, Sep 25, 2020 at 5:09 PM Sergio Fenoll  wrote:

O 25/09/20 ás 08:41, Chris Angelico escribiu:


It would have to not only look at get(), but everything that it calls.
Either that, or you're back to the Java thing of "catch it or declare
it", and we've seen from Java that that's a bad idea.

ChrisA

I don't really understand why those are the only options. The way I see it,
whomever implements get() can just have a list of expected Exceptions and as
a caller you just handle those (now with the added ease that you don't
need to
look them up out-of-bounds, they're already in your IDE!) and if an
unexpected
exception happens, well you handle it the same way you do it now (i.e.
either not
at all, with a bare except, or something along those lines).

Don't think in terms of "expected exceptions". That's the mentality
that leads to coding styles where you try to catch every exception the
function can raise, which is the wrong way to think about it.

Instead, think in terms of "exceptions that I can handle".

ChrisA
I see what you mean now, and I agree. But surely there's a fairly big 
overlap between
"exceptions I expect to raise" (from the POV of the callee) and 
"exceptions I can handle"
(from the POV of the caller). Wouldn't it still be useful to have the 
first part (exceptions
I expect to raise) defined in the code (i.e., via some kind of 
annotation) in such a way that

makes it easier for the caller to handle the exceptions they want to catch?

Or, in your opinion, should the callee not even have a notion of 
"exceptions I expect to raise"?


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/V36GLZXCT63ODWMTRQY23VT7WUVMO6W2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 5:09 PM Sergio Fenoll  wrote:
>
> O 25/09/20 ás 08:41, Chris Angelico escribiu:
>
> > It would have to not only look at get(), but everything that it calls.
> > Either that, or you're back to the Java thing of "catch it or declare
> > it", and we've seen from Java that that's a bad idea.
> >
> > ChrisA
> I don't really understand why those are the only options. The way I see it,
> whomever implements get() can just have a list of expected Exceptions and as
> a caller you just handle those (now with the added ease that you don't
> need to
> look them up out-of-bounds, they're already in your IDE!) and if an
> unexpected
> exception happens, well you handle it the same way you do it now (i.e.
> either not
> at all, with a bare except, or something along those lines).

Don't think in terms of "expected exceptions". That's the mentality
that leads to coding styles where you try to catch every exception the
function can raise, which is the wrong way to think about it.

Instead, think in terms of "exceptions that I can handle".

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/CGQESM3SQA3BYOEVG4OBLABNIGKXEFR5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 09:12, Serhiy Storchaka escribiu:

Did not the experience of C++ show that exceptions declaration was a bad
idea? In C++ it was optional and is abandoned now. Java developers still
suffer from necessary to declare all raised exception, or just declare
the most general exception class, that makes the feature useless.

Could you elaborate as to why it was bad in C++? I looked it up and see that
it was deprecated before I learned C++ so I never used it myself, so I'm
interested to hear what was wrong with it.
I do agree that the way Java does it is too cumbersome, but I'm not 
proposing
that Python gets Java-like checked exceptions, so I don't think it's a 
good comparison.

This may be especially bad for Python where virtually every line of code
can raise arbitrary exception. KeybordInterrupt and MemoryError are most
common examples, but in rare cases which usually are not covered by
tests it can be also UnicodeError, NameError, AttributeError, TypeError,
BufferError, etc.
Agreed! This is why I think this feature would be useful to document 
_expected_ Exceptions.
Something that already happens anyway (in actual documentation) but 
would be (IMO)
far more useful if it were to happen in the code itself (much like type 
annotations).

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/OWYWMYFKDOLWNQX2R4P4LGUXJIZR3NSW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Serhiy Storchaka]

24.09.20 11:47, Sergio Fenoll пише:
> In the same vein as adding type annotations to code, I think it'd be
> very useful to have exception "raises" annotations, i.e. a way to
> annotate what exceptions a function raises. Again, like type
> annotations, it shouldn't be mandatory nor actually be enforced at
> runtime. It would purely serve as a feature that IDEs can make use of.
> 
> An example of how it may look:
> 
> def divide(numerator: float, denominator: float) raises [ZeroDivisionError] 
> -> float:
>     return numerator / denominator
> 
> I'd love to know if this is an idea you'd be interested in having added
> to the language.

Did not the experience of C++ show that exceptions declaration was a bad
idea? In C++ it was optional and is abandoned now. Java developers still
suffer from necessary to declare all raised exception, or just declare
the most general exception class, that makes the feature useless.

This may be especially bad for Python where virtually every line of code
can raise arbitrary exception. KeybordInterrupt and MemoryError are most
common examples, but in rare cases which usually are not covered by
tests it can be also UnicodeError, NameError, AttributeError, TypeError,
BufferError, etc.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/F244P5HTWU666LZSE3XLULFMNGPZQ27M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 08:41, Chris Angelico escribiu:


On Fri, Sep 25, 2020 at 4:25 PM Sergio Fenoll  wrote:

What I had in mind was that an IDE could use this information to show
autocomplete options when writing except blocks. The other day I was
writing some code like this:

import requests

try:

  requests.get('https://python.org')

except WhateverExceptionTheRequestsLibraryRaises:

  pass


It would have to not only look at get(), but everything that it calls.
Either that, or you're back to the Java thing of "catch it or declare
it", and we've seen from Java that that's a bad idea.

ChrisA

I don't really understand why those are the only options. The way I see it,
whomever implements get() can just have a list of expected Exceptions and as
a caller you just handle those (now with the added ease that you don't 
need to
look them up out-of-bounds, they're already in your IDE!) and if an 
unexpected
exception happens, well you handle it the same way you do it now (i.e. 
either not

at all, with a bare except, or something along those lines).
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/AZ6FLPAYPWBYBI65EGCPCO3B2KBOZGBM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 4:25 PM Sergio Fenoll  wrote:
> What I had in mind was that an IDE could use this information to show
> autocomplete options when writing except blocks. The other day I was
> writing some code like this:
>
> import requests
>
> try:
>
>  requests.get('https://python.org')
>
> except WhateverExceptionTheRequestsLibraryRaises:
>
>  pass
>

It would have to not only look at get(), but everything that it calls.
Either that, or you're back to the Java thing of "catch it or declare
it", and we've seen from Java that that's a bad idea.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/L6M7WROB7FF7SPMPY7DPRW2ZQP4ISR4F/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Sergio Fenoll]

O 25/09/20 ás 07:37, Steven D'Aprano escribiu:


I think that it's a truism that any function written in Python could
raise any exception :-)

In practice, though, I think it is reasonable to say that functions will
*typically* but not necessarily exclusively raise a certain set of
exceptions.

What you seem to be describing is similar to Java's "Checked
Exceptions", which are widely agreed to be an failure. Can you explain
why this would be more successful?

The last thing I want to see is people being encouraged to write code
like this:

 def demo(arg)->Something:
 # Raises ValueError
 try:
 processing...
 except ValueError:
 raise
 except:
 # Any other unexpected error.
 raise ValueError('something unexpected')


just to ensure that the declared exception is correct.
I haven't extensively used Java, but the way I understand it in Java 
you're meant to either handle every single exception that a function can 
throw, or explicitly throw anything you haven't handled.
I do agree that such an approach wouldn't be great for Python. I don't 
think (but don't quote me on that) that introducing this idea of 
annotating exceptions would lead people to writing code like the above 
(in the same way introducing typing annotations didn't lead to people 
checking all function argument types in every function :-P ). After all, 
there wouldn't be any runtime checks to make sure you handled any 
raisable exceptions.



In the same vein as adding type annotations to code, I think it'd be
very useful to have exception "raises" annotations, i.e. a way to
annotate what exceptions a function raises. Again, like type
annotations, it shouldn't be mandatory nor actually be enforced at
runtime. It would purely serve as a feature that IDEs can make use of.

How would an IDE make use of this?
What I had in mind was that an IDE could use this information to show 
autocomplete options when writing except blocks. The other day I was 
writing some code like this:


import requests

try:

    requests.get('https://python.org')

except WhateverExceptionTheRequestsLibraryRaises:

    pass

And I remember having to dive into the documentation of the library to 
figure out what exception was actually raised by this method, and what 
the base class of said exception was.
Whereas, IMO, it would be a lot nicer to have a built-in way to know 
what exceptions may be raised by a function call.



An example of how it may look:

def divide(numerator: float, denominator: float) raises [ZeroDivisionError] -> 
float:
     return numerator / denominator

If either argument is a subclass of float, this could raise any
exception.
I see! Well, the way I had it in mind, you'd only really annotate the 
exceptions you as a function implementer think are meaningful. It 
doesn't have to be an exhaustive list, just exceptions the callers of 
the function will (probably) want to handle.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LWUWC4RXI5WOWAIYHWSTJK23VWNHBDOG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Chris Angelico]

On Fri, Sep 25, 2020 at 3:38 PM Steven D'Aprano  wrote:
> The last thing I want to see is people being encouraged to write code
> like this:
>
> def demo(arg)->Something:
> # Raises ValueError
> try:
> processing...
> except ValueError:
> raise
> except:
> # Any other unexpected error.
> raise ValueError('something unexpected')
>
>
> just to ensure that the declared exception is correct.
>

Another thing I don't want to see is:

def demo(arg):
try:
some_func() # declared to raise ValueError, TypeError, and TimeoutError
return 42
except ValueError:
return 0
except TypeError
# Shouldn't happen, just return whatever
return -1
except TimeoutError
# I have no idea why this would happen! Just return zero and hope
# for the best. TODO: Figure out a better return value.
return 0

where people feel they HAVE to catch everything that a function could
raise. (Which, btw, still ignores the fact that basically any code
could raise MemoryError, KeyboardInterrupt, etc.)

Only catch those exceptions that you can actually handle. Otherwise,
it's irrelevant what the function might raise.

ChrisA
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WC2SRZ64HSS4JUXEQTPCPIT2R62ZNNAT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Steven D'Aprano]

On Thu, Sep 24, 2020 at 10:47:21AM +0200, Sergio Fenoll wrote:
> Hi,
> 
> I was wondering if the following idea would be a useful addition to the
> Python language and if it could use a new PEP.
> I personally find myself often looking into the documentation &
> implementation of libraries I use to try and figure out what exceptions
> a function may raise.

I think that it's a truism that any function written in Python could 
raise any exception :-)

In practice, though, I think it is reasonable to say that functions will 
*typically* but not necessarily exclusively raise a certain set of 
exceptions.

What you seem to be describing is similar to Java's "Checked 
Exceptions", which are widely agreed to be an failure. Can you explain 
why this would be more successful?

The last thing I want to see is people being encouraged to write code 
like this:

def demo(arg)->Something:
# Raises ValueError
try:
processing...
except ValueError:
raise
except:
# Any other unexpected error.
raise ValueError('something unexpected')


just to ensure that the declared exception is correct.

> In the same vein as adding type annotations to code, I think it'd be
> very useful to have exception "raises" annotations, i.e. a way to
> annotate what exceptions a function raises. Again, like type
> annotations, it shouldn't be mandatory nor actually be enforced at
> runtime. It would purely serve as a feature that IDEs can make use of.

How would an IDE make use of this?

> An example of how it may look:
> 
> def divide(numerator: float, denominator: float) raises [ZeroDivisionError] 
> -> float:
>     return numerator / denominator

If either argument is a subclass of float, this could raise any 
exception.


-- 
Steve
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WKANCUIL2XTQXYRWP7SVINWB5ANT2ZOR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Suggestion: annotated exceptions

[Ricky Teachey]

I like the idea.

In python 3.9 you could actually experiment with implementing something
like this yourself using the new Annotated type introduced in PEP 593:

https://docs.python.org/3.9/library/typing.html#typing.Annotated

https://www.python.org/dev/peps/pep-0593/

Maybe create a raises helper type and write your annotation like this?

from typing import Annotated

def divide(numerator: float, denominator: float) -> Annotated[float, raises
[ZeroDivisionError]]:
return numerator / denominator

---
Ricky.

"I've never met a Kentucky man who wasn't either thinking about going home
or actually going home." - Happy Chandler


On Thu, Sep 24, 2020 at 8:45 AM Sergio Fenoll  wrote:

> Hi,
>
> I was wondering if the following idea would be a useful addition to the
> Python language and if it could use a new PEP.
> I personally find myself often looking into the documentation &
> implementation of libraries I use to try and figure out what exceptions
> a function may raise.
>
> In the same vein as adding type annotations to code, I think it'd be
> very useful to have exception "raises" annotations, i.e. a way to
> annotate what exceptions a function raises. Again, like type
> annotations, it shouldn't be mandatory nor actually be enforced at
> runtime. It would purely serve as a feature that IDEs can make use of.
>
> An example of how it may look:
>
> def divide(numerator: float, denominator: float) raises
> [ZeroDivisionError] -> float:
> return numerator / denominator
>
> I'd love to know if this is an idea you'd be interested in having added
> to the language.
>
> Kind regards,
> Sergio Fenoll
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/LFUZR6NH35HOLTVGV5YRB6457KZ2KJ5A/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/6IEUEUP7KDZUSV6W5P56JSC2KDBX32NS/
Code of Conduct: http://python.org/psf/codeofconduct/