[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Nathaniel Smith]

On Thu, Feb 25, 2021 at 10:23 PM Glenn Linderman  wrote:
> So then why do you need  except*  at all?  Only to catch unwrapped
> ExceptionGroup before it gets wrapped?
>
> So why not write except ExceptionGroup, and let it catch unwrapped
> ExceptionGroup?
>
> That "CUTE BIT" could be done only when hitting an except chain that
> doesn't include an except ExceptionGroup.
>
> Nope, I didn't read the PEP, and don't understand the motivation, but
> the discussion sure sounded confusing. This is starting to sound almost
> reasonable.

I'm not sure what to make of the complaint that not reading the
motivation makes the motivation confusing :-).

But very briefly: the core reason we need ExceptionGroup is because in
concurrent programs, multiple things can go wrong at the same time, so
our error handling system needs to have some way to represent and cope
with that. This means that in concurrent programs (e.g. anything using
asyncio, trio, etc.), it's safest to assume that *any* exception could
be wrapped in an ExceptionGroup and use except* for everything. The
question is how to make error handling in those programs as ergonomic
as Python exceptions are currently in non-concurrent programs, without
creating too many issues for existing code.

So in programs like this, you have to assume that 'except
ExceptionGroup' catches *all* exceptions, or worse, a
random/unpredictable subset. Saying that that would be good enough is
like saying that bare 'except:' is good enough for regular
non-concurrent Python (after all, you can always do an isinstance
check inside the 'except' block and re-raise the ones you don't want,
right?), and that 'except ' is a pointless frivolity. I
think most people would disagree with that :-).

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BYCDFS5FVIF4FYNKUPVEVWRSCCEIU6CB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Glenn Linderman]

On 2/25/2021 9:40 PM, Nathaniel Smith wrote:

On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum  wrote:

So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? 
(Do we call this out in the PEP?)

We may believe that "except Exception" is an abuse, but it is too common to dismiss out of hand. I 
think if some app has e.g. a main loop where they repeatedly do something that may fail in many ways (e.g. 
handle a web request), catch all errors and then just log the error and continue from the top, it's a better 
experience if it logs "ExceptionGroup:  []" than if it 
crashes.

Yeah, 'except Exception' happens a lot in the wild, and what to do
about that has been a major sticking point in the ExceptionGroup
debates all along. I wouldn't say that 'except Exception' is an abuse
even -- what do you want gunicorn to do if your buggy flask app raises
some random exception? Crash your entire web server, or log it and
attempt to keep going? (This is almost your example, but adding in the
part where gunicorn is reliable and well-respected, and that its whole
job is to invoke arbitrarily flaky code written by random users.)
Yury/I/others did discuss the idea of a
BaseExceptionGroup/ExceptionGroup split a lot, and I think the general
feeling is that it could potentially work, but feels like a
complicated and awkward hack, so no-one was super excited about it.
For a while we also had a compromise design where only
BaseExceptionGroup was built-in, but we left it non-final specifically
so asyncio could define an ExceptionsOnlyExceptionGroup.

Another somewhat-related awkward part of the API is how ExceptionGroup
and plain-old 'except' should interact *in general*. The intuition is
that if you have 'except ValueError' and you get an
'ExceptionGroup(ValueError)', then the user's code has some kind of
problem and we should probably do something? to let them know? One
idea I had was that we should raise a RuntimeError if this happens,
sort of similar to PEP 479. But I could never quite figure out how
this would help (gunicorn crashing with a RuntimeError isn't obviously
better than gunicorn crashing with an ExceptionGroup).

== NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS ==

Proposal:

- any time an unwinding ExceptionGroup encounters a traditional
try/except, then it gets replaced with a RuntimeError whose __cause__
is set to the original ExceptionGroup and whose first traceback entry
points to the offending try/except block

- CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right
*before* we start evaluating 'except' clauses for this try/except

So for example:

If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is
replaced by a RuntimeError. RuntimeError is an Exception, so the
'except Exception' clause catches it. And presumably logs it or
something. This way your log contains both a notification that you
might want to switch to except* (from the RuntimeError), *along with*
the full original exception details (from the __cause__ attribute). If
it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught
and that's not so great, but at least you get the RuntimeError to
point out that something has gone wrong and tell you where?

If an ExceptionGroup(ValueError) hits an 'except ValueError': it
doesn't get caught, *but* a RuntimeError keeps propagating out to tell
you you have a problem. And when that RuntimeError eventually hits the
top of your program or ends up in your webserver logs or whatever,
then the RuntimeError's traceback will point you to the 'except
ValueError' that needs to be fixed.

If you write 'except ExceptionGroup': this clause is a no-op that will
never execute, because it's impossible to still have an ExceptionGroup
when we start matching 'except' clauses. (We could additionally emit a
diagnostic if we want.)

If you write bare 'except:', or 'except BaseException': the clause
always executes (as before), but they get the RuntimeError instead of
the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can
retrieve it from the __cause__ attribute. (The only case I can think
of where this would be useful is if you're writing code that has to
straddle both old and new Python versions *and* wants to do something
clever with ExceptionGroups. I think this would happen if you're
implementing Trio, or implementing a higher-level backport library for
catching ExceptionGroups, something like that. So this only applies to
like half a dozen users total, but they are important users :-).)

-n

I wondered why an ExceptionGroup couldn't just be an Exception. This 
effectively makes it so (via wrapping).


So then why do you need  except*  at all?  Only to catch unwrapped 
ExceptionGroup before it gets wrapped?


So why not write except ExceptionGroup, and let it catch unwrapped 
ExceptionGroup?


That "CUTE BIT" could be done only when hitting an except chain that 
doesn't include an except ExceptionGroup.


Nope, I didn't read the PEP, and don't und

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Nathaniel Smith]

On Thu, Feb 25, 2021 at 2:13 PM Guido van Rossum  wrote:
>
> So is "fail-fast if you forget to handle an ExceptionGroup" really a feature? 
> (Do we call this out in the PEP?)
>
> We may believe that "except Exception" is an abuse, but it is too common to 
> dismiss out of hand. I think if some app has e.g. a main loop where they 
> repeatedly do something that may fail in many ways (e.g. handle a web 
> request), catch all errors and then just log the error and continue from the 
> top, it's a better experience if it logs "ExceptionGroup:  [ subexceptions>]" than if it crashes.

Yeah, 'except Exception' happens a lot in the wild, and what to do
about that has been a major sticking point in the ExceptionGroup
debates all along. I wouldn't say that 'except Exception' is an abuse
even -- what do you want gunicorn to do if your buggy flask app raises
some random exception? Crash your entire web server, or log it and
attempt to keep going? (This is almost your example, but adding in the
part where gunicorn is reliable and well-respected, and that its whole
job is to invoke arbitrarily flaky code written by random users.)
Yury/I/others did discuss the idea of a
BaseExceptionGroup/ExceptionGroup split a lot, and I think the general
feeling is that it could potentially work, but feels like a
complicated and awkward hack, so no-one was super excited about it.
For a while we also had a compromise design where only
BaseExceptionGroup was built-in, but we left it non-final specifically
so asyncio could define an ExceptionsOnlyExceptionGroup.

Another somewhat-related awkward part of the API is how ExceptionGroup
and plain-old 'except' should interact *in general*. The intuition is
that if you have 'except ValueError' and you get an
'ExceptionGroup(ValueError)', then the user's code has some kind of
problem and we should probably do something? to let them know? One
idea I had was that we should raise a RuntimeError if this happens,
sort of similar to PEP 479. But I could never quite figure out how
this would help (gunicorn crashing with a RuntimeError isn't obviously
better than gunicorn crashing with an ExceptionGroup).

== NEW IDEA THAT MAYBE SOLVES BOTH PROBLEMS ==

Proposal:

- any time an unwinding ExceptionGroup encounters a traditional
try/except, then it gets replaced with a RuntimeError whose __cause__
is set to the original ExceptionGroup and whose first traceback entry
points to the offending try/except block

- CUTE BIT I ONLY JUST THOUGHT OF: this substitution happens right
*before* we start evaluating 'except' clauses for this try/except

So for example:

If an ExceptionGroup hits an 'except Exception': The ExceptionGroup is
replaced by a RuntimeError. RuntimeError is an Exception, so the
'except Exception' clause catches it. And presumably logs it or
something. This way your log contains both a notification that you
might want to switch to except* (from the RuntimeError), *along with*
the full original exception details (from the __cause__ attribute). If
it was an ExceptionGroup(KeyboardInterrupt), then it still gets caught
and that's not so great, but at least you get the RuntimeError to
point out that something has gone wrong and tell you where?

If an ExceptionGroup(ValueError) hits an 'except ValueError': it
doesn't get caught, *but* a RuntimeError keeps propagating out to tell
you you have a problem. And when that RuntimeError eventually hits the
top of your program or ends up in your webserver logs or whatever,
then the RuntimeError's traceback will point you to the 'except
ValueError' that needs to be fixed.

If you write 'except ExceptionGroup': this clause is a no-op that will
never execute, because it's impossible to still have an ExceptionGroup
when we start matching 'except' clauses. (We could additionally emit a
diagnostic if we want.)

If you write bare 'except:', or 'except BaseException': the clause
always executes (as before), but they get the RuntimeError instead of
the ExceptionGroup. If you really *wanted* the ExceptionGroup, you can
retrieve it from the __cause__ attribute. (The only case I can think
of where this would be useful is if you're writing code that has to
straddle both old and new Python versions *and* wants to do something
clever with ExceptionGroups. I think this would happen if you're
implementing Trio, or implementing a higher-level backport library for
catching ExceptionGroups, something like that. So this only applies to
like half a dozen users total, but they are important users :-).)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JYRWNHRBPFQQPT44TZQRTVFNXPAY27UL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum  wrote:

> Then we should get some more opinions on this. I think it's the best idea
> so far for kindness towards code using `except Exception:`.
>

I agree that it's the best idea so far.
If someone says 'except Exception' they better mean it because we'll
believe them, and if someone forgets to handle an ExceptionGroup then they
have a bug and that's how it is.



> OT: Is ExceptionGroup *really* immutable in the current implementation? As
> long as the 'errors' field is a list, I think one could mutate the list
> directly.
>

It's not, but we were going to make it an immutable tuple.


> Which brings me to the question, do you have a branch that matches the PEP
> yet?
>
>
The implementation matches the PEP less and less every day because the PEP
is developing faster than the implementation.
But these aren't differences that are more than a technicality to fix
(rename things, move an error from runtime to the parser, things like that).
The except* implementation is probably pretty close to the PEP because it's
the most recent bit.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I473Z5E3FA3SV7MLT74UNPWEG3W2FOCA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Stephen J. Turnbull]

Mike Miller writes:

 > "sys-admin" is a bit of an overstatement in my phrasing.  The core
 > is that you need to understand how a PATH works and be able to run
 > pip and cd into folders and perhaps run rm/del, etc.  Basic
 > command-line skills?

That's what I would mean by basic sys-admin skills.  And *surprise!*
my students don't have them, and don't need them ... until they start
using Python.  They don't even understand installing packages in
conda.

If I had my druthers, we wouldn't have classes in Excel and
Powerpoint, and we wouldn't have classes in SPSS and SAS, and we
wouldn't even have classes in Python -- we'd have classes in envars,
registry, and shell scripts/bat files.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RJAZJLN6W76QONKD3K2UG6BPTOXPWE4D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

Great. Let’s wait for others to catch up with the discussion then.

On Thu, Feb 25, 2021 at 18:44 Irit Katriel 
wrote:

>
>
> On Fri, Feb 26, 2021 at 2:00 AM Guido van Rossum  wrote:
>
>> Then we should get some more opinions on this. I think it's the best idea
>> so far for kindness towards code using `except Exception:`.
>>
>
> I agree that it's the best idea so far.
> If someone says 'except Exception' they better mean it because we'll
> believe them, and if someone forgets to handle an ExceptionGroup then they
> have a bug and that's how it is.
>
>
>
>> OT: Is ExceptionGroup *really* immutable in the current implementation?
>> As long as the 'errors' field is a list, I think one could mutate the list
>> directly.
>>
>
> It's not, but we were going to make it an immutable tuple.
>
>
>> Which brings me to the question, do you have a branch that matches the
>> PEP yet?
>>
>>
> The implementation matches the PEP less and less every day because the PEP
> is developing faster than the implementation.
> But these aren't differences that are more than a technicality to fix
> (rename things, move an error from runtime to the parser, things like that).
> The except* implementation is probably pretty close to the PEP because
> it's the most recent bit.
>
>
>
>
>
>
> --
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VS2JC7OK67D3DA3E45IR3IKHZWSLGBFO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 9:06 PM Guido van Rossum  wrote:

>
> Hm, a different idea: maybe it's simple enough that we can just add an
> example showing how to do this? Then people can tailor that e.g. to use
> various traversal orders. (We could just link to the code in traceback.py,
> but it probably is full of distractions.)
>


I've added it here: https://github.com/python/peps/pull/1841
I'd rather do this for now, and add it to the standard library only when we
have a better idea about common patterns // best practices for handling
ExceptionGroups.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4UDHU7ZMZBGS5V6A62HHYBPG4JCJICQZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

Then we should get some more opinions on this. I think it's the best idea
so far for kindness towards code using `except Exception:`. I even think
that the names I came up with are reasonable. The pseudo-code needs work:
when instantiating ExceptionGroup, all errors should inherit from
Exception, and I can't decide which base class for ExceptionGroup should
come first: Exception or BaseExceptionGroup. I'm currently leaning slightly
towards putting BaseExceptionGroup first. This is my current attempt:
```
class BaseExceptionGroup(BaseException):
def __new__(cls, msg, errors):
if cls is BaseExceptionGroup and all(isinstance(e, Exception) for e
in errors):
cls = ExceptionGroup
else:
assert all(isinstance(e, Exception) for e in errors)
return super().__new__(cls, msg, errors)
def __init__(self, msg, errors):
self.msg = msg
self.errors = errors
class ExceptionGroup(BaseExceptionGroup, Exception):
pass
```

OT: Is ExceptionGroup *really* immutable in the current implementation? As
long as the 'errors' field is a list, I think one could mutate the list
directly. Which brings me to the question, do you have a branch that
matches the PEP yet?

--Guido

On Thu, Feb 25, 2021 at 3:26 PM Irit Katriel 
wrote:

>
> We don't call it out but we talked about this at the sprint.  I think the
> reason we didn't come up with this solution then is that at the time
> ExceptionGroups in our plans were not immutable, so this would not have
> been possible (you don't know at construction time what it is going to
> contain).
>
>
> On Thu, Feb 25, 2021 at 10:08 PM Guido van Rossum 
> wrote:
>
>> So is "fail-fast if you forget to handle an ExceptionGroup" really a
>> feature? (Do we call this out in the PEP?)
>>
>> We may believe that "except Exception" is an abuse, but it is too common
>> to dismiss out of hand. I think if some app has e.g. a main loop where they
>> repeatedly do something that may fail in many ways (e.g. handle a web
>> request), catch all errors and then just log the error and continue from
>> the top, it's a better experience if it logs "ExceptionGroup: 
>> []" than if it crashes.
>>
>> There's also code that catches Exception, logs the error and some
>> relevant data, and then re-raises it. The logged error may go to a more
>> specialized destination than the generic traceback, and the log message may
>> contain additional data that's only available in that stack frame.
>>
>> So I think there are enough serious use cases that we should do what's
>> best for those use cases, and not try to lecture users about abuse of the
>> idiom.
>>
>> I don't know what we would have done if we were building Python from
>> scratch. Possibly we would not have BaseException at all, and the whole
>> mess would go away. (But there are some good reasons why we introduced
>> BaseException, so I don't know that that would really be a better overall
>> experience.)
>>
>> --Guido
>>
>>
>>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZAEF52BHSTDPD5GF2O7YDYL6OSTOEJS2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

We don't call it out but we talked about this at the sprint.  I think the
reason we didn't come up with this solution then is that at the time
ExceptionGroups in our plans were not immutable, so this would not have
been possible (you don't know at construction time what it is going to
contain).


On Thu, Feb 25, 2021 at 10:08 PM Guido van Rossum  wrote:

> So is "fail-fast if you forget to handle an ExceptionGroup" really a
> feature? (Do we call this out in the PEP?)
>
> We may believe that "except Exception" is an abuse, but it is too common
> to dismiss out of hand. I think if some app has e.g. a main loop where they
> repeatedly do something that may fail in many ways (e.g. handle a web
> request), catch all errors and then just log the error and continue from
> the top, it's a better experience if it logs "ExceptionGroup: 
> []" than if it crashes.
>
> There's also code that catches Exception, logs the error and some relevant
> data, and then re-raises it. The logged error may go to a more specialized
> destination than the generic traceback, and the log message may contain
> additional data that's only available in that stack frame.
>
> So I think there are enough serious use cases that we should do what's
> best for those use cases, and not try to lecture users about abuse of the
> idiom.
>
> I don't know what we would have done if we were building Python from
> scratch. Possibly we would not have BaseException at all, and the whole
> mess would go away. (But there are some good reasons why we introduced
> BaseException, so I don't know that that would really be a better overall
> experience.)
>
> --Guido
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DJ265GSMTHIIDLKR5JTV3NILOYCLVJDX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

So is "fail-fast if you forget to handle an ExceptionGroup" really a
feature? (Do we call this out in the PEP?)

We may believe that "except Exception" is an abuse, but it is too common to
dismiss out of hand. I think if some app has e.g. a main loop where they
repeatedly do something that may fail in many ways (e.g. handle a web
request), catch all errors and then just log the error and continue from
the top, it's a better experience if it logs "ExceptionGroup: 
[]" than if it crashes.

There's also code that catches Exception, logs the error and some relevant
data, and then re-raises it. The logged error may go to a more specialized
destination than the generic traceback, and the log message may contain
additional data that's only available in that stack frame.

So I think there are enough serious use cases that we should do what's best
for those use cases, and not try to lecture users about abuse of the idiom.

I don't know what we would have done if we were building Python from
scratch. Possibly we would not have BaseException at all, and the whole
mess would go away. (But there are some good reasons why we introduced
BaseException, so I don't know that that would really be a better overall
experience.)

--Guido



On Thu, Feb 25, 2021 at 2:46 AM Irit Katriel 
wrote:

>
>
> On Thu, Feb 25, 2021 at 5:59 AM Guido van Rossum  wrote:
>
>>
>> Here's a potentially alternative plan, which is also complex, but doesn't
>> require asyncio or other use cases to define special classes. Let's define
>> two exceptions, BaseExceptionGroup which wraps BaseException instances, and
>> ExceptionGroup which only wraps Exception instances. (Names to be
>> bikeshedded.) They could share a constructor (always invoked via
>> BaseExceptionGroup) which chooses the right class depending on whether
>> there are any non-Exception instances being wrapped -- this would do the
>> right thing for split() and subgroup() and re-raising unhandled exceptions.
>>
>> Then 'except Exception:' would catch ExceptionGroup but not
>> BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be
>> caught (even if there's also e.g. a ValueError among the wrapped errors).
>>
>> Pseudo-code:
>>
>> class BaseExceptionGroup(BaseException):
>> def __new__(cls, msg, errors):
>> if cls is BaseExceptionGroup and all(isinstance(e, Exception) for
>> e in errors):
>> cls = ExceptionGroup
>> return BaseException.__new__(cls, msg, errors)
>>
>> class ExceptionGroup(Exception, BaseExceptionGroup):
>> pass
>>
>
>
> This could be a valid compromise.
>
> We keep the ability to wrap any exception, while we lose the "fail-fast if
> you forget to handle an ExceptionGroup" feature, which was intended as a
> kindness towards those who abuse "except Exception".
>
> If we adopt this solution then letting an ExceptionGroup escape from code
> that is not supposed to raise it, is not a fatal error, it's just some
> exception like any other.
> So there is no longer a distinction between code that raises
> ExceptionGroups and code that doesn't. Any code can propagate them, like
> any code can raise any other exception.
> Does this mean that more code needs to be aware of the possibility of them
> showing up?  Is that a problem?  Maybe this a simpler state of affairs
> overall.
>
> What would we have done here if we were building Python from scratch?
>
> Irit
>
>
>
>
>
>
>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/45U4VDHH3E7JJNX2L5JCUE4CJJEMFRNH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Guido van Rossum]

Good question. The main use case I see for iterating over individual
exceptions is logging frameworks that want to format their own exception
tracebacks for whatever reason. I know that the traceback.py module has
extensive utilities for exactly that, but I betcha there are plenty of
people who still want to roll their own, e.g. for compatibility with other
tooling on their platform.

Presumably the code for traversing a tree of exceptions exists in
traceback.py. I honestly don't think it matters much whether we do it as an
iterator or using callbacks, as long as it visits all the leaves in the
tree.

Hm, a different idea: maybe it's simple enough that we can just add an
example showing how to do this? Then people can tailor that e.g. to use
various traversal orders. (We could just link to the code in traceback.py,
but it probably is full of distractions.)

On Thu, Feb 25, 2021 at 3:01 AM Irit Katriel 
wrote:

>
>
> On Thu, Feb 25, 2021 at 5:19 AM Guido van Rossum  wrote:
>
>> [Subthread: handling individual errors]
>>
>> > Rather than iterator, I think we should add a visitor that calls a
>> function for each leaf exception,
>>
>> I agree that we shouldn't add an `__iter__` method. But what if we added
>> a separate method that returns an iterator? Then you could write
>> ```
>> for e in eg.something():
>> print("caught", e)
>> ```
>> This looks cleaner to me than having to write
>> ```
>> def handler(e):
>> print("caught", e)
>> eg.something(handler)
>> ```
>> (Does my bias against callback functions shine through? :-)
>>
>
>
> I hear you. I suggested a visitor to make it a bit awkward to use because
> I'm not sure why people should iterate over individual exceptions in an
> ExceptionGroup, and I think that by providing an iteration utility we are
> implying that this is what you should do.
> So let me be more direct instead of proposing passive-aggressive APIs.
>
> Can we take a step back and talk about how we think people would want to
> handle ExceptionGroups?
>
> In the rejected ideas section we discuss this a bit, with a reference to
> Yury's writeup:
> https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284
>
> TL;DR:  when you get a group of exceptions from asyncio or the like, you
> may want to query it for exception types is contains (with subgroup()), you
> may want to format it into a log (with traceback.* methods), but you are
> unlikely to care whether there are 1, 2 or 300 ValueErrors. Your program
> will probably do the same thing regardless. If you allowed your ValueError
> get collected into an ExceptionGroup you already lost the context in which
> it happened to it's unlikely that you can make a targeted recovery which is
> relevant to this particular exception.
>
> So, what is the use case for iterating over single exceptions?
>
>
>>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KA6PVBBGZUNJ43YF44C63SAPQOODUPJH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

One aspect of conda packaging that's worth adopting is the 'test' block of
the meta.yaml which contains commands necessary to *test* the built package.

You can add a pytest command to setup.py, but it's not as easy as:
https://github.com/conda-forge/python-feedstock/blob/master/recipe/meta.yaml#L227-L291
:

```yaml
test:
  downstreams:
- cython
- setuptools
  requires:
- ripgrep
- cmake
- ninja
- {{ compiler('c') }}
# Tried to use enable_language(C) to avoid needing this. It does
not work.
- {{ compiler('cxx') }}
  files:
- tests/distutils/*
- tests/cmake/*
- tests/cython/*
- tests/prefix-replacement/*
  commands:
- echo on  # [win]
- set  # [win]
- python -V
- python3 -V# [not win]
- 2to3 -h
- pydoc -h
- python3-config --help  # [not win]
- set "PIP_NO_BUILD_ISOLATION=False"  # [win]
- set "PIP_NO_DEPENDENCIES=True"  # [win]
- set "PIP_IGNORE_INSTALLED=True"  # [win]
- set "PIP_NO_INDEX=True"  # [win]
- set "PIP_CACHE_DIR=%CONDA_PREFIX%/pip_cache"  # [win]
- set "TEMP=%CONDA_PREFIX%/tmp"  # [win]
- mkdir "%TEMP%"  # [win]
- python -Im ensurepip --upgrade --default-pip  # [win]
# tzdata/zoneinfo test that will need the tzdata package to pass
- python -c "from zoneinfo import ZoneInfo; from datetime import
datetime; dt = datetime(2020, 10, 31, 12,
tzinfo=ZoneInfo('America/Los_Angeles')); print(dt.tzname())"
- python -m venv test-venv
- python -c "import sysconfig;
print(sysconfig.get_config_var('CC'))"  # [not win]
-
_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
python -c "import sysconfig; print(sysconfig.get_config_var('CC'))"  #
[linux64]
# check for unreplaced @ symbols in sysconfig files
- for f in ${CONDA_PREFIX}/lib/python*/_sysconfig*.py; do echo
"Checking $f:"; if [[ `rg @ $f` ]]; then echo "FAILED ON $f"; cat $f; exit
1; fi; done  # [linux64 or osx]
- test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.a  # [unix]
- test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.nolto.a  #
[unix]
# https://github.com/conda-forge/python-feedstock/issues/384
- if exist %PREFIX%\\Scripts\\pydoc exit 1  # [win]
- if exist %PREFIX%\\Scripts\\idle exit 1  # [win]
- if exist %PREFIX%\\Scripts\\2to3 exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\pydoc-script.py exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\idle-script.py exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\2to3-script.py exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\idle.exe exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\2to3.exe exit 1  # [win]
- if not exist %PREFIX%\\Scripts\\pydoc.exe exit 1  # [win]
- pushd tests
-   pushd distutils
- python setup.py install -v -v
- python -c "import foobar"
-   popd
-   pushd prefix-replacement  # [unix]
- bash build-and-test.sh  # [unix]
-   popd  # [unix]
-   pushd cmake
- cmake -GNinja -DPY_VER={{ version }}
  # --trace --debug-output --debug-trycompile .
-   popd
- popd
- test ! -f default.profraw   # [osx]
```


In which package you then store integration tests for your whole
application and it's integrated dependencies is then up to you.
Unfortunately, we often don't include tests/ in our packages, so it's
literally not possible to run the tests in production even if you have
pytest_requires installed.

Perhaps distros would do well to implement support for integration and
per-package tests.

On Thu, Feb 25, 2021, 12:55 Fred Drake  wrote:

> On Thu, Feb 25, 2021 at 5:35 AM Wes Turner  wrote:
>
>> The challenge with version conflicts is often less that you need to go
>> update the constraints (which has little to do with sysadmin'ing, TBH) and
>> more that you have insufficient *integration tests* and you're relying upon
>> something else running the per-package tears.
>>
>
> Sometimes, auto-correct really does understand!
>
> Your point is right on target, and really can't be understated.
>
>
>   -Fred
>
> --
> Fred L. Drake, Jr.
> "There is nothing more uncommon than common sense."
>   --Frank Lloyd Wright
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2QMXHSDF5L4QEJU5C2FVZ3AZHWEHUMXA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Mike Miller]

On 2021-02-24 21:08, Peter Wang wrote:
With binary extension modules, version conflicts lead to (at best) runtime 
segfault and (at worst) subtle *data* bugs that return incorrect results.  There 
are also deeper concerns around security and reproducibility.


If your app has requirements, by all means use an isolated environment.  Wasn't 
recommending against it, when needed.


Rather, that it is not a requirement for *every single script,* or even most. 
As we've seen some folks have been scared into a heavy-handed solution.




Perhaps it has to do with
the decline of sys-admin skills over the years?

Many millions of users of new Python users show up every year, using the 
language and its powerful ecosystem for data analytics and scientific computing, 
and they have no hope of having sys-admin skills.



"sys-admin" is a bit of an overstatement in my phrasing.  The core is that you 
need to understand how a PATH works and be able to run pip and cd into folders 
and perhaps run rm/del, etc.  Basic command-line skills?


In any case, if you can't do these, virtual envs are not a full solution either.

-Mike
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PWQDTGRQDQWPB6NBEVKDXVRECHQHYSSI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 637 - Support for indexing with keyword arguments: request for feedback for SC submission

[Brett Cannon]

On Thu, Feb 25, 2021 at 8:46 AM Thor Whalen  wrote:

> Finally! One of my top python wishes!
>
> Where can I vote?
>

There is no explicit voting, but thanks for sharing your opinion!


>
> How can I get my hands on a back port?
>

Probably can't since it will be new to 3.10 if the PEP gets accepted.


>
> How can I help getting this sooner?
>

As of right now there's nothing to do as the steering council has not even
begun reviewing the PEP yet. If the PEP gets accepted then you can
volunteer to help with the implementation and/or the review of the code.

-Brett


> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2QPAZ26E77MV2AZLDF3Z6DNHWZHQUPH5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HWNMML442Q72LQVPUCAYAI3BTKLN4Y2U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Fred Drake]

On Thu, Feb 25, 2021 at 5:35 AM Wes Turner  wrote:

> The challenge with version conflicts is often less that you need to go
> update the constraints (which has little to do with sysadmin'ing, TBH) and
> more that you have insufficient *integration tests* and you're relying upon
> something else running the per-package tears.
>

Sometimes, auto-correct really does understand!

Your point is right on target, and really can't be understated.


  -Fred

-- 
Fred L. Drake, Jr.
"There is nothing more uncommon than common sense."
  --Frank Lloyd Wright
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KZX2DOJBICJWJ3Y6EMUA2F675TMSYSPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 637 - Support for indexing with keyword arguments: request for feedback for SC submission

[Thor Whalen]

Finally! One of my top python wishes!

Where can I vote? 

How can I get my hands on a back port?

How can I help getting this sooner? 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2QPAZ26E77MV2AZLDF3Z6DNHWZHQUPH5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 5:19 AM Guido van Rossum  wrote:

> [Subthread: handling individual errors]
>
> > Rather than iterator, I think we should add a visitor that calls a
> function for each leaf exception,
>
> I agree that we shouldn't add an `__iter__` method. But what if we added a
> separate method that returns an iterator? Then you could write
> ```
> for e in eg.something():
> print("caught", e)
> ```
> This looks cleaner to me than having to write
> ```
> def handler(e):
> print("caught", e)
> eg.something(handler)
> ```
> (Does my bias against callback functions shine through? :-)
>


I hear you. I suggested a visitor to make it a bit awkward to use because
I'm not sure why people should iterate over individual exceptions in an
ExceptionGroup, and I think that by providing an iteration utility we are
implying that this is what you should do.
So let me be more direct instead of proposing passive-aggressive APIs.

Can we take a step back and talk about how we think people would want to
handle ExceptionGroups?

In the rejected ideas section we discuss this a bit, with a reference to
Yury's writeup:
https://github.com/python/exceptiongroups/issues/3#issuecomment-716203284

TL;DR:  when you get a group of exceptions from asyncio or the like, you
may want to query it for exception types is contains (with subgroup()), you
may want to format it into a log (with traceback.* methods), but you are
unlikely to care whether there are 1, 2 or 300 ValueErrors. Your program
will probably do the same thing regardless. If you allowed your ValueError
get collected into an ExceptionGroup you already lost the context in which
it happened to it's unlikely that you can make a targeted recovery which is
relevant to this particular exception.

So, what is the use case for iterating over single exceptions?


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GYSBF6MZDAYG7SMRZGTDXTXY3TYWZAT3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 654 -- Exception Groups and except* : request for feedback for SC submission

[Irit Katriel via Python-Dev]

On Thu, Feb 25, 2021 at 5:59 AM Guido van Rossum  wrote:

>
> Here's a potentially alternative plan, which is also complex, but doesn't
> require asyncio or other use cases to define special classes. Let's define
> two exceptions, BaseExceptionGroup which wraps BaseException instances, and
> ExceptionGroup which only wraps Exception instances. (Names to be
> bikeshedded.) They could share a constructor (always invoked via
> BaseExceptionGroup) which chooses the right class depending on whether
> there are any non-Exception instances being wrapped -- this would do the
> right thing for split() and subgroup() and re-raising unhandled exceptions.
>
> Then 'except Exception:' would catch ExceptionGroup but not
> BaseExceptionGroup, so if a group wraps e.g. KeyboardError it wouldn't be
> caught (even if there's also e.g. a ValueError among the wrapped errors).
>
> Pseudo-code:
>
> class BaseExceptionGroup(BaseException):
> def __new__(cls, msg, errors):
> if cls is BaseExceptionGroup and all(isinstance(e, Exception) for
> e in errors):
> cls = ExceptionGroup
> return BaseException.__new__(cls, msg, errors)
>
> class ExceptionGroup(Exception, BaseExceptionGroup):
> pass
>


This could be a valid compromise.

We keep the ability to wrap any exception, while we lose the "fail-fast if
you forget to handle an ExceptionGroup" feature, which was intended as a
kindness towards those who abuse "except Exception".

If we adopt this solution then letting an ExceptionGroup escape from code
that is not supposed to raise it, is not a fatal error, it's just some
exception like any other.
So there is no longer a distinction between code that raises
ExceptionGroups and code that doesn't. Any code can propagate them, like
any code can raise any other exception.
Does this mean that more code needs to be aware of the possibility of them
showing up?  Is that a problem?  Maybe this a simpler state of affairs
overall.

What would we have done here if we were building Python from scratch?

Irit
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L3GKLVTZWICQYH5JCJRGQ6K5QX7EWIZX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Steering Council update for January

[Thomas Wouters]

The Steering Council just published the community update for January
(covering November/December as well):
https://github.com/python/steering-council/blob/master/updates/2021-01-steering-council-update.md
(included verbatim below). We'll be trying to keep this up monthly from now
on. I'll note that January was a very busy month with a lot happening, but
not a lot of long-term planning. I hope we'll be able to get back to that
soon.

# November

- Steering Council discussed PEP 634 (Structural Pattern Matching) and
  decided it would not make a decision on it, since this SC is on its way
  out. Instead, the SC will make a recommendation to the next Steering
  Council. Brett sent a note to the PEP authors.
- SC reviewed Ewa's proposed CPython sponsorship benefits and everyone is
  fine with it as is. We will have each SC review the benefits especially
  the benefit pertaining to meeting with the Steering Council.
- SC discussed the internship priority update to [PEP
  594](https://www.python.org/dev/peps/pep-0594/) and everyone is fine with
  it.
- SC discussed the draft response to the proposal for funded CPython
  optimisation work, and decided it was fine to send. Thomas can also
  publish his blog. Thomas will wait for Victor to chime in on the Community
  Updates before sending them out.
- SC discussed the Pattern Matching PEPs again, and put out a
  discuss.python.org poll to see if there is consensus among core
  developers.
- SC discussed adding platform.freedesktop_os_release()
  (https://bugs.python.org/issue28468), and approved it.
- SC discussed PEPs 634-636 (Structural Pattern Matching) yet again. The SC
  decided to recommended to the 2021 SC that they accept PEP 634. Brett
  [drafted an announcement](
https://mail.python.org/archives/list/steering-coun...@python.org/thread/I745B6GLVRDA42ZI5HMCN3YOQG5ZXEY3/
).
- Google sponsored the Core Dev sprint and chose the SC chat as its reward.
  Thomas is going to schedule a chat between the SC and Google for Dec 14

# December
- Steering Council worked on a final announcement to python-dev@ about PEP
  634.
- SC voted on PEP 632 and will get Victor's vote before Brett sends a note
  to the PEP author.
- SC is fine with Barry's & Brett's response to
  https://github.com/python/steering-council/issues/41 so Brett closed the
  issue.
- Barry gave an overview of the PSF, SC meeting with Seagate.
- SC discussed the elections and possible improvements that would be made to
  PEP 13 around the election timelines, staggered terms. SC also decided it
  would not have anymore meetings in 2020. Ewa will send a Doodle out for
  the first week of January once the new SC members are known for a handoff
  meeting.
- SC had their first sponsor meeting with Google per their Core Dev Sprint
  sponsorship.

# January

## January 7
- The Steering Council had a hand-off meeting with both Victor and Pablo,
  covering current goings on and how the SC normally operates.
- SC decided on communication platform for meetings and out of bounds
  communications.
- SC discussed PEP 632, deprecating distutils.
- SC discussed PEP 634/635/636, Structural Pattern Matching, and brought
  Pablo up to date on previous discussions.
- SC discussed community updates, decided to aim for once a month. Pablo and
  Thomas will alternate duty on this.
- SC collectively and individually thanked Victor for all his work on the
  SC.

## January 11
- Carol [shared the Documentation WG charter](
https://docs.google.com/document/d/175bjJA7hKRaqvLjiRXSvUxKgGWGJ5RMuW-jYKUsXm8c/edit#heading=h.4jjd55oa8ox7
)
  and Steering Council members should review it.
- SC reviewed the Developer-in-Residence job description that is left over
  from the CZI grant application. The Steering Council decided how this role
  needs to do certain analysis and capture certain metrics to determine
  where the impact will be. Ewa will update the job description.
- SC discussed delaying the removal of ABC aliases from collections. It was
  decided it would be removed in 3.10 as scheduled. Pablo will comment on
  the issue.
- SC discussed how to handle differences between accepted PEPs and
  implementation. It was decided that any changes should be decided between
  at least two people (i.e. the author and PEP delegate). If an agreement
  cannot be reached, it should be sent to the Steering Council for review.

## January 18
- Steering Council reviewed PEP 632 (Deprecating distutils) and agreed
  distutils should be deprecated. Brett will email Barry (who was absent
  from the meeting) for his input.
- SC reviewed PEP 634 (Structural Pattern Matching) and decided that the
  major hold up is the load/store confusion. Brett will ask if Guido could
  join next week to discuss this.
- SC reviewed PEP 642 (Explicit Pattern Syntax for Pattern Matching), and
  decided to reject it. Brett will email Barry about it and Thomas will help
  draft the response to the PEP author.
- SC discussed how Debian (and thus Ubuntu) ships Python, in parti

[Python-Dev] Re: Have virtual environments led to neglect of the actual environment?

[Wes Turner]

The challenge with version conflicts is often less that you need to go
update the constraints (which has little to do with sysadmin'ing, TBH) and
more that you have insufficient *integration tests* and you're relying upon
something else running the per-package tears.

On Thu, Feb 25, 2021, 00:10 Peter Wang  wrote:

> On Wed, Feb 24, 2021 at 8:50 PM Mike Miller 
> wrote:
>
>> I never understood the fear around version conflicts.
>
>
> With binary extension modules, version conflicts lead to (at best) runtime
> segfault and (at worst) subtle *data* bugs that return incorrect results.
> There are also deeper concerns around security and reproducibility.
>
>
>> Perhaps it has to do with
>> the decline of sys-admin skills over the years?
>
>
> Many millions of users of new Python users show up every year, using the
> language and its powerful ecosystem for data analytics and scientific
> computing, and they have no hope of having sys-admin skills.
>
> -Peter
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TYZ6QT4BVQY7SVXV6E63YKNV6SCNFZ7V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O2TB7YIP6NHTBORZJNFFNU45VV2C26OV/
Code of Conduct: http://python.org/psf/codeofconduct/