Re: [Python-Dev] Better support for consuming vendored packages

[Barry Warsaw]

On Mar 22, 2018, at 12:33, Oleg Broytman <p...@phdru.name> wrote:
> 
> On Thu, Mar 22, 2018 at 12:30:02PM -0700, Barry Warsaw <ba...@python.org> 
> wrote:
>> Developers are mostly going to use pip, and maybe a requirements.txt,
> 
>   +virtual envs to avoid problems with global site-packages.

Yep, that was implied but of course it’s better to be explicit. :)

>   IMO virtualenv for development and frozen app for distribution solve
> the problem much better than vendoring.

+1
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Better support for consuming vendored packages

[Barry Warsaw]

On Mar 22, 2018, at 09:58, Gregory Szorc  wrote:
> 
> Not all consumers of Python packages wish to consume Python packages in the 
> common `pip install ` + `import ` manner. Some Python 
> applications may wish to vendor Python package dependencies such that known 
> compatible versions are always available.

It’s important to understand how painful vendoring is to some downstream 
consumers.  Debian is a good example.  There we often have to go through a lot 
of hoops to unvendor packages, both for policy and for good distribution 
practices.  The classic example is a security vulnerability in a library.  It’s 
the distro’s responsibility to fix that, but in the face of vendored 
dependencies, you can’t just patch the system package.  Now you also have to 
hunt down all the vendored versions and figure out if *they’re* vulnerable, 
etc.  It certainly doesn’t help that you can easily have vendored libraries 
vendoring their own dependencies.  I think I found one application in Debian 
once that had like 4 or 5 versions of urllib3 inside it!

You mention dependency hell for downstream consumers like a Linux distro, but 
this type of integration work is exactly the job of a distro.  They have to 
weigh the health and security of all the applications and libraries they 
support, so it doesn’t bother me that it’s sometimes challenging to work out 
the right versions of library dependencies.  It bothers me a lot that I have to 
(sometimes heavily) modify packages to devendorize dependencies, especially 
because it’s not always clearly evident that that has happened.

That said, I completely understand the desire for application and library 
authors to pin their dependency versions.  We’ve had some discussions in the 
past (not really leading anywhere) about how to satisfy both communities.  I 
definitely don’t go so far as to discourage global imports, and I definitely 
don’t like vendoring all your dependencies.  For applications distributed 
outside of a distro, there are lots of options, from zip apps (e.g. pex) to 
frozen binaries, etc.

Developers are mostly going to use pip, and maybe a requirements.txt, so I 
think that use case is well covered.  Downstream consumers need to be able to 
easily devendorize, but I think ultimately, the need to vendorize just points 
to something more fundamental missing from Python’s distribution and import 
system.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Any way to only receive emails for threads that I am participating in?

[Barry Warsaw]

On Mar 2, 2018, at 20:41, Dan Stromberg  wrote:
> 
> Maybe gmane combined with a threaded newsreader (NNTP client) would be
> close enough?

While I guzzle the firehose of python-dev from my inbox, Gmane+NNTP is how I 
read many other Python mailing lists, including python-ideas.  It’s a great 
solution if you want to participate in just a few threads.  I personally use 
Postbox for that, but there good free open source MUAs too, depending on your 
platform.  My favorite on Linux is Claws Mail.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Barry Warsaw]

On Feb 22, 2018, at 09:09, Eric Snow  wrote:
> 
> I had exactly that experience on one particularly large Go project (on
> GitHub, with slow CI, driven by bots).

One thing that’s nice to set up if you can is multiple, parallel, independent 
CI runs.  E.g. if your full test suite takes a long time to run, but you can 
run your code style tests pretty quickly (e.g. flake8 and mypy), then you can 
get more immediate feedback about your PR code style, and it’s not so painful 
if your test suite is slow.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Barry Warsaw]

> On Feb 22, 2018, at 08:08, Antoine Pitrou  wrote:
> 
> That's a fair point I hadn't considered.  OTOH the style issues I
> usually comment on as a reviewer aren't the kind that would be caught
> by an automated style check (I tend to ask for comments or docstrings,
> or be nitpicky about some variable or function name).  YMMV :-)

Those are aesthetic comments that are important, and are also difficult to 
automate.  What I really don’t want to comment on are things like whitespace 
(vertical, horizontal, trailing), import order or format, consistent 
indentation, etc.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Barry Warsaw]

On Feb 22, 2018, at 02:12, Antoine Pitrou  wrote:

> Everytime I contribute to a project which has automatic style checks in
> CI, I find myself having to push pointless "cleanup" commits because
> the CI is barking at me for some ridiculous reason (such as putting two
> linefeeds instead of one between two Python functions).  Then I have to
> wait some more until CI finishes, especially if builds take long or
> build queues are clogged.
> 
> Overall it makes contributing more of a PITA than it needs be.  Do
> automatic style *fixes* if you want, but please don't annoy me with
> automatic style checks that ask me to do tedious grunt work on my spare
> time.

Look at it from the other side though.  If I’m reviewing someone’s PR, and the 
style is so inconsistent with the existing code base, then I’m distracted by 
constantly asking the contributor to clean up little stylistic things.  This 
wastes my time as a reviewer, where I’d rather be thinking about the 
substantive content of the change.  To have no style checks can leave you with 
a chaotic jumbled ness that is much harder to navigate around, understand, and 
contribute to by others, and once you’ve gone down that path, you just build up 
technical debt to eventually try to clean it up.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Barry Warsaw]

On Feb 21, 2018, at 19:03, Dan Stromberg  wrote:
> 
> Is flake8 that much better than pylint, that pylint wouldn't even be 
> discussed?

I honestly don’t use pylint all that much these days, but when I was evaluating 
them years ago, I found pylint to be too noisy about not-incorrect code.  I 
also liked how flake8 has a very nice plugin system, and I use that on my 
personal projects to enforce a consistent style.  It’s certainly possible that 
both tools have advanced significantly since I last made my personal decision.

I agree with Guido in general though, that we’d need to do some actual analysis 
on the stdlib and see what the results are.  I do think that a plugin system, 
or really good configurability, will help us tailor whatever tool we decide to 
use, so that it’s only warning about the things we care about and ignoring the 
rest, without requiring lots of comment pragmas in the code.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The `for y in [x]` idiom in comprehensions

[Barry Warsaw]

On Feb 22, 2018, at 11:04, Serhiy Storchaka  wrote:
> 
> Stephan Houben proposed an idiom which looks similar to new hypothetic syntax:
> 
>result = [y + g(y) for x in range(10) for y in [f(x)]]
> 
> `for y in [expr]` in a comprehension means just assigning expr to y. I never 
> seen this idiom before, but it can be a good replacement for a hypothetic 
> syntax for assignment in comprehensions. It changes the original 
> comprehension less than other approaches, just adds yet one element in a 
> sequence of for-s and if-s. I think that after using it more widely it will 
> become pretty idiomatic.

My questions are 1) will this become idiomatic enough to be able to understand 
at a glance what is going on, rather than having to pause to reason about what 
that 1-element list-like syntax actually means, and 2) will this encourage even 
more complicated comprehensions that are less readable than just expanding the 
code into a for-loop?

for-loops-are-not-evil-ly y’rs,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making "Provisional" an explicit PEP state

[Barry Warsaw]

On Feb 20, 2018, at 22:42, Nick Coghlan  wrote:
> 
> In the current PEP workflow, provisionally accepted PEPs are marked as
> "Accepted", and remain in that state until they're declared stable and
> moved to Final.

I left a review on the PR, but the substance of the changes LGTM!

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] How is the GitHub workflow working for people?

[Barry Warsaw]

On Feb 21, 2018, at 13:22, Guido van Rossum  wrote:
> 
> I'm willing to reconsider if there's a good enough tool. Ditto for C code (or 
> do we already do it for C?).

For Python code, flake8 --possibly with our own custom plugins— is the way to 
go.  It would probably take some kind of ratchet or transition period before 
all of the stdlib were compliant.  We’d have to be careful of the inevitable 
raft of PRs to fix things, which may distract from actual bug fixes and 
improvements.  OTOH, that would be another external dependency pulled in for 
core Python development.

> FWIW I'm personally hugely happy with the new workflow -- my only regret is 
> that we're not using GitHub for issue tracking yet.

I’m very happy with the workflow too, and Mariatta’s and others work has been 
hugely important in making things work so well.  I still miss my “commit when 
CI completes” button, but oh well.

As for the bug tracker, I still do like Roundup, and we have a huge investment 
in it, not just in resources expended to make it rock, but also in all the 
history in it and everything that integrates with it.  I wouldn’t stop anybody 
who’s motivated to spearhead a move to GH issues, but I also don’t think that 
can be taken up lightly.  Just look at the vast amount of work Brett and others 
had to do to migrate code hosting.  It would be nice to have integrated reviews 
and issues (e.g. for auto-closing perhaps), but I also honestly don’t miss much 
with the current suite of tools.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is object the most base type? (bpo-20285)

[Barry Warsaw]

On Feb 2, 2018, at 02:25, Steven D'Aprano  wrote:
> 
> On Fri, Feb 02, 2018 at 01:53:00AM -0500, Terry Reedy wrote:
> object.__doc__
>> 'The most base type’

Clearly that’s a typo.  It should be: “The most bass type” as in: "It all 
starts with the bass, the most important part of any band or class hierarchy.  
Sure, the drums type is very important, and the bass type and drums type work 
hand-in-hand to provide you with a solid funky foundation on which to groove, 
er, code.  Don’t let those guitar types tell you anything else."

> "the base class of the class heirarchy"
> 
> "the root of the class heirarchy"

In all seriousness, either one of those works for me.  While the first one does 
repeat the word “class”, it seems the most clear description of what this thing 
is.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Why is Python for Windows compiled with MSVC?

[Barry Warsaw]

On Feb 1, 2018, at 04:19, Oleg Sivokon  wrote:
> 
> Oh, so this is the real reason... well, corporate interests are hard to argue 
> against.  But, this is an interesting statistic nevertheless.  Thanks for 
> letting me know.

Maybe it hasn’t happened because no volunteer has stepped up to do it.  Or 
maybe no corporation thinks it a good business investment to pay employees to 
do it.  What other options are there, and if none, then which of those can you 
solve?

Or to paraphrase a timeless quote: Ask not what Python can do for you, ask what 
you can do for Python.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guido's Python 1.0.0 Announcement from 27 Jan 1994

[Barry Warsaw]

On Jan 27, 2018, at 21:45, Guido van Rossum  wrote:
> 
> For me personally, the fondest memories are of 1.5.2, which Paul Everitt 
> declared, while we were well into 2.x territory, was still the best Python 
> ever. (I didn't agree, but 1.5.2 did serve us very well for a long time.)

What, not the Contractual Obligation release? :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Welcome the 3.8 and 3.9 Release Manager - Łukasz Langa!

[Barry Warsaw]

On Jan 27, 2018, at 17:04, Guido van Rossum > wrote:
> 
> Hardly a surprising choice! Congrats, Łukasz. (And never forget that at every 
> Mac OS X upgrade I have to install the extended keyboard just so I can type 
> that darn Ł. :-)

Heh, I *just* learned that, at least on macOS High Sierra (and probably going 
back several releases), on a US keyboard you can press and hold the ‘L’ (cap-L) 
key.  A little popup will appear like the attached image (if this doesn’t get 
stripped by Mailman).  Hit ‘1’ and the slashy-L will get entered: Ł.

Cheers
-Barry


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Welcome the 3.8 and 3.9 Release Manager - Łukasz Langa!

[Barry Warsaw]

As Ned just announced, Python 3.7 is very soon to enter beta 1 and thus feature 
freeze.  I think we can all give Ned a huge round of applause for his amazing 
work as Release Manager for Python 3.6 and 3.7.  Let’s also give him all the 
support he needs to make 3.7 the best version yet.

As is tradition, Python release managers serve for two consecutive releases, 
and so with the 3.7 release branch about to be made, it’s time to announce our 
release manager for Python 3.8 and 3.9.

By unanimous and enthusiastic consent from the Python Secret Underground (PSU, 
which emphatically does not exist), the Python Cabal of Former and Current 
Release Managers, Cardinal Ximénez, and of course the BDFL, please welcome your 
next release manager…

Łukasz Langa!

And also, happy 24th anniversary to Guido’s Python 1.0.0 announcement[1].  It’s 
been a fun and incredible ride, and I firmly believe that Python’s best days 
are ahead of us.

Enjoy,
-Barry

[1] 
https://groups.google.com/forum/?hl=en#!original/comp.lang.misc/_QUzdEGFwCo/KIFdu0-Dv7sJ



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GH-NNNN vs #NNNN in merge commit

[Barry Warsaw]

On Jan 25, 2018, at 16:55, Mariatta Wijaya  wrote:

> My problem has been that I almost always still need to rewrite the commit 
> message.
> Especially when someone wrote "fix a typo" or "fix several typos".
> 
> If it automatically merges, then there's no opportunity to adjust the commit 
> message.

Right, that’s the bit that’s different here than the GitLab workflow.  In the 
latter, you can set the commit message when you click on “merge when CI 
succeeds”.

> So I suggest the option to provide the proper commit message to the mergebot.
> If not provided, I guess we'll use the GitHub PR title and description.

I think that’d be fine actually.  When I merge things I almost always just 
delete most of the default commit message anyway.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GH-NNNN vs #NNNN in merge commit

[Barry Warsaw]

On Jan 25, 2018, at 13:38, Mariatta Wijaya  wrote:
> 
> +1 for the mergebot! :)

Yes, +1 from me too.  As you know, GitLab has the option to “merge when CI 
completes successfully” and it’s a great workflow.  Once I’ve reviewed and 
approved the branch, I can hit this button and… we’re done!  Assuming of course 
that no additional commits get pushed (that’s actually configurable I think), 
and that CI doesn’t fail.  I’d be very happy to see how close we can get to 
that with GitHub and a mergebot.

I don’t know that we need a @mergebot mention though.  Why not just auto merge 
if the PR is approved, CI is all green, and no additional commits have been 
pushed?  I suppose the reason would be because in GH, you can’t modify the 
commit message any other way pre-merge.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unique loader per module

[Barry Warsaw]

On Jan 05, 2018, at 05:12 PM, Nick Coghlan wrote:

>I think the main reason you're seeing a problem here is because
>ResourceReader has currently been designed to be implemented directly
>by loaders, rather than being a subcomponent that you can request
>*from* a loader.
>
>If you instead had an indirection API (that could optionally return
>self in the case of non-shared loaders), you'd keep the current
>resource reader method signatures, but the way you'd access the itself
>would be:
>
>resources = module.__spec__.loader.get_resource_reader(module)
># resources implements the ResourceReader ABC

BTW, just as a quick followup, this API suggestion was brilliant, Nick.  It
solved the problem nicely, and let me add support for ResourceReader to
zipimport with only touching the bare minimum of zipimport.c. :)

https://github.com/python/cpython/pull/5248

Cheers,
-Barry


pgpmPkp2zc3oF.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Positional-only parameters in Python

[Barry Warsaw]

On Jan 17, 2018, at 08:14, Serhiy Storchaka  wrote:
> 
> The main problem -- designing a syntax that does not look ugly. I think there 
> are too small time is left before features freezing for experimenting with 
> it. It would be better to make such changes at the early stage of development.

A PEP is definitely needed.  Can someone get a PEP written, approved, with the 
feature implemented in 12 days?  As much as I like the idea, I think it’s 
unlikely.  But hey, you never know so if someone is really motivated, go for 
it!  The effort won’t be wasted in any case, since 3.8’s right around the 
corner .

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?

[Barry Warsaw]

On Jan 13, 2018, at 12:06, Christian Heimes  wrote:

> These days a lot of packages are using setuptools' entry points to
> create console scripts. Entry point have no option to create a console
> script with -s or -I flag. On my system, only 40 out of 360 scripts in
> /usr/bin have -s or -I.

-I should be the default for system scripts; it’s not on Debian/Ubuntu though 
unfortunately.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?

[Barry Warsaw]

On Jan 13, 2018, at 11:12, Miro Hrončok  wrote:
> 
> We would very much like to see --user the default rather than having it 
> removed.

Very much +1.  In Debian/Ubuntu we’ve carried patches to do exactly that for 
years, and I think our users have been very happy about it.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Whatever happened to 'nonlocal x = y'?

[Barry Warsaw]

On Jan 6, 2018, at 11:43, Guido van Rossum  wrote:
> 
> Maybe we should not delete them outright but add something like "(UPDATE: 
> during later discussions it was decided that this feature shouldn't be 
> added.)"

+1

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Unique loader per module

[Barry Warsaw]

We have what I think is one last design question for importlib.resources.

https://gitlab.com/python-devs/importlib_resources/issues/49

The problem is that the ResourceReader ABC omits the package from the function 
signatures, so that on a compatible loader, you only need to specify the 
resource you are interested in.

This is fine for file loaders because every package will have a unique loader 
instance associated with it, so it will know which package the requested 
resource is homed to.

But AFAICT, there is no specification or requirement in Python that every 
module/package have a unique loader instance.  In fact, it’s not unreasonable 
given some of the text in PEP 302 to think that loader instances can be shared. 
 The PEP says "In many cases the finder and loader can be one and the same 
object: finder.find_module() would just return self” and you aren’t going to 
typically have a unique finder per module, so that would implied a shared 
loader per finder.

We even have an existence proof in the zip importer:

>>> import test.test_importlib.zipdata02
>>> import sys, os
>>> sys.path.insert(0, 
>>> os.path.join(os.path.dirname(test.test_importlib.zipdata02.__file__), 
>>> 'ziptestdata.zip'))
>>> import ziptestdata.two
>>> import ziptestdata.one
>>> ziptestdata.one.__spec__.loader == ziptestdata.two.__spec__.loader
True

The issue above proposes two solutions.  The first is to change the ABC so that 
it includes the package argument in the ABC method signatures.  That way, a 
shared loader will know which package the requested resource is relative to.

Brett doesn’t like this, for several reasons (quoting):

1. redundant API in all cases where the loader is unique to the module
2. the memory savings of sharing a loader is small
3. it's implementation complexity/overhead for an optimization case.

The second solution, and the one Brett prefers, is to reimplement zip importer 
to not use a shared loader.  This may not be that difficult, if for example we 
were to use a delegate loader wrapping a shared loader.

The bigger problem IMHO is two-fold:

1. It would be backward incompatible.  If there’s any code out there expecting 
a shared loader in zipimport, it would break
2. More problematic is that we’d have to impose an additional requirement on 
loaders - that they always be unique per module, contradicting the advice in 
PEP 302

The reason for this is third party finder/loaders.  Sure, we can fix zipimport 
but any third party finder/loaders could have the same problem, and they’d be 
within their rights given the current specification.  We’d have to prohibit 
that, or at least say that any third party finder/loaders that shared their 
loader can’t implement ResourceReader (which would be the practical effect 
anyway).  I think that would be a shame.

So while I agree with Brett that it’s uglier, and that once we decide we’re 
essentially locked into the API, I don’t see a whole lot of options.

Thoughts, feedback, suggestions are welcome.
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Documenting types outside of typing

[Barry Warsaw]

On Dec 27, 2017, at 18:59, Ivan Levkivskyi  wrote:
> 
> FWIW the same problem was discussed a year ago when documenting typing. At 
> that time the discussion was not conclusive,
> so that some types use class:: directive while other use data:: directive. At 
> that time Guido was in favour of data:: and now in view of
> PEP 560 many types in typing will stop being class objects, and will be just 
> (compact) objects. Therefore, my understanding is that
> all special forms like Union, Any, ClassVar, etc. will use data:: in the docs.

Thanks.  I see that typing.rst has been updated to use data:: so I’ll change my 
branch accordingly.

> Concerning the question whether it makes to document types, I think it makes 
> sense if it is a publicly available type (or type alias)
> that will be useful to annotate user code.

Thanks, that’s my feeling about it too.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Documenting types outside of typing

[Barry Warsaw]

In his review of PR#4911, Antoine points to the documentation of two type 
definitions in importlib.resources, Package and Resource.

https://github.com/python/cpython/pull/4911/files#diff-2a479c407f7177f3d7cb876f244e47bcR804

One question is what markup to use for type definitions.  I’m using class:: 
because that’s what’s used in typing and there doesn’t seem to be any better 
alternative.

More to the point, Antoine questions whether these two types should be 
documented at all:

https://github.com/python/cpython/pull/4911#discussion_r158801065

"What I mean is that a class is supposed to specify concrete behaviour, but 
being a type, Package doesn't have any methods or attributes of its own. So I 
don't see the point of mentioning it in the docs.”

I suggest that they are worth documenting because they help to organize the 
discussion about what API is expected from the arguments to the functions, 
without having to duplicate that information in every function description.  I 
also think that since you’ll see those types in the code, they are worth 
documenting.  I don’t think you *lose* anything by including their 
documentation.

But Antoine makes a good point that we probably don’t have a lot of precedence 
here, so suggests we discuss it on python-dev to come up with some useful 
conventions.  I haven’t kept up on the dataclasses discussion, but given that 
types are important in that API too, have the same issues come up there and if 
so, how are they being handled?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Dec 19, 2017, at 20:32, Nathaniel Smith  wrote:

> I guess the underlying issue here is partly the question of what the
> pprint module is for. In my understanding, it's primarily a tool for
> debugging/introspecting Python programs, and the reason it talks about
> "valid input to the interpreter" isn't because we want anyone to
> actually feed the data back into the interpreter, but to emphasize
> that it provides an accurate what-you-see-is-what's-really-there view
> into how the interpreter understands a given object. It also
> emphasizes that this is not intended for display to end users; making
> the output format be "Python code" suggests that the main intended
> audience is people who know how to read, well, Python code, and
> therefore can be expected to care about Python's semantics.

pprint.pprint() is indeed mostly for debugging, but not always.  As an example 
of what will break if you change the sorting guarantee: in Mailman 3 the REST 
etag is calculated from the pprint.pformat() of the result dictionary before 
it’s JSON-ified.  If the order is changed, then it’s possible a client will 
have an incorrect etag for a structure that is effectively the same.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Dec 18, 2017, at 22:37, Nathaniel Smith  wrote:

> Wait, what? Why would changing pprint (so that it accurately reflects
> dict's new underlying semantics!) be a breaking change? Are you
> suggesting it shouldn't be changed in 3.7?

As others have pointed out, exactly because the current behavior is documented. 
 And we all know that if it’s documented (and often even if it’s not, but 
that’s besides the point here) it will be relied upon.

So we can’t change the default behavior.  But I have no problems conceptually 
with giving users options.  The devil is in the details though, e.g. should we 
special case dictionary sorting only?  Should we use a sort `key` to mirror 
sorted() and list.sort()?

We can figure those things out and whether it’s even worth doing.  I don’t 
think that’s PEP-worthy, so if someone is sufficiently motivated, please open 
an issue on bpo.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Dec 18, 2017, at 21:11, Chris Barker  wrote:

> Will changing pprint be considered a breaking change?

Yes, definitely.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is static typing still optional?

[Barry Warsaw]

On Dec 18, 2017, at 21:41, Chris Barker  wrote:
> 
> TL;DR:
> My point is that having type annotation syntax required for something in the 
> stdlib is a significant step toward "normalizing" type hinting in Python. 
> Whether that's a good idea or not is a judgement call, but it IS a big step.

This is something we’re discussing for importlib.resources:

https://bugs.python.org/issue32248#msg308495

In the standalone version, we’re using annotations for the Python 3 bits.  It 
would make our lives easier if we kept them for the stdlib version (applying 
diffs and keeping them in sync would be easier).  Brett says in the follow up:

"As for the type hints, I thought it was lifted such that new code could 
include it but we wouldn't be taking PRs to add them to pre-existing code?”

So, what’s the deal?

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New crash in test_embed on macOS 10.12

[Barry Warsaw]

On Dec 15, 2017, at 15:14, Raymond Hettinger  
wrote:
> 
> I saw this same test failure.  After a "make distclean", it went away.

Dang, not for me.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] New crash in test_embed on macOS 10.12

[Barry Warsaw]

I haven’t bisected this yet, but with git head, built and tested on macOS 
10.12.6 and Xcode 9.2, I’m seeing this crash in test_embed:

==
FAIL: test_bpo20891 (test.test_embed.EmbeddingTests)
--
Traceback (most recent call last):
  File "/Users/barry/projects/python/cpython/Lib/test/test_embed.py", line 207, 
in test_bpo20891
out, err = self.run_embedded_interpreter("bpo20891")
  File "/Users/barry/projects/python/cpython/Lib/test/test_embed.py", line 59, 
in run_embedded_interpreter
(p.returncode, err))
AssertionError: -6 != 0 : bad returncode -6, stderr is 'Fatal Python error: 
PyEval_SaveThread: NULL tstate\n\nCurrent thread 0x7fffcb58a3c0 (most 
recent call first):\n'

Seems reproducible across different machines (all running 10.12.6 and Xcode 
9.2), even after a make clean and configure.  I don’t see the same failure on 
Debian, and I don’t see the crashes on the buildbots.

Can anyone verify?

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mostly Official Python Development Container Image

[Barry Warsaw]

On Dec 10, 2017, at 19:36, Ryan Gonzalez  wrote:
> 
> Question: why is this using GitLab while CPython itself is using GitHub + 
> Travis?

Mostly because Brett gave me the option to use GitLab for importlib_resources, 
and this grew out of that.  Enjoy!

overturn-pep-507-ly y’rs,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Mostly Official Python Development Container Image

[Barry Warsaw]

As part of our work on importlib_resources, and with some fantastic help from 
Abhilash Raj, we now have a mostly official Python development container image 
that you can use for CI testing and other development purposes.

This image is based on Ubuntu 16.04 LTS and provides the latest stable releases 
of Python 2.7, and 3.4-3.6, along with a mostly up-to-date git checkout of 
master, currently Python 3.7 of course.  Once 3.7 is released to beta, we 
intend to track its release tarballs too.  Note that these are pristine builds 
of upstream releases, so they don’t have any of the Ubuntu or Debian platform 
changes.

We also install a few other commonly needed tools, like pip, git, unzip, wget, 
mypy, and tox.

We do *not* recommend this image for application deployment purposes; use this 
for development and testing only, please.

Here’s the project repo:

https://gitlab.com/python-devs/ci-images

We’re publishing this image automatically to quay.io, so you can pull the image 
in a .gitlab-ci.yml file to run tests against all these versions of Python.  
Here’s an example from the importlib_resources project:

https://gitlab.com/python-devs/importlib_resources/blob/master/.gitlab-ci.yml

We welcome contributors on the ci-images GitLab project!

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Announcing importlib_resources 0.1

[Barry Warsaw]

Brett and I have been working on a little skunkworks project for a few weeks, 
and it’s now time to announce the first release.  We’re calling it 
importlib_resources and its intent is to replace the “Basic Resource Access” 
APIs of pkg_resources with more efficient implementations based directly on 
importlib.

importlib_resources 0.1 provides support for Python 2.7, and 3.4-3.7.  It 
defines an ABC that loaders can implement to provide direct access to resources 
inside packages.  importlib_resources has fallbacks for file system and zip 
file loaders, so it should work out of the box in most of the places that 
pkg_resources is currently used.  We even have a migration guide for folks who 
want to drop pkg_resources altogether and adopt importlib_resources.  
importlib_resources explicitly does not support pkg_resources features like 
entry points, working sets, etc.  Still, we think the APIs provided will be 
good enough for most current use cases.

http://importlib-resources.readthedocs.io/

We are calling it “importlib_resources” because we intend to port this into 
Python 3.7 under a new importlib.resources subpackage, so starting with Python 
3.7, you will get this for free.  The API is going to officially be 
provisional, but I’ve already done an experimental port of at least one big 
application (I’ll let you guess which one :) and it’s fairly straightforward, 
if not completely mechanical unfortunately.  Take a look at the migration guide 
for details:

http://importlib-resources.readthedocs.io/en/latest/migration.html

We also intend to include the ABC in Python 3.7:

http://importlib-resources.readthedocs.io/en/latest/abc.html

You can of course `pip install importlib_resources`.

We’re hosting the project on GitLab, and welcome feedback, bug fixes, 
improvements, etc!

 * Project home: https://gitlab.com/python-devs/importlib_resources
 * Report bugs at: https://gitlab.com/python-devs/importlib_resources/issues
 * Code hosting: https://gitlab.com/python-devs/importlib_resources.git
 * Documentation: http://importlib_resources.readthedocs.io/

Cheers.
-Barry and Brett



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__

[Barry Warsaw]

On Dec 5, 2017, at 13:24, Guido van Rossum  wrote:

> But the whole point of the PEP is that it only warns about deprecations in 
> code over which the user has control -- likely __main__ is their own code, 
> and they *can* handle it.

I’m not so sure how true that is.  I have no sense of the relative popularity 
of hand crafted dunder-mains vs entry point crafted ones.  I know that in my 
own applications, I tend to use the latter (although pkg_resources performance 
issues bum me out).  But then you have applications like pex that use fairly 
complex hand crafted dunder-mains in their zip files.  In either case I don’t 
want consumers of my applications to have to worry about DeprecationWarnings, 
since *they* really can’t do anything about them.

All that to say I really don’t know what the right thing to do here is.  All of 
our fiddling with the reporting of DeprecationWarnings, not to mention 
PendingDeprecationWarnings and FutureWarnings feels like experimental shots in 
the dark, and I suspect we won’t really know if PEP 565 will be helpful, 
harmful, or neutral until it’s out in the wild for a while.  I suspect either 
that what we’re trying to accomplish really can’t be done, or that we really 
don’t have a good understanding of the problem and we’re just chipping away at 
the edges.

I know that’s unhelpful in deciding whether to accept the PEP or not.  In the 
absence of any clear consensus, I’m happy to trust Guido’s instincts or keep 
the status quo.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Barry Warsaw]

On Nov 29, 2017, at 12:40, David Mertz  wrote:

> I think some syntax could be possible to only "catch" some exceptions and let 
> others propagate.  Maybe:
> 
>val = name.strip()[4:].upper() except (AttributeError, KeyError): -1
> 
> I don't really like throwing a colon in an expression though.  Perhaps some 
> other word or symbol could work instead.  How does this read:
> 
>val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError)

I don’t know whether I like any of this  but I think a more natural 
spelling would be:

   val = name.strip()[4:].upper() except (AttributeError, KeyError) as -1

which could devolve into:

   val = name.strip()[4:].upper() except KeyError as -1

or:

   val = name.strip()[4:].upper() except KeyError # Implicit `as None`

I would *not* add any spelling for an explicit bare-except equivalent.  You 
would have to write:

   val = name.strip()[4:].upper() except Exception as -1

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Barry Warsaw]

> On Nov 28, 2017, at 15:59, Paul Moore <p.f.mo...@gmail.com> wrote:
> 
> On 28 November 2017 at 20:38, Barry Warsaw <ba...@python.org> wrote:

>> I had occasional to speak with someone very involved in Rust development.  
>> They have a process roughly similar to our PEPs.  One of the things he told 
>> me, which I found very interesting and have been mulling over for PEPs is, 
>> they require a section in their specification discussion how any new feature 
>> will be taught, both to new Rust programmers and experienced ones.  I love 
>> the emphasis on teachability.  Sometimes I really miss that when considering 
>> some of the PEPs and the features they introduce (look how hard it is to 
>> teach asynchronous programming).
> 
> That's a really nice idea. I'd like to see Python adopt something
> similar (even just as a guideline on how to write a PEP).

https://github.com/python/peps/issues/485

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's the status of PEP 505: None-aware operators?

[Barry Warsaw]

On Nov 28, 2017, at 15:31, Raymond Hettinger  
wrote:

> Put me down for a strong -1.   The proposal would occasionally save a few 
> keystokes but comes at the expense of giving Python a more Perlish look and a 
> more arcane feel.

I am also -1.

> One of the things I like about Python is that I can walk non-programmers 
> through the code and explain what it does.  The examples in PEP 505 look like 
> a step in the wrong direction.  They don't "look like Python" and make me 
> feel like I have to decrypt the code to figure-out what it does.

I had occasional to speak with someone very involved in Rust development.  They 
have a process roughly similar to our PEPs.  One of the things he told me, 
which I found very interesting and have been mulling over for PEPs is, they 
require a section in their specification discussion how any new feature will be 
taught, both to new Rust programmers and experienced ones.  I love the emphasis 
on teachability.  Sometimes I really miss that when considering some of the 
PEPs and the features they introduce (look how hard it is to teach asynchronous 
programming).

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 559 - built-in noop()

[Barry Warsaw]

On Nov 22, 2017, at 19:32, Victor Stinner  wrote:
> 
> Aha, contextlib.nullcontext() was just added, cool!

So, if I rewrite PEP 559 in terms of decorators it won’t get rejected?

from functools import wraps

def noop(func):
@wraps(func)
def wrapper(*args, **kws):
return None
return wrapper


@noop
def do_something_important(x, y, z):
return blah_blah_blah(x, y, z)


print(do_something_important(1, 2, z=3))

Cheers?
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Show DeprecationWarning in debug mode?

[Barry Warsaw]

On Nov 17, 2017, at 20:22, Victor Stinner  wrote:
> 
> Or maybe we should start adding new modes like -X
> all-warnings-except-PendingDeprecationWarning, -X
> I-really-really-love-warnings and -X warnings-hater, as Barry
> proposed?

Well, if I can’t convince you about a `-X the-flufls-gonna-gitcha` mode, +1 for 
turning on all those other warnings in debug mode.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a developer mode to Python: -X dev command line option

[Barry Warsaw]

On Nov 16, 2017, at 09:47, Yury Selivanov  wrote:

> Let's keep it simple.  I'm big -1 on adding different "debug levels",
> they are always confusing.

Oh, this one’s easy.

-X dev   == some debugging
-X deve  == a little more
-X devel == give it to me!
-X develo== now you’re talking (literally)
-X develop   == thank you sir, may I have another?
-X develope  == here comes the flood
-X developer == needle inna haystack!

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a developer mode to Python: -X dev command line option

[Barry Warsaw]

On Nov 15, 2017, at 21:57, Victor Stinner  wrote:
> 
> Since Brett and Nick like the idea and nobody complained against it, I
> implemented the -X dev option:

Cool!  What would you think about printing a summary of the settings under the 
standard banner when you run the REPL under -X dev?  I’d rather not have to 
look it up in some obscure docs page whenever I use it.  If not that, then what 
about having a -msettings module or some such that prints it out?

-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a developer mode to Python: -X dev command line option

[Barry Warsaw]

On Nov 16, 2017, at 06:53, Victor Stinner  wrote:

> What do you think? Is it ok to include asyncio in the global "developer mode"?

I’m +1 on that, and the performance hit doesn’t bother me for a developer mode.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 8, 2017, at 23:57, Nick Coghlan  wrote:

> Putting that quibble first: could we adjust the feature flag to be
> either "from __future__ import lazy_annotations" or "from __future__
> import str_annotations"?
> 
> Every time I see "from __future__ import annotations" I think "But
> we've had annotations since 3.0, why would they need a future
> import?".

+1 for lazy_annotations for the same reason.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Barry Warsaw]

On Nov 9, 2017, at 07:27, Tres Seaver  wrote:

> IIUC, that would be as expected:  you would see the warnings when running
> your test suite exercising that imported code (which should run with all
> warnings enabled), but not when running the app.
> 
> Seems like a reasonable choice to me.

I’m coming around to that view too.  FWIW, I definitely do want to see the 
DeprecationWarnings in libraries I use, even if I didn’t write them.  That 
let’s me help that package’s author identify them, maybe even provide a fix, 
and let’s me evaluate whether maybe some other library is better suited to my 
needs.  It probably does strike the right balance to see that in my own test 
suite only.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Enabling depreciation warnings feature code cutoff

[Barry Warsaw]

On Nov 8, 2017, at 16:10, Nick Coghlan  wrote:

> The rationale for that change was so that end users of applications
> that merely happened to be written in Python wouldn't see deprecation
> warnings when Linux distros (or the end user) updated to a new Python
> version.

Instead they’d see breakage as DeprecationWarnings turned into errors. :(

I’m not saying that Python apps, regardless of who they are provided by, should 
expose DeprecationWarnings to their end users.  I actually think it’s good that 
they don’t because I don’t think most users care if their apps are written in 
Python, and wouldn’t know what to do about those warnings anyway.  And they do 
cause anxiety.

I suppose there are lots of ways to do this, but at least I’m pretty sure we 
all agree that end users shouldn’t see DeprecationWarnings, while developers 
should.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 8, 2017, at 12:02, Guido van Rossum  wrote:
> 
> I hadn't seen that, but it requires too much cooperation of library owners.

Actually, mostly just setuptools and as Paul points out, pip.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 8, 2017, at 08:47, Guido van Rossum  wrote:
> 
> You seem to have missed Nick's posts where he clearly accepts that a middle 
> ground is necessary. R D Murray is also still unconvinced. (And obviously I 
> myself am against reverting to the behavior from 7 years ago.) If we can't 
> agree on some middle ground, the status quo will be maintained.

I haven’t seen a response to my suggestion, so it’s possible that it got missed 
in the flurry.  With coordination with setuptools, we could:

* Re-enable DeprecationWarning by default
* Add a simplified API for specifically silencing DeprecationWarnings
* Modify setuptools to call this API for generated entry point scripts

I think this would mean that most application users would still not see the 
warnings.  The simplified API would be available for handcrafted scripts to 
call to accomplish the same thing the setuptools enhancement would provide.  
Developers would see DeprecationWarnings in their development and test 
environments.

The simplified API would be the equivalent of ignore::DeprecationWarning, so 
with some additional documentation even versions of applications running on 
versions of Python < 3.7 would still have an “out”.  (Yes, the simplified API 
is just a convenience moving forward.)

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

Antoine Pitrou wrote:
> On Mon, 6 Nov 2017 23:23:25 +1000
>> - tweak the default warning filters to turn DeprecationWarning back on
>> for __main__ only
> 
> Thats sounds error-prone.  I'd rather have them on by default
> everywhere.

If DeprecationWarnings were on by default, and setuptools were modified
to silence them in entry point generated mains, and we had a simple API
to easily silence them for manually written mains, wouldn't that handle
the majority of relevant use cases nicely?

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 16:15, Chris Barker - NOAA Federal  
wrote:

> Actually, there is a LOT of code out there that expects reference counting. I 
> know a lot of my code does. So if cPython does abandon it some day, there 
> will be issues.

I see this all the time in code reviews:

content = open(some file).read()

I never let that go uncommented.

So while you’re right that CPython is the reference implementation, and few 
people read the language spec, it’s still encombunt on us to point out broken 
code, code with implicit assumptions, and code that is not portable between 
implementations.  Having the reference manual to point to chapter and verse is 
critical to avoid Python just devolving into an ad-hoc language ruled by its 
most popular implementation.  This is something I believe Guido realized early 
on when JPython was first invented, and it was an important distinction that 
Python held, e.g. versus Perl.  I still believe it’s an important principle to 
maintain.

Cheers,
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 7, 2017, at 13:34, Jakub Wilk  wrote:
> "import async" would indeed cause deprecation warning, but that's not what 
> ldap3 does. The only uses of the now-keyword "async" in their codebase are 
> like this:
> 
>  from ..strategy.async import AsyncStrategy
>  from .async import AsyncStrategy
> 
> These do not provoke deprecation warnings from Python 3.6. (They probably 
> should!)
> 
> I'm afraid that showing deprecation warnings by default wouldn't have helped 
> in this particular case.

Oh gosh, I should have tried that instead of assuming it would generate the 
same warning.  Yes, that’s definitely a bug.  I wonder if we should push back 
making async/await reserved words until Python 3.8?

https://bugs.python.org/issue31973

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 12:35, Paul G  wrote:
> 
> If dictionary order is *not* guaranteed in the spec and the dictionary order 
> isn't randomized (which I think everyone agrees is a bit messed up), it would 
> probably be useful if you could enable "random order mode" in CPython, so you 
> can stress-test that your code isn't making any assumptions about dictionary 
> ordering without having to use an implementation where order isn't 
> deterministic.

As has been suggested elsewhere, if we decide not to make that guarantee, then 
we should probably deliberately randomize iteration order.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Barry Warsaw]

Antoine Pitrou wrote:

> Well... It really depends what kind of problem you're solving.  I
> certainly delete or pop items from dicts quite often.
> 
> Let's not claim that deleting items from a dict is a rare or advanced
> feature.  It is not.

+1.  It's a pretty common pattern for handling optional keyword
arguments, e.g. in subclass methods.

class Foo(Bar):
def foo(self, *args **kws):
mine = kws.pop('mine', None)
super().foo(self, *args, **kws)
do_something_myself(mine)

Now the question is, what guarantees does the language make about the
ordering of kws that Foo.foo() is passing to Bar.foo()?

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Barry Warsaw]

On Nov 7, 2017, at 10:41, Lukasz Langa  wrote:

> 3. Does that mean that Debian is going to rip it out and make people install 
> a `python-typing` .deb? Sadly, probably yes. We need to figure out what that 
> means for us.

Maybe.  Full disclosure, I’ve recently scaled back my contributions to Debian, 
so I won’t be doing this work, but if I was, I’d probably handle it very 
similarly to other replaceable external dependencies (e.g. pip).

There is a small loophole in policy to allow for the building and use of wheels 
for just this *limited* purpose.  So roughly I would propose packaging the 
external python-typing package as a separately installable deb, but also to 
build this into a wheel that we can pull in at python3.7 interpreter package 
build time.

It’s fairly easy to do, and all the infrastructure is already there.  What 
would be useful is for upstream CPython to make it easy to import an externally 
built and installed wheel, from some location outside of its own installed tree 
(/usr/share/python-wheels).

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 09:39, Paul Sokolovsky  wrote:

> So, the problem is that there's no "Python language spec”.

There is a language specification: 
https://docs.python.org/3/reference/index.html

But there are still corners that are undocumented, or topics that are 
deliberately left as implementation details.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 7, 2017, at 05:44, Paul Moore  wrote:

> If you're a user and your application developer didn't do (1) or a
> library developer developing one of the libraries your application
> developer chose to use didn't do (2), you're hosed. If you're a user
> who works in an environment where moving to a new version of the
> application is administratively complex, you're hosed.

“hosed” feels like too strong of a word here.  DeprecationWarnings usually 
don’t break anything.  Sure, they’re annoying but they can usually be easily 
ignored.

Yes, there are some situations where DWs do actively break things (as I’ve 
mentioned, some Debuntu build/test environments).  But those are also 
relatively easier to silence, or at least the folks running those environments, 
or writing the code for those environments, are usually more advanced 
developers for whom setting an environment variable or flag isn’t that big of a 
deal.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 7, 2017, at 01:51, Petr Viktorin  wrote:
> 
> When I explained this in 3.5, dicts rearranging themselves seemed quite weird 
> to the newcomers.
> This year, I'm not looking forward to saying that dicts behave "intuitively", 
> but you shouldn't rely on that, because they're theoretically allowed to 
> rearrange themselves.
> The concept of "implementation detail" and language spec vs. multiple 
> interpreter implementations isn't easy to explain to someone in a "basic 
> coding literacy" course.

Perhaps, but IME, it’s not hard to teach someone that in a code review.  Today, 
if I see someone submit a change that includes an implicit assumption about 
ordering, I’ll call that out.  I can say “you can’t rely on dicts being 
ordered, so if that’s what you want, use an OrderedDict or sort your test 
data”.  That’s usually a localized observation, meaning, I can look at the diff 
and see that they are assuming dict iteration ordering.

What happens when built-in dict’s implementation behavior becomes a language 
guarantee?  Now the review is much more difficult because I probably won’t be 
able to tell just from a diff whether the ordering guarantees are preserved.  
Do they delete a key somewhere?  Who knows?   I’m not even sure that would be 
statically determinable since I’d have to trace the use of that dictionary at 
run time to see if some “del d[key]” is deleting the key in the dict under 
review or not.  I can probably only tell that at run time.

So how to I accurately review that code?  Is the order presumption valid or 
invalid?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 22:33, Steven D'Aprano  wrote:

> I think it would be reasonable to say that builtin dicts only maintain
> insertion order for insertions, lookups, and changing the value. Any
> mutation which deletes keys may arbitrarily re-order the dict.
> 
> If the user wants a stronger guarantee, then they should use
> OrderedDict.

In fact, that *is* leaking CPython’s implementation into the language 
specification.  If by chance CPython’s implementation preserved order even 
after key deletion, either now or in the future, would that be defined as the 
ordering guarantees for built-in dict in the Python Language Reference?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The current dict is not an "OrderedDict"

[Barry Warsaw]

On Nov 7, 2017, at 07:12, Antoine Pitrou  wrote:

> The problem is this is taking things to a level of precision that makes
> the guarantee tedious to remember and reason about.
> 
> The only thing that's friendly to (non-expert) users is either "dicts
> are always ordered [by insertion order], point bar" or "dicts are not
> ordered, point bar".  Anything in-between, with reservations depending
> on which operations are invoked and when, is not really helpful to the
> average (non-expert) user.
> 
> Which is why I think the user-friendliness argument does not apply if
> order ceases to be guaranteed after __del__ is called.

That’s a very important point.  If it’s difficult to explain, teach, and retain 
the different ordering guarantees between built-in dict and OrderedDict, it 
might in fact be better to not guarantee any ordering for built-in dict *in the 
language specification*.  Otherwise we might need a section as big as chapter 5 
in the Python Language Reference just to dict ordering semantics. ;)

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

Nick Coghlan wrote:
> On the 12-weeks-to-3.7-feature-freeze thread, Jose Bueno & I both
> mistakenly though the async/await deprecation warnings were missing
> from 3.6.

Sometimes the universe just throws synchronicity right in your face.

I'm working on building an internal tool against Python 3.7 to take
advantage of the very cool -X importtime feature.  It's been a fun
challenge, but mostly because of our external dependencies.  For
example, PyThreadState renamed its structure members, so both Cython and
lxml needed new releases to adjust for this.  That's the "easy" part;
they've done it and those fixes work great.

We also depend on ldap3 .  Suddenly we
get a SyntaxError because ldap3 has a module ldap3/strategy/async.py.  I
say "suddenly" because of course *if* DeprecationWarnings had been
enabled by default, I'm sure someone would have noticed that those
imports were telling the developers about the impending problem in
Python 3.6.

https://github.com/cannatag/ldap3/issues/428

This just reinforces my opinion that even though printing
DeprecationWarning by default *can* be a hardship in some environments,
it is on the whole a positive beneficial indicator that gives developers
some runway to fix such problems.  These types of apparently sudden
breakages are the worse of all worlds.

Cheers,
-Barry


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 11:23, Paul G  wrote:
> 
> Is there a major objection to just adding in explicit syntax for 
> order-preserving dictionaries?

I don’t think new syntax is necessary.  We already have OrderedDict, which to 
me is a perfectly sensible way to spell “I need a mapping that preserves 
insertion order”, and the extra import doesn’t bother me.

I’m not saying whether or not to make the language guarantee that built-in dict 
preserves order.  I’m just saying that if we don’t make that language change, 
we already have everything we need to support both use cases.

If we did make the change, it’s possible we would need a way to explicit say 
that order is not preserved.  That seems a little weird to me, but I suppose it 
could be useful.  I like the idea previously brought up that iteration order be 
deliberately randomized in that case, but we’d still need a good way to spell 
that.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Allow annotations using basic types in the stdlib?

[Barry Warsaw]

On Nov 6, 2017, at 08:02, Victor Stinner  wrote:
> 
> While discussions on the typing module are still hot, what do you
> think of allowing annotations in the standard libraries, but limited
> to a few basic types:

I’m still -1 on adding annotations to the stdlib, despite their increasing use 
out in the wild, for the reasons that Steve and David have pointed out.  (Let’s 
let Eric be the one that breaks the mold with data classes.  Then we can blame 
him!)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 6, 2017, at 02:18, Paul Sokolovsky  wrote:

> What it will lead to is further fragmentation of the community. Python2
> vs Python3 split is far from being over, and now there're splits
> between:
> 
> * people who use "yield from" vs "await"
> * people who use f-strings vs who don't
> * people who rely on sorted nature of dict's vs who don't

This is the classic argument of, do you proceed conservatively and use the 
lowest-common denominator that makes your code work with the widest range of 
versions, or do you ride the wave and adopt the latest and greatest features as 
soon as they’re available?

Neither answer is wrong or right… for everyone.  It’s also a debate as old as 
the software industry. Every package, project, company, developer, community 
will have to decide for themselves.  Once you realize you can’t win, you’ve 
won! :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 20:47, Nick Coghlan  wrote:

>> warnings.silence_deprecations()
>> python -X silence-deprecations
>> PYTHONSILENCEDEPRECATIONS=x
> 
> It could be interesting to combine this with Tim's suggestion of
> putting an upper version limit on the silencing, so the above may look
> like:
> 
>warnings.ignore_deprecations((3, 7))
>python -X ignore-deprecations=3.7
>PYTHONIGNOREDEPRECATIONS=3.7

That could be cool as long as we also support wildcards, e.g. defaults along 
the lines of my suggestions above to ignore everything.

-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 23:08, Serhiy Storchaka  wrote:

> Following issues on GitHub related to new Python releases I have found that 
> many projects try to fix deprecation warning, but there are projects that are 
> surprised by ending of deprecation periods and removing features.

Like others here, I’ve also been bitten by silently ignored 
DeprecationWarnings.  We had some admittedly dodgy code in a corner of Mailman 
that we could have fixed earlier if we’d seen the warnings.  But we never did, 
so the first indication of a problem was when code actually *broke* with the 
new version of Python.  The problem was compounded because it wasn’t us that 
saw it first, it was a user, so now they had a broken installation and we had 
to issue a hot fix.  If we’d seen the DeprecationWarnings in the previous 
version of Python, we would have fixed them and all would have been good.

It’s true that those warnings can cause problems though.  There are certain 
build/CI environments, e.g. in Ubuntu, that fail when they see unexpected 
stderr output.  So when we start seeing new deprecations, we got build(-ish) 
time failures.  I still think that’s a minor price to pay for projects that 
*want* to do the right thing but don’t because those warnings are essentially 
hidden until they are breakages.  We have tools to help, so let’s use them.

Staying current and code clean is hard and never ending.  Welcome to software 
development!

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposal: go back to enabling DeprecationWarning by default

[Barry Warsaw]

On Nov 5, 2017, at 18:05, Nick Coghlan  wrote:

> So my proposal is simple (and not really new): let's revert back to
> the way things were in 2.6 and earlier, with DeprecationWarning being
> visible by default

+1

> As part of this though, I'd suggest amending the documentation for
> DeprecationWarning [1] to specifically cover how to turn it off
> programmatically (`warnings.simplefilter("ignore",
> DeprecationWarning)`), at the command line (`python -W
> ignore::DeprecationWarning ...`), and via the environment
> (`PYTHONWARNINGS=ignore::DeprecationWarning`).

+1

I’d also consider adding convenient shortcuts for each of these.  I think 
DeprecationWarning is special enough to warrant it.  Possibly:

warnings.silence_deprecations()
python -X silence-deprecations
PYTHONSILENCEDEPRECATIONS=x

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Barry Warsaw]

On Nov 4, 2017, at 11:35, Guido van Rossum  wrote:
> 
> This sounds reasonable -- I think when we introduced this in 3.6 we were 
> worried that other implementations (e.g. Jython) would have a problem with 
> this, but AFAIK they've reported back that they can do this just fine. So 
> let's just document this as a language guarantee.

The other concern was backward compatibility issues.  For example, if 3.7 makes 
this guarantee official, and folks write code with this assumption that has to 
work with older versions of Python, then that code could be buggy in previous 
versions and work in 3.7.  This will probably be most evident in test suites, 
and such failures are often mysterious to debug (as we’ve seen in the past).

That doesn’t mean we shouldn’t do it, but it does mean we have to be careful 
and explicit to educate users about how to write safe multi-Python-version code.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 3, 2017, at 17:09, Luca Sbardella  wrote:
> 
> Impressive stats! I didn’t know this command, thanks!

Neither did I until a day or so ago.  I already only want to use Python 3.7.  :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations

[Barry Warsaw]

On Nov 2, 2017, at 23:22, Nick Coghlan  wrote:
> Another point worth noting is that merely importing the typing module
> is expensive:
> 
> $ python -m perf timeit -s "from importlib import reload; import
> typing" "reload(typing)"
> .
> Mean +- std dev: 10.6 ms +- 0.6 ms
> 
> 10 ms is a *big* chunk out of a CLI application's startup time budget.

Far and away so, except for the re module.

% ./python.exe -X importtime -c "import typing"
import time: self [us] | cumulative | imported package
import time:72 | 72 | _codecs
import time:   625 |696 |   codecs
import time:   354 |354 |   encodings.aliases
import time:   713 |   1762 | encodings
import time:   198 |198 | encodings.utf_8
import time:98 | 98 | _signal
import time:   233 |233 | encodings.latin_1
import time:   353 |353 | _weakrefset
import time:   264 |617 |   abc
import time:   402 |   1018 | io
import time:   136 |136 |   _stat
import time:   197 |333 | stat
import time:   227 |227 |   genericpath
import time:   377 |604 | posixpath
import time:  2812 |   2812 | _collections_abc
import time:   787 |   4534 |   os
import time:   315 |315 |   _sitebuiltins
import time:   336 |336 |   sitecustomize
import time:   114 |114 |   usercustomize
import time:  1064 |   6361 | site
import time:   160 |160 |   _operator
import time:  1412 |   1571 | operator
import time:   371 |371 | keyword
import time:   817 |817 |   _heapq
import time:   762 |   1579 | heapq
import time:   272 |272 | itertools
import time:   635 |635 | reprlib
import time:99 | 99 | _collections
import time:  3580 |   8104 |   collections
import time:   112 |112 |   _functools
import time:   781 |892 | functools
import time:  1774 |   2666 |   contextlib
import time:   272 |272 |   types
import time:   861 |   1132 | enum
import time:76 | 76 |   _sre
import time:   426 |426 | sre_constants
import time:   446 |872 |   sre_parse
import time:   414 |   1361 | sre_compile
import time:79 | 79 | _locale
import time:   190 |190 | copyreg
import time: 17200 |  19961 |   re
import time:   374 |374 |   collections.abc
import time: 15124 |  46226 | typing

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Barry Warsaw]

On Nov 3, 2017, at 10:00, Steve Dower  wrote:
> 
> On 03Nov2017 0915, Victor Stinner wrote:
>> Hi,
>> 2017-11-03 15:36 GMT+01:00 Guido van Rossum :
>>> Maybe we should remove typing from the stdlib?
>>> https://github.com/python/typing/issues/495
>> I'm strongly in favor on such move.
> 
> I'm torn.

Me too.  We’re seeing much greater adoption of type annotations, and it’s 
becoming one of the killer features for adopting Python 3.  I’d be hesitant to 
accept anything that slows that adoption down.  While it’s been technically 
provisional, a lot of code is beginning to depend on it and it would be a shame 
to break that code as we also start to adopt Python 3.7.  But I can appreciate 
the need to iterate on its API faster.

I don’t know if a middle ground is feasible.  What core functionality and 
stable-enough APIs can be kept in stdlib typing, and can we provide an 
extension or override mechanism if you want the latest and greatest?

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Barry Warsaw]

On Nov 1, 2017, at 19:42, Guido van Rossum  wrote:
> 
> Maybe we should try it on some other list too? I know it works "in principle" 
> and I'd love for all Python mailing lists to migrate, but I'd like to have 
> some more experience with community mailing lists before tackling python-dev.

What about core-workflow or committers?  I think some of the criteria are:

* Gets a fair bit of traffic, but not too much
* Is okay with a little bit of downtime for the migration
* Willing to put up with any transient migration snafus
* Amenable to trying the new UI
* Has the BDFL as a member :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Barry Warsaw]

On Nov 1, 2017, at 18:49, Mariatta Wijaya  wrote:
> 
> Anything I can do to help make the migration to MM3 + HyperKitty happen? :)

Thanks for the offer Mariatta!  Assuming this is something we want to go 
through with, probably the best way to get there is to work with the 
postmaster, especially Mark Sapiro, on the migration.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Barry Warsaw]

On Oct 29, 2017, at 11:42, Serhiy Storchaka  wrote:

> Does Mailman 3 provide a NNTP interface? The NNTP interface of Gmane still 
> works, but it can be switched off at any time. It would be more reliable to 
> not depend on an unstable third-party service.

I use the NNTP interface of Gmane too (although not for python-dev), and agree 
with everything your saying here.  Right now however, MM3 does not have a 
built-in NNTP server.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP Post-History

[Barry Warsaw]

We’ve made a small change to the PEP process which may affect readers of 
python-list and python-ideas, so I’d like to inform you of it.  This change was 
made to PEP 1 and PEP 12.

PEPs must have a Post-History header which records the dates at which the PEP 
is posted to mailing lists, in order to keep the general Python community in 
the loop as a PEP moves to final status.  Until now, PEPs in development were 
supposed to be posted at least to python-dev and optionally to python-list[1].  
This guideline predated the creation of the python-ideas mailing list.

We’ve now changed this guideline so that Post-History will record the dates at 
which the PEP is posted to python-dev and optionally python-ideas.  python-list 
is dropped from this requirement.

python-dev is always the primary mailing list of record for Python development, 
and PEPs under development should be posted to python-dev as appropriate.  
python-ideas is the list for discussion of more speculative changes to Python, 
and it’s often where more complex PEPs, and even proto-PEPs are first raised 
and their concepts are hashed out.  As such, it makes more sense to change the 
guideline to include python-ideas and/or python-dev.  In the effort to keep the 
forums of record to a manageable number, python-list is dropped.

If you have been watching for new PEPs to be posted to python-list, you are 
invited to follow either python-dev or python-ideas.

Cheers,
-Barry (on behalf of the Python development community)

https://mail.python.org/mailman/listinfo/python-dev
https://mail.python.org/mailman/listinfo/python-ideas

Both python-dev and python-ideas are available via Gmane.

[1] PEPs may have a Discussions-To header which changes the list of forums 
where the PEP is discussed.  In that case, Post-History records the dates that 
the PEP is posted to those forums.  See PEP 1 for details.



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP Post-History (was Re: PEP 561: Distributing and Packaging Type Information)

[Barry Warsaw]

On Oct 27, 2017, at 00:12, Guido van Rossum  wrote:
> 
> Heh, you're right that was the reasoning. But I think python-list is much 
> less valuable than python-ideas for PEP authors. So let's change it.

Sounds good.  I just want to make sure we keep python-dev in the loop.

This is a process change though, so I’ll work with the PR#441 author to get the 
update into the PEPs, and then make the announcement on the relevant mailing 
lists.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Barry Warsaw]

On Oct 26, 2017, at 10:01, Donald Stufft  wrote:

> Pipermail is *horrible*

Pipermail also has a fatal flaw, and we have been hit by it several times in 
our past.  It’s fundamental to Pipermail’s design and can’t be fixed.  
Fortunately, HyperKitty was designed and implemented correctly so it doesn’t 
suffer this flaw.

Pipermail indexes messages sequentially, and if you ever regenerate the archive 
from the source mbox, it’s is almost guaranteed that your messages will get 
different URLs.  Worse, you can’t even automate a mapping from new URLs to old 
URLs.  This is especially likely in archives that go back as far as python-dev 
does, because there was a bug back in the day where even the source mbox file 
got corrupted, where the separator between messages was broken.  We tried to 
implement a fix for that, but it’s a heuristic and it’s not perfect.

We say that Pipermail does not have “stable urls”.  Thankfully HyperKitty does! 
 So even if you regenerate the HyperKitty archive, your messages will end up 
with the same URLs.

This let’s us implement Archived-At stably, and the algorithm at [1] even lets 
us pre-calculate the URL, so we can even include the URL to where the message 
*will* be once it’s archived, even without talking to HyperKitty, or any of the 
other archivers that are enabled (and support the algorithm of course).

So HyperKitty is miles ahead of Pipermail in design and implementation.  Sure 
it’s different, but people also forget how really buggy Pipermail was for a 
long time.  (And trust me, you really don’t even want to look at the code. ;)

Cheers,
-Barry

[1] https://wiki.list.org/DEV/Stable%20URLs


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Migrate python-dev to Mailman 3?

[Barry Warsaw]

On Oct 26, 2017, at 06:15, Victor Stinner  wrote:

> I don't think that Mailman 3 gives the choice of the UI for archives.

Technically, it does.

Mailman 3 has a pluggable architecture and supports multiple archives enabled 
site-wide and opt-in by individual lists.  HyperKitty is the default archiver, 
and the one we promote, but it doesn’t have to be the only archiver enabled.  
In fact, we come with plugins for mail-archive.com and MHonarc.  It *might* 
even be possible to enable a standalone Pipermail and route messages to that if 
one were so inclined.  The choice of archivers is not mutually exclusive.

Practically speaking though, there just aren’t a ton of well maintained FLOSS 
archivers to choose from.  HyperKitty *is* well maintained.  Frankly speaking, 
Pipermail is not.

Cheers,
-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 561: Distributing and Packaging Type Information

[Barry Warsaw]

On Oct 26, 2017, at 20:03, Guido van Rossum  wrote:
> 
> I think python-ideas does count here. Many PEPs evolve mostly there.

True, but there was some discussion of this way back when.

The way I remember it was that, while there are many outlets to discuss PEPs 
(including those pointed to by the optional Discussions-To header), python-dev 
is the “forum of record”.  This means that python-dev is the only mailing list 
you *have* to follow if you want to be informed of a PEP’s status in a timely 
manner.  Thus Post-History is supposed to reflect the history of when the PEP 
is sent to python-dev.  python-list is included because it’s the primary 
mailing list followed by people who aren’t developing Python but are still 
interested in it.

Maybe this needs to be reconsidered here in 2017, but that’s the rationale for 
the wording of PEP 1.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Barry Warsaw]

On Oct 12, 2017, at 10:46, Guido van Rossum  wrote:
> 
> I am still firmly convinced that @dataclass is the right name for the 
> decorator (and `dataclasses` for the module).

Darn, and I was going to suggest they be called EricTheHalfABees, with enums 
being renamed to EricTheHalfNotBees.

-Barry





signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553; the interaction between $PYTHONBREAKPOINT and -E

[Barry Warsaw]

> I don't know what is the best option, but I dislike adding two
> options, PYTHONBREAKPOINT and -X nobreakpoint, for the same features.
> I would become complicated to know which option has the priority.

Just to close the loop, I’ve landed the PEP 553 PR treating PYTHONBREAKPOINT 
the same as all other environment variables when -E is present.  Let’s see how 
that goes.

Thanks all for the great feedback and reviews.  Now I’m thinking about putting 
a backport version on PyPI. :)

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Barry Warsaw]

On Oct 4, 2017, at 13:53, Benjamin Peterson  wrote:

> It might be helpful to enumerate the usecases for such an API. Perhaps a
> narrow, specialized API could satisfy most needs in a supportable way.

Currently `python -m dis thing.py` compiles the source then disassembles it.  
It would be kind of cool if you could pass a .pyc file to -m dis, in which case 
you’d need to unpack the header to get to the code object.  A naive 
implementation would unpack the magic number and refuse to disassemble any 
files that don’t match whatever that version of Python understands.  A more 
robust (possibly 3rd party) implementation could potentially disassemble a 
range of magic numbers and formats, and an API to get at the code object and 
metadata would help.

I was thinking about the bytecode hacking that some debuggers do.  This API 
would help them support multiple versions of Python.  They could use the API to 
discover what pyc format was in use, extract the code object, hack the bytecode 
and possibly rewrite a new PEP 3147 style pyc file with the debugger bytecodes 
inserted.

Third party bytecode optimizers could use the API to unpack multiple versions 
of pyc files, do their optimizations, and rewrite new files with the proper 
format.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553; the interaction between $PYTHONBREAKPOINT and -E

[Barry Warsaw]

On Oct 4, 2017, at 21:52, Nick Coghlan  wrote:
> 
>> Unfortunately we probably won’t really get a good answer in practice until 
>> Python 3.7 is released, so maybe I just choose one and document that the 
>> behavior of PYTHONBREAKPOINT under -E is provision for now.  If that’s 
>> acceptable, then I would just treat -E for PYTHONBREAKPOINT the same as all 
>> other environment variables, and we’ll see how it goes.
> 
> I'd be fine with this as the main reason I wanted PYTHONBREAKPOINT=0
> was for pre-merge CI systems, and those tend to have tightly
> controlled environment settings, so you don't need to rely on -E or -I
> when running your tests.
> 
> That said, it may also be worth considering a "-X nobreakpoints"
> option (and then -I could imply "-E -s -X nobreakpoints").

Thanks for the feedback Nick.  For now we’ll go with the standard behavior of 
-E and see how it goes.  We can always add a -X later.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553

[Barry Warsaw]

On Oct 4, 2017, at 20:22, Yarko Tymciurak  wrote:

> I've recently started using a simple conditional breakpoint in ipython, and 
> wonder if  - in addition to Nick Coghlan's request for the env 
> 'PYTHONBREAKPOINT'  (boolean?), it would make sense (I _think_ so) to add a 
> condition parameter to the breakpoint() call.  This does raise several 
> questions, but it seems that it could make for a simple unified way to 
> conditionally call an arbitrary debugger.  What I found useful (in the 
> contecxt of ipython - but general enough) you can see in this gist: 
> https://gist.github.com/yarko/bdaa9d3178a6db03e160fdbabb3a9885
> 
> If PEP 553's breakpoint() were to follow this sort of interface (with 
> "condition"), it raises a couple of questions:
> - how would a missing (default) parameter be done?
> - how would parameters to be passed to the debugger "of record" be passed in 
> (named tuple? - sort of ugly)
> - would PYTHONBREAKPOINT be a global switch (I think yes), vs a `condition` 
> default.
> 
> I have no dog in the fight, but to raise the possibility (?) of having PEP 
> 553 implement simple conditional breakpoint processing.

Thanks for bringing this up Yarko.  I think this could be done with the current 
specification for PEP 553 and an additional API from the various debuggers.  I 
don’t think it needs to be part of PEP 553 explicitly, given the additional 
complications you describe above.

Remember that both built-in breakpoint() and sys.breakpointhook() accept *args, 
**kws, and it is left up to the actual debugger API to interpret/accept those 
additional arguments.  So let’s say you wanted to implement this behavior with 
pdb.  I think you could do something as simple as:

def conditional_set_trace(*, condition=True):
if condition:
pdb.set_trace()

sys.breakpointhook = conditional_set_trace

Then in your code, you would just write:

def foo(value):
breakpoint(condition=(value < 0))

With the IPython gist you referenced, you wouldn’t even need that convenience 
function.  Just set sys.breakpointhook=conditional_breakpoint.breakpoint_ and 
viola!

You could also PYTHONBREAKPOINT=conditional_breakpoint.breakpoint_ python3.7 … 
and it should Just Work.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553; the interaction between $PYTHONBREAKPOINT and -E

[Barry Warsaw]

> """Special cases aren't special enough to break the rules."""
> 
> People expect -E to disable envvar-driven overrides, so just treat it
> like that and don't try to second-guess the user.

And of course "Although practicality beats purity.” :)

So while I agree that the consistency argument makes sense, does it make the 
most practical sense?

I’m not sure.  On the PR, Nick suggests even another option: treat -E as all 
other environment variables, but then -I would be PYTHONBREAKPOINT=0.  Since 
the documentation for -I says "(implies -E and -s)” that seems even more 
special-case-y to me.

"In the face of ambiguity, refuse the temptation to guess.”

I’m really not sure what the right answer is, including to *not* make 
PYTHONBREAKPOINT obey -E.

Unfortunately we probably won’t really get a good answer in practice until 
Python 3.7 is released, so maybe I just choose one and document that the 
behavior of PYTHONBREAKPOINT under -E is provision for now.  If that’s 
acceptable, then I would just treat -E for PYTHONBREAKPOINT the same as all 
other environment variables, and we’ll see how it goes.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 553; the interaction between $PYTHONBREAKPOINT and -E

[Barry Warsaw]

Victor brings up a good question in his review of the PEP 553 implementation.

https://github.com/python/cpython/pull/3355
https://bugs.python.org/issue31353

The question is whether $PYTHONBREAKPOINT should be ignored if -E is given?

I think it makes sense for $PYTHONBREAKPOINT to be sensitive to -E, but in 
thinking about it some more, it might make better sense for the semantics to be 
that when -E is given, we treat it like PYTHONBREAKPOINT=0, i.e. disable the 
breakpoint, rather than fallback to the `pdb.set_trace` default.

My thinking is this: -E is often used in production environments to prevent 
stray environment settings from affecting the Python process.  In those 
environments, you probably also want to prevent stray breakpoints from stopping 
the process, so it’s more helpful to disable breakpoint processing when -E is 
given rather than running pdb.set_trace().

If you have a strong opinion either way, please follow up here, on the PR, or 
on the bug tracker.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Barry Warsaw]

On Oct 3, 2017, at 13:29, Benjamin Peterson  wrote:

> I'm not sure turning the implementation details of our internal formats
> into APIs is the way to go.

I still think an API in the stdlib would be useful and appropriate, but it’s 
not like this couldn’t be done as a 3rd party module.

-Barry




signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reorganize Python categories (Core, Library, ...)?

[Barry Warsaw]

On Oct 4, 2017, at 05:52, Victor Stinner  wrote:

> My problem is that almost all changes go into "Library" category. When
> I read long changelogs, it's sometimes hard to identify quickly the
> context (ex: impacted modules) of a change.
> 
> It's also hard to find open bugs of a specific module on
> bugs.python.org, since almost all bugs are in the very generic
> "Library" category. Using full text returns "false positives".
> 
> It's hard to find categories generic enough to not only contain a
> single item, but not contain too many items neither. Other ideas:
> 
> * XML: xml.doc, xml.etree, xml.parsers, xml.sax modules
> * Import machinery: imp and importlib modules
> * Typing: abc and typing modules

I often run into the same problem.  If we’re going to split up the Library 
section, then I think it makes sense to follow the top-level organization of 
the library manual:

https://docs.python.org/3/library/index.html

That already provides a mapping from module to category, and for the most part 
it’s a taxonomy that makes sense and is time proven.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)

[Barry Warsaw]

Guido van Rossum wrote:
> There have been no further comments. PEP 552 is now accepted.
> 
> Congrats, Benjamin! Go ahead and send your implementation for review.Oops.
> Let me try that again.

While I'm very glad PEP 552 has been accepted, it occurs to me that it
will now be more difficult to parse the various pyc file formats from
Python.  E.g. I used to be able to just open the pyc in binary mode,
read all the bytes, and then lop off the first 8 bytes to get to the
code object.  With the addition of the source file size, I now have to
(maybe, if I have to also read old-style pyc files) lop off the front 12
bytes, but okay.

With PEP 552, I have to do a lot more work to just get at the code
object.  How many bytes at the front of the file do I need to skip past?
 What about all the metadata at the front of the pyc, how do I interpret
that if I want to get at it from Python code?

Should the PEP 552 implementation add an API, probably to
importlib.util, that would understand all current and future formats?
Something like this perhaps?

class PycFileSpec:
magic_number: bytes
timestamp: Optional[bytes] # maybe an int? datetime?
source_size: Optional[bytes]
bit_field: Optional[bytes]
code_object: bytes

def parse_pyc(path: str) -> PycFileSpec:

Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make re.compile faster

[Barry Warsaw]

On Oct 3, 2017, at 01:41, Serhiy Storchaka  wrote:
> 
> 03.10.17 06:29, INADA Naoki пише:
>> More optimization can be done with implementing sre_parse and sre_compile in 
>> C.
>> But I have no time for it in this year.
> 
> And please don't do this! This would make maintaining the re module hard. The 
> performance of the compiler is less important than correctness and 
> performance of matching and searching.

What if the compiler could recognize constant arguments to re.compile() and do 
the regex compilation at that point?  You’d need a way to represent the 
precompiled regex in the bytecode, and it would technically be a semantic 
change since regex problems would be discovered at compilation time instead of 
runtime - but that might be a good thing.  You could also make that an 
optimization flag for opt-in, or a flag to allow opt out.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Make re.compile faster

[Barry Warsaw]

On Oct 3, 2017, at 01:35, Serhiy Storchaka  wrote:
> 
>> diff --git a/Lib/string.py b/Lib/string.py
>> index b46e60c38f..fedd92246d 100644
>> --- a/Lib/string.py
>> +++ b/Lib/string.py
>> @@ -81,7 +81,7 @@ class Template(metaclass=_TemplateMetaclass):
>>  delimiter = '$'
>>  idpattern = r'[_a-z][_a-z0-9]*'
>>  braceidpattern = None
>> -flags = _re.IGNORECASE
>> +flags = _re.IGNORECASE | _re.ASCII
>>  def __init__(self, template):
>>  self.template = template
>> patched:
>> import time:  1191 |   8479 | string
>> Of course, this patch is not backward compatible. [a-z] doesn't match with 
>> 'ı' or 'ſ' anymore.
>> But who cares?
> 
> This looks like a bug fix. I'm wondering if it is worth to backport it to 
> 3.6. But the change itself can break a user code that changes idpattern 
> without touching flags. There is other way, but it should be discussed on the 
> bug tracker.

It’s definitely an API change, as I mention in the bug tracker.  It’s 
*probably* safe in practice given that the documentation does say that 
identifiers are ASCII by default, but it also means that a client who wants to 
use Unicode previously didn’t have to touch flags, and after this change would 
now have to do so.  `flags` is part of the public API.

Maybe for subclasses you could say that if delimiter, idpattern, or 
braceidpattern are anything but the defaults, fall back to just re.IGNORECASE.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553

[Barry Warsaw]

On Oct 2, 2017, at 18:43, Guido van Rossum  wrote:
> 
> OK. That then concludes the review of your PEP. It is now accepted! Congrats. 
> I am looking forward to using the backport. :-)

Yay, thanks!  We’ll see if I can sneak that backport past Ned. :)

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553

[Barry Warsaw]

On Oct 2, 2017, at 17:36, Guido van Rossum  wrote:

> I've seen your updates and it is now acceptable, except for *one* nit: in 
> builtins.breakpoint() the pseudo code raises RuntimeError if 
> sys.breakpointhook is missing or None. OTOH sys.breakpointhook() just issues 
> a RuntimeWarning when something's wrong with the hook. Maybe 
> builtins.breakpoint() should also just warn if it can't find the hook? 
> Setting `sys.breakpointhook = None` might be the simplest way to 
> programmatically disable breakpoints. Why not allow it?

Oh, actually the pseudocode doesn’t match the C implementation exactly in this 
regard.  Currently the C implementation is more like:

def breakpoint(*args, **kws):
import sys
missing = object()
hook = getattr(sys, 'breakpointhook', missing)
if hook is missing:
raise RuntimeError('lost sys.breakpointhook')
return hook(*args, **kws)

The intent being, much like the other sys-hooks, that if 
PySys_GetObject("breakpointhook”) returns NULL, Something Bad Happened, so we 
have to set an error string and bail.  (PySys_GetObject() does not set an 
exception.)

E.g.

>>> def foo():
...   print('yes')
...   breakpoint()
...   print('no')
...
>>> del sys.breakpointhook
>>> foo()
yes
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in foo
RuntimeError: lost sys.breakpointhook


Setting `sys.breakpoint = None` could be an interesting use case, but that’s 
not currently special in any way:

>>> sys.breakpointhook = None
>>> foo()
yes
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in foo
TypeError: 'NoneType' object is not callable


I’m open to special-casing this if you think it’s useful.

(I’ll update the pseudocode in the PEP.)

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating time for `import requests`

[Barry Warsaw]

On Oct 2, 2017, at 14:56, Brett Cannon  wrote:

> So Mercurial specifically is an odd duck because they already do lazy 
> importing (in fact they are using the lazy loading support from importlib). 
> In terms of all of this discussion of tweaking import to be lazy, I think the 
> best approach would be providing an opt-in solution that CLI tools can turn 
> on ASAP while the default stays eager. That way everyone gets what they want 
> while the stdlib provides a shared solution that's maintained alongside 
> import itself to make sure it functions appropriately.

The problem I think is that to get full benefit of lazy loading, it has to be 
turned on globally for bare ‘import’ statements.  A typical application has 
tons of dependencies and all those libraries are also doing module global 
imports, so unless lazy loading somehow covers them, it’ll be an incomplete 
gain.  But of course it’ll take forever for all your dependencies to use 
whatever new API we come up with, and if it’s not as convenient to write as 
‘import foo’ then I suspect it won’t much catch on anyway.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553

[Barry Warsaw]

Thanks for the review Guido!  The PEP and implementation have been updated to 
address your comments, but let me briefly respond here.

> On Oct 2, 2017, at 00:44, Guido van Rossum  wrote:

> - There's a comma instead of a period at the end of the 4th bullet in the 
> Rationale: "Breaking the idiom up into two lines further complicates the use 
> of the debugger,” .

Thanks, fixed.

> Also I don't understand how this complicates use

I’ve addressed that with some additional wording in the PEP.  Basically, it’s 
my contention that splitting it up on two lines introduces more opportunity for 
mistakes.

> TBH the biggest argument (to me) is that I simply don't know *how* I would 
> enter some IDE's debugger programmatically. I think it should also be pointed 
> out that even if an IDE has a way to specify conditional breakpoints, the UI 
> may be such that it's easier to just add the check to the code -- and then 
> the breakpoint() option is much more attractive than having to look up how 
> it's done in your particular IDE (especially since this is not all that 
> common).

This is a really excellent point!  I’ve reworked that section of the PEP to 
make this clear.

> - There's no rationale for the *args, **kwds part of the breakpoint() 
> signature. (I vaguely recall someone on the mailing list asking for it but it 
> seemed far-fetched at best.)

I’ve added some rationale.  The idea comes from optional `header` argument to 
IPython’s programmatic debugger API.  I liked that enough to add it to 
pdb.set_trace() for 3.7.  IPython accepts other optional arguments, so I think 
we do want to allow those to be passed through the call chain.  I expect any 
debugger’s advertised entry point to make these optional, so `breakpoint()` 
will always just work.

> - The explanation of the relationship between sys.breakpoint() and 
> sys.__breakpointhook__ was unclear to me

I think you understand it correctly, and I’ve hopefully clarified that in the 
PEP now, so you wouldn’t have to read the __displayhook__ (or __excepthook__) 
docs to understand how it works.

> - Some pseudo-code would be nice.

Great idea; added that to the PEP (pretty close to what you have, but with the 
warnings handling, etc.)

> I think something like `os.environ['PYTHONBREAKPOINT'] = 'foo.bar.baz'; 
> breakpoint()` should result in foo.bar.baz() being imported and called, right?

Correct.  Clarified in the PEP now.

> - I'm not quite sure what sort of fast-tracking for PYTHONBREAKPOINT=0 you 
> had in mind beyond putting it first in the code above.

That’s pretty close to it.  Clarified.

> - Did you get confirmation from other debuggers? E.g. does it work for IDLE, 
> Wing IDE, PyCharm, and VS 2015?

From some of them, yes.  Terry confirmed for IDLE, and I posted a statement in 
favor of the PEP from the PyCharm folks.  I’m pretty sure Steve confirmed that 
this would be useful for VS, and I haven’t heard from the Wing folks.

> - I'm not sure what the point would be of making a call to breakpoint() a 
> special opcode (it seems a lot of work for something of this nature). ISTM 
> that if some IDE modifies bytecode it can do whatever it well please without 
> a PEP.

I’m strongly against including anything related to a new bytecode to PEP 553; 
they’re just IMHO orthogonal issues, and I’ve removed this as an open issue for 
553.

I understand why some debugger developers want it though.  There was a talk at 
Pycon 2017 about what PyCharm does.  They have to rewrite the bytecode to 
insert a call to a “trampoline function” which in many ways is the equivalent 
of breakpoint() and sys.breakpointhook().  I.e. it’s a small function that sets 
up and calls a more complicated function to do the actual debugging.  IIRC, 
they open up space for 4 bytecodes, with all the fixups that implies.  The idea 
was that there could be a single bytecode that essentially calls builtin 
breakpoint().  Steve indicated that this might also be useful for VS.

There’s a fair bit that would have to be fleshed out to make this idea real, 
but as I say, I think it shouldn’t have anything to do with PEP 553, except 
that it could probably build on the APIs we’re adding here.

> - I don't see the point of calling `pdb.pm()` at breakpoint time. But it 
> could be done using the PEP with `import pdb; sys.breakpointhook = pdb.pm` 
> right? So this hardly deserves an open issue.

Correct, and I’ve removed this open issue.

> - I haven't read the actual implementation in the PR. A PEP should not depend 
> on the actual proposed implementation for disambiguation of its specification 
> (hence my proposal to add pseudo-code to the PEP).
> 
> That's what I have!

Cool, that’s very helpful, thanks!

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 

Re: [Python-Dev] Investigating time for `import requests`

[Barry Warsaw]

On Oct 1, 2017, at 22:34, Nathaniel Smith  wrote:
> 
> In principle re.compile() itself could be made lazy -- return a
> regular exception object that just holds the string, and then compiles
> and caches it the first time it's used. Might be tricky to do in a
> backwards compatibility way if it moves detection of invalid regexes
> from compile time to use time, but it could be an opt-in flag.

I already tried that experiment.  1) there are tricky corner cases; 2) nobody 
liked the change in semantics when re.compile() was made lazy.

https://bugs.python.org/issue31580
https://github.com/python/cpython/pull/3755

I think there are opportunities for an explicit API for lazy compilation of 
regular expressions, but I’m skeptical of the adoption curve making it 
worthwhile.  But maybe I’m wrong!

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Barry Warsaw]

On Oct 2, 2017, at 10:48, Christian Heimes  wrote:
> 
> That approach could work, but I think that it is the wrong approach. I'd
> rather keep Python optimized for long-running processes and introduce a
> new mode / option to optimize for short-running scripts.

What would that look like, how would it be invoked, and how would that change 
the behavior of the interpreter?

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New security-annou...@python.org mailing list

[Barry Warsaw]

On Sep 27, 2017, at 12:24, Guido van Rossum  wrote:
> 
> Can we add redirects to the old list locations?

I’m not sure we need it for security-announce given how new that list is and 
that we only have one message to it.  I’ve forwarded this request to 
postmaster@ though.

-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New security-annou...@python.org mailing list

[Barry Warsaw]

On Sep 27, 2017, at 11:56, Jesus Cea <j...@jcea.es> wrote:
> 
> On 21/09/17 17:30, Barry Warsaw wrote:
>> https://mail.python.org/mailman/listinfo/security-announce
> 
> "No such list security-announce".
> 
>> https://mail.python.org/mailman/listinfo/security-sig
> 
> "No such list security-sig".

These lists were just migrated to Mailman 3.  Please use:

https://mail.python.org/mm3/mailman3/lists/security-announce.python.org/

https://mail.python.org/mm3/mailman3/lists/security-sig.python.org/

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] New security-annou...@python.org mailing list

[Barry Warsaw]

I’m happy to announce the availability of a new mailing list, with the mission 
of providing security announcements to the Python community from the Python 
Security Response Team (PSRT):

security-annou...@python.org

You can sign up in the usual Mailman way:

https://mail.python.org/mailman/listinfo/security-announce

This joins our suite of security related forums.  As always, if you believe 
you’ve found a security issue in Python, you should contact the PSRT directly 
and securely via:

secur...@python.org

For more information on how you can contact us, see:

https://www.python.org/news/security/

We also have a public security-focused discussion mailing list that you can 
subscribe and contribute to.

security-...@python.org
https://mail.python.org/mailman/listinfo/security-sig

Please don’t report security vulnerabilities here, since this is a publicly 
archived mailing list.  We welcome you to collaborate here to help make Python 
and its ecosystem even more secure than it already is.

Once a security vulnerability is identified and fixed, it becomes public 
knowledge.  Generally, these are captured in a ReadTheDocs site for posterity:

https://python-security.readthedocs.io/

This new security-announce mailing list fills a void — one-way communication 
about security related matters from the PSRT back to the community.  This is an 
area that we’ve not done a great job at, frankly, and this new announcement 
list is intended to improve that situation.  The PSRT will use this low 
traffic, high value forum as the primary way the PSRT will communicate security 
issues of high importance back to the wider Python community.  All follow-ups 
to postings to this list are redirected to the security-sig mailing list.

Cheers,
-Barry (on behalf of the PSRT)



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553, v3

[Barry Warsaw]

Barry Warsaw wrote:
> Here’s an update to PEP 553, which makes $PYTHONBREAKPOINT a first class 
> feature.  I’ve also updated PR #3355 with the implementation to match.

I've been in contact with Elizaveta Shashkova from JetBrains.  She gave
a great talk at Pycon 2017 which influenced my thinking about the
feature that lead to PEP 553.

https://www.youtube.com/watch?v=NdObDUbLjdg

She wasn't able to post a follow up directly, but here is her response,
pasted here with her permission.

"In the beginning of 2017 we implemented frame evaluation debugger in
the PyCharm IDE - the debugger, based on the frame evaluation API
(included to Python 3.6 in PEP 523). This implementation improved
debugger's performance significantly.
The main idea of this debugger is using custom frame evaluation function
for implementing breakpoints instead of standard tracing function. In
fact, before the entering a new frame we're modifying the code object,
generated for this frame and we're inserting breakpoints right into the
code object. At the moment we're doing quite complicated things: we
define our own local `breakpoint()` function (which leads debugger to a
suspend state), create a wrapper for it, and insert the wrapper's code
into the user's code. In order to be able to call this function, we add
it to frame's globals dict. It's a bit tricky and it still has some
problems.

So if PEP 553 is accepted, it will help us to make this part of the
debugger much simpler: we will be able to specify our breakpoint
function for the whole program and call built-in breakpoint() function
instead of all these manipulations with wrappers and frame's globals. It
will make our frame evaluation debugger less complicated and more stable."

Cheers,
-Barry

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com