Re: [Python-Dev] modules people want installed automatically (was: Re: re performance)

[Brett Cannon]

Proper support in the loader API is possible and was on its way to being
coded up until GitHub took precedence (realize this project has put all
other major python projects of mine on hold for nearly 2 years hence the
delay). And pkg_resources I'm sure could be updated to use any new API when
available, so I don't think it's appropriate to bring the name into the
stdlib.

On Mon, Jan 30, 2017, 10:28 Barry Warsaw,  wrote:

> On Jan 30, 2017, at 06:14 PM, Brett Cannon wrote:
>
> >What functionality are you after here from pkg_resources? If it's the
> >reading of data files then you will get that in the stdlib through
> >importlib as soon as I'm not working on getting our workflow to work
> >through GitHub's web UI (which obviously includes the migration itself).
>
>
> http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access
>
> Mostly I use:
>
> * resource_filename()
> * resource_string() (really, resource_bytes!)
> * resource_stream() (although I'd really like a more open()-like API)
>
> This might fall under a simpler PEP 365.  Also, while I would love to have
> these available say in importlib, I also like to consider a backward
> compatible API where the stdlib provides the `pkg_resources` module name.
> That's not totally required though because you can play ImportError games
> until Python 3.7 (presumably) is in widespread -and only- use.
>
> Cheers,
> -Barry
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-30 22:19, David Cournapeau wrote:
> Hm. Is this documented anywhere ? We have customers needing
> "private/custom" certificates, and I am unsure where to look for.

For full control it is advised to use a custom SSLContext that only
loads the internal CA.
https://docs.python.org/3/library/ssl.html#ssl.create_default_context

With OpenSSL you can also set SSL_CERT_FILE and SSL_CERT_DIR env vars.
It doesn't work with LibreSSL, though.

import os, ssl
os.environ['SSL_CERT_FILE'] = '/path/to/internalca.pem'
os.environ['SSL_CERT_DIR'] = os.devnull

ctx = ssl.create_default_context()

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[David Cournapeau]

On Mon, Jan 30, 2017 at 9:14 PM, Christian Heimes 
wrote:

> On 2017-01-30 22:00, David Cournapeau wrote:
> >
> >
> > On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  > > wrote:
> >
> >
> >
> > > On 30 Jan 2017, at 13:53, David Cournapeau  > wrote:
> > >
> > > Are there any official recommendations for downstream packagers
> beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch
> python's default cert locations ?
> >
> > There *are* no default cert locations on Windows or macOS that can
> > be accessed by OpenSSL.
> >
> > I cannot stress this strongly enough: you cannot provide a
> > platform-native certificate validation logic for Python *and* use
> > OpenSSL for certificate validation on Windows or macOS. (macOS can
> > technically do this when you link against the system OpenSSL, at the
> > cost of using a catastrophically insecure version of OpenSSL.)
> >
> >
> > Ah, thanks, that's already useful information.
> >
> > Just making sure I understand: this means there is no way to use
> > python's SSL library to use the system store on windows, in particular
> > private certifications that are often deployed by internal ITs in large
> > orgs ?
>
> That works with CPython because we get all trust anchors from the cert
> store. However Python is not able to retrieve *additional* certificates.
> A new installation of Windows starts off with a minimal set of trust
> anchors. Chrome, IE and Edge use the proper APIs.
>

Hm. Is this documented anywhere ? We have customers needing
"private/custom" certificates, and I am unsure where to look for.

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-30 22:00, David Cournapeau wrote:
> 
> 
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  > wrote:
> 
> 
> 
> > On 30 Jan 2017, at 13:53, David Cournapeau  > wrote:
> >
> > Are there any official recommendations for downstream packagers beyond 
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's 
> default cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can
> be accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a
> platform-native certificate validation logic for Python *and* use
> OpenSSL for certificate validation on Windows or macOS. (macOS can
> technically do this when you link against the system OpenSSL, at the
> cost of using a catastrophically insecure version of OpenSSL.)
> 
> 
> Ah, thanks, that's already useful information.
> 
> Just making sure I understand: this means there is no way to use
> python's SSL library to use the system store on windows, in particular
> private certifications that are often deployed by internal ITs in large
> orgs ?

That works with CPython because we get all trust anchors from the cert
store. However Python is not able to retrieve *additional* certificates.
A new installation of Windows starts off with a minimal set of trust
anchors. Chrome, IE and Edge use the proper APIs.

There is no way to get internal CA on macOS with 3.6, with certifi, or
with self-build OpenSSL.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-30 21:50, Cory Benfield wrote:
> 
> 
>> On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
>>
>> Are there any official recommendations for downstream packagers beyond PEP 
>> 476 ? Is it "acceptable" for downstream packagers to patch python's default 
>> cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can be 
> accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a platform-native 
> certificate validation logic for Python *and* use OpenSSL for certificate 
> validation on Windows or macOS. (macOS can technically do this when you link 
> against the system OpenSSL, at the cost of using a catastrophically insecure 
> version of OpenSSL.) 

In theory it is possible for Python and OpenSSL, too. I looked into a
custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is
trying an alternative approach in https://bugs.python.org/issue28747. It
ain't pretty and we are not there yet, too. Native support for SChannel
and SecureTransport has some benefits.

It's too bad OpenSSL lacks support for PKCS#11 Trust Assertion Objects.
We could use
https://p11-glue.freedesktop.org/doc/pkcs11-trust-assertions/#pkcs11-objects
under Linux and the PKCS#11 under Windows and macOS.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[David Cournapeau]

On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  wrote:

>
>
> > On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
> >
> > Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?
>
> There *are* no default cert locations on Windows or macOS that can be
> accessed by OpenSSL.
>

Also, doesn't that contradict the wording of PEP 476, specifically " Python
would use the system provided certificate database on all platforms.
Failure to locate such a database would be an error, and users would need
to explicitly specify a location to fix it." ?

Or is that PEP a long term goal, and not a description of the current
status ?

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[David Cournapeau]

On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  wrote:

>
>
> > On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
> >
> > Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?
>
> There *are* no default cert locations on Windows or macOS that can be
> accessed by OpenSSL.
>
> I cannot stress this strongly enough: you cannot provide a platform-native
> certificate validation logic for Python *and* use OpenSSL for certificate
> validation on Windows or macOS. (macOS can technically do this when you
> link against the system OpenSSL, at the cost of using a catastrophically
> insecure version of OpenSSL.)
>

Ah, thanks, that's already useful information.

Just making sure I understand: this means there is no way to use python's
SSL library to use the system store on windows, in particular private
certifications that are often deployed by internal ITs in large orgs ?


> The only program I am aware of that does platform-native certificate
> validation on all three major desktop OS platforms is Chrome. It does this
> using a fork of OpenSSL to do the actual TLS, but the platform-native
> crypto library to do the certificate validation. This is the only
> acceptable way to do this, and Python does not expose the appropriate hooks
> to do it from within Python code. This would require that you carry
> substantial patches to the standard library to achieve this, all of which
> would be custom code. I strongly recommend you don't undertake to do this
> unless you are very confident of your ability to write this code correctly.
>

That's exactly what I was afraid of and why I asked before attempting
anything.


>
> The best long term solution to this is to stop using OpenSSL on platforms
> that don't consider it the 'blessed' approach. If you're interested in
> following that work, we're currently discussing it on the security-SIG, and
> you'd be welcome to join.
>

Thanks, I will see if it looks like I have anything to contribute.

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
> 
> Are there any official recommendations for downstream packagers beyond PEP 
> 476 ? Is it "acceptable" for downstream packagers to patch python's default 
> cert locations ?

There *are* no default cert locations on Windows or macOS that can be accessed 
by OpenSSL.

I cannot stress this strongly enough: you cannot provide a platform-native 
certificate validation logic for Python *and* use OpenSSL for certificate 
validation on Windows or macOS. (macOS can technically do this when you link 
against the system OpenSSL, at the cost of using a catastrophically insecure 
version of OpenSSL.) 

The only program I am aware of that does platform-native certificate validation 
on all three major desktop OS platforms is Chrome. It does this using a fork of 
OpenSSL to do the actual TLS, but the platform-native crypto library to do the 
certificate validation. This is the only acceptable way to do this, and Python 
does not expose the appropriate hooks to do it from within Python code. This 
would require that you carry substantial patches to the standard library to 
achieve this, all of which would be custom code. I strongly recommend you don't 
undertake to do this unless you are very confident of your ability to write 
this code correctly.

The best long term solution to this is to stop using OpenSSL on platforms that 
don't consider it the 'blessed' approach. If you're interested in following 
that work, we're currently discussing it on the security-SIG, and you'd be 
welcome to join. 

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-30 14:53, David Cournapeau wrote:
> Hi,
> 
> I am managing the team responsible for providing python packaging at
> Enthought, and I would like to make sure we are providing a good (and
> secure) out of the box experience for SSL.
> 
> My understanding is that PEP 476 is the latest PEP that concerns this
> issue, and that PEP recommends using the system store:
> https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at
> binary python distributions from python.org , that
> does not seem to a.ways be the case. I looked at the following:
> 
> * 3.5.3 from python.org  for OS X (64 bits): this
> uses the old, system openssl

On macOS, system OpenSSL uses TEA to look up trust anchors from the
keyring, https://hynek.me/articles/apple-openssl-verification-surprises/

> * 3.6.0 from python.org  for OS X: this embeds a
> recent openssl, but ssl seems to be configured to use non existing paths
> (ssl..get_default_verify_paths()), and indeed, cert validation seems to
> fail by default with those installers

I guess you haven't read the warning on the download page,
https://www.python.org/downloads/release/python-360/ :)

tl;dr Python on macOS can no longer use the keyring for certificates.
You have to install certifi. Personally I (as a maintainer of the ssl
module) do neither like certifi nor the approach taken. It's wrong and
dangerous because certifi by-passes system policies and does not include
local CAs. But it's the best we got.

> * 3.6.0 from python.org  for windows: I have not
> found how the ssl module finds the certificate, but certification seems
> to work

The magic happens in load_default_certs(),
https://hg.python.org/cpython/file/tip/Lib/ssl.py#l445 . The approach
does work most of the time. Sometimes it fails because Windows download
root CA's on demand. The code does not trigger a download.

Steve and I have some ideas to improve the situation,
https://bugs.python.org/issue28747 .

> 
> Are there any official recommendations for downstream packagers beyond
> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
> default cert locations ?

My semi-official recommendations are:

* On Linux and BSD, use OpenSSL from your vendor (Red Hat, SuSE, Ubuntu,
Debian, Arch, Gentoo, ...). Vendors compile OpenSSL with special flags
to make get_default_verify_paths() work.

* If your vendor prefers LibreSSL, you might run into minor issue.
LibreSSL is not fully compatible to OpenSSL. Your vendor also compiles
LibresSL with special flags.

* Windows kinda works. We are going to improve the situation for 3.7.

* On macOS ... well, please contact Apple service and demand a solution.
Cory Benfield's TLS ABC PEP and Paul Kehrer's work on SecureTransport
bindings for Python will let you use Apple's native TLS lib (with some
restrictions regarding fork()). In the mean time you are screwed and
have to rely on certifi.

As last resort, use certifi. You HAVE to make sure to keep it up to date
and you have to deal with local policies and CAs yourself.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Lukasz Langa]

> On Jan 30, 2017, at 6:26 AM, Barry Warsaw  wrote:
> 
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.  Splitting
> them may mean that pkg_resources could eventually be added to the stdlib, but
> as an intermediate step, it could also test out this idea.  It probably has a
> lot less of the baggage that you outline.

FWIW this is what we do at Facebook. We ship pkg_resources with everything but 
not setuptools. So there's definitely value in that. It's a little surprising 
we haven't been doing that yet, given that zipped applications need it.

- Ł
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] modules people want installed automatically (was: Re: re performance)

[Barry Warsaw]

On Jan 30, 2017, at 06:14 PM, Brett Cannon wrote:

>What functionality are you after here from pkg_resources? If it's the
>reading of data files then you will get that in the stdlib through
>importlib as soon as I'm not working on getting our workflow to work
>through GitHub's web UI (which obviously includes the migration itself).

http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resource-access

Mostly I use:

* resource_filename()
* resource_string() (really, resource_bytes!)
* resource_stream() (although I'd really like a more open()-like API)

This might fall under a simpler PEP 365.  Also, while I would love to have
these available say in importlib, I also like to consider a backward
compatible API where the stdlib provides the `pkg_resources` module name.
That's not totally required though because you can play ImportError games
until Python 3.7 (presumably) is in widespread -and only- use.

Cheers,
-Barry


pgpbwx9wa1J1Z.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] modules people want installed automatically (was: Re: re performance)

[Brett Cannon]

On Mon, 30 Jan 2017 at 06:27 Barry Warsaw  wrote:

> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
> >I think there are 3 main candidates that could fit that bill:
> >
> >- requests
> >- setuptools
> >- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.
> Splitting
> them may mean that pkg_resources could eventually be added to the stdlib,
> but
> as an intermediate step, it could also test out this idea.  It probably
> has a
> lot less of the baggage that you outline.


What functionality are you after here from pkg_resources? If it's the
reading of data files then you will get that in the stdlib through
importlib as soon as I'm not working on getting our workflow to work
through GitHub's web UI (which obviously includes the migration itself).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Brett Cannon]

On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues  wrote:

> On Thu, Jan 26, 2017 at 4:09 AM, Ivan Levkivskyi 
> wrote:
>
>
>
> Concerning list/set/dict comprehensions, I am much more in favor of making
> comprehensions simply equivalent to for-loops (more or less like you
> proposed using yield from). The only reason to introduce auxiliary function
> scope was to prevent the loop variables from leaking outside comprehensions.
> Formally, this is indeed backward incompatible, but I doubt many people
> depend on the current counter-intuitive behavior.
>
> Concerning generator expressions, probably it is indeed better to simply
> prohibit yield inside them.
>
>
> Thank you to everyone who responded to my post and provided excellent
> analysis.
>
> For Python, I don't know what the best way to proceed is:
>
> OPTION 1
> 
>
> Make a SyntaxError:  [(yield 1) for x in range(10)]
> and update the documentation to explain that this is an invalid construct.
>
> This would have certainly helped me identify the source of the problem as
> I tried porting buildbot 0.9 to Python 3.
>
> However, while not very common, there is Python 2.x code that uses that.
> I found these cases in the buildbot code which I changed so as to work on
> Python 2 and 3:
>
> https://github.com/buildbot/buildbot/pull/2661
> https://github.com/buildbot/buildbot/pull/2673
>
>
>  OPTION 2
> =
> Make this return a list on Python 3, like in Python 2:  [
> (yield 1) for x in range(10)]
>
> As pointed out by others on the this mailing list, there are some
> problems associated with that.  I don't know if there are many Python 2
> codebases out there
> with this construct, but it would be nice to have one less Python 2 -> 3
> porting gotcha.
>
>
> I'm OK with either approach.  Leaving things the way they are in Python 3
> is no good, IMHO.
>

My vote is it be a SyntaxError since you're not getting what you expect
from the syntax.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Barry Warsaw]

On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:

>I think there are 3 main candidates that could fit that bill:
>
>- requests
>- setuptools
>- regex

Actually, I think pkg_resources would make an excellent candidate.  The
setuptools crew is working on a branch that would allow for setuptools and
pkg_resources to be split, which would be great for other reasons.  Splitting
them may mean that pkg_resources could eventually be added to the stdlib, but
as an intermediate step, it could also test out this idea.  It probably has a
lot less of the baggage that you outline.

Cheers,
-Barry


pgpD4mq5b21qp.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] SSL certificates recommendations for downstream python packagers

[David Cournapeau]

Hi,

I am managing the team responsible for providing python packaging at
Enthought, and I would like to make sure we are providing a good (and
secure) out of the box experience for SSL.

My understanding is that PEP 476 is the latest PEP that concerns this
issue, and that PEP recommends using the system store:
https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at
binary python distributions from python.org, that does not seem to a.ways
be the case. I looked at the following:

* 3.5.3 from python.org for OS X (64 bits): this uses the old, system
openssl
* 3.6.0 from python.org for OS X: this embeds a recent openssl, but ssl
seems to be configured to use non existing paths
(ssl..get_default_verify_paths()), and indeed, cert validation seems to
fail by default with those installers
* 3.6.0 from python.org for windows: I have not found how the ssl module
finds the certificate, but certification seems to work

Are there any official recommendations for downstream packagers beyond PEP
476 ? Is it "acceptable" for downstream packagers to patch python's default
cert locations ?

David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Berker Peksağ]

On Mon, Jan 30, 2017 at 2:56 PM, Antoine Pitrou  wrote:
> On Sun, 29 Jan 2017 20:30:38 +
> Steve Holden  wrote:
>> Why not declare re deprecated and remove it in Python 4?
>
> Why deprecate and remove a library that's perfectly usable and
> satisfactory for the vast majority of regular expression usage?  It's
> not like regex presents a radically different API...
>
> Do we really have to face another wave of stdlib bashing on this
> mailing-list?

I agree with Antoine. And note that re has an active maintainer who
has fixed a bunch of bugs and added some new features in the last few
years.

--Berker
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Antoine Pitrou]

On Sun, 29 Jan 2017 20:30:38 +
Steve Holden  wrote:
> Why not declare re deprecated and remove it in Python 4?

Why deprecate and remove a library that's perfectly usable and
satisfactory for the vast majority of regular expression usage?  It's
not like regex presents a radically different API...

Do we really have to face another wave of stdlib bashing on this
mailing-list?

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Nick Coghlan]

On 28 January 2017 at 18:07, Barry Warsaw  wrote:
> On Jan 28, 2017, at 03:43 PM, Nick Coghlan wrote:
>
>>I still think it could be a good candidate for a first "bundled"
>>module, where we don't migrate it fully into the CPython development
>>process, but *do* officially bless it and provide it by default in the
>>form of a bundled wheel file (similar to what we do with pip).
>
> How would that work exactly.  I.e. is there a PEP?

There's no PEP, and I don't think it would make sense to have one
without a specific concrete example to drive the discussion.

I think there are 3 main candidates that could fit that bill:

- requests
- setuptools
- regex

Using requests to drive it would bring in a lot of incidental
complexity around SSL/TLS certificate management and integrating with
platform trust stores, which I think would overshadow the core policy
discussion of switching from a pure co-development model for the
standard library to one where some components are independently
versioned with recent snapshots being included in CPython feature and
maintenance branches.

For setuptools, one of our main goals in PyPA is to make it just one
build system option amongst many, as well as ensuring that it's easy
to do service deployments that don't have any build system present at
all, so bundling it by default with the stdlib would run counter to
those goals

regex by contrast is a relatively simple standalone module that offers
a more actively developed alternative to the standard library's re
module, but would still require a PEP to address the following
technical and policy questions that are normally handled by copying
code wholesale into the CPython tree under a contributor license
agreement:

- what licensing and other IP assurances would the PSF need to be
comfortable redistributing the bundled component?
- what maintenance sustainability questions would need to be
addressed? (e.g. succession planning for the ability to publish new
releases)
- how would bundled modules be handled in the CPython test suite?
- how would bundled modules be handled during local development,
during installation on supported OSes, and when creating virtual
environments?
- how does platform support in bundled modules relate to the platform
support guidelines given in PEP 11?
- how are additional build requirements for bundled modules (e.g. C++
compilers) handled?
- would bundled modules be permitted to gain features in CPython
maintenance releases, or would we expect any bundled modules to offer
conservative maintenance branches for the life of a given CPython
feature release?
- how would maintenance collaboration and issue escalation work? Would
bundled module maintainers need to follow both their own issue tracker
and the CPython one?

In many respects, PEP 453 and the bundling of pip already offers
precedent for how these are handled (e.g. it's collectively maintained
by PyPA, and that group is as closely associated with the PSF from a
legal and support perspective as python-dev is), so any PEP along
these lines would be about deciding which parts of what we did for pip
were a technical and policy precedent for 3rd party module bundling in
general, and which are unique to pip.

> While I think it could be a good idea to bundle (more?) third party packages
> for a variety of reasons, I want to make sure it's done in a way that's still
> friendly to downstream repackagers, as I'm sure you do to. :)

Indeed :)

> For Debian/Ubuntu, we already have some additional machinations to make
> ensurepip and such work properly in a distro packaging environment.  We'd want
> to be able to do the same for any bundled packages as well.

Yeah, I suspect any development along these lines would make it even
more important to have something like PEP 534 in place so that the
bundled modules could be omitted by default in some cases (such as in
virtual environments), and require people to depend on them explicitly
if they needed them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com