Re: [Python-Dev] re performance

[Wang, Peter Xihong]

Regarding to the performance difference between "re" and "regex" and packaging 
related options, we did a performance comparison using Python 3.6.0 to run some 
micro-benchmarks in the Python Benchmark Suite 
(https://github.com/python/performance):

Results in ms, and the lower the better (running on Ubuntu 15.10)
re  regex (via pip install regex, 
and a replacement of "import re" with "import regex as re")
bm_regex_compile.py 229 298
bm_regex_dna.py 171 267
bm_regex_effbot.py  2.773.04
bm_regex_v8.py  24.814.1
This data shows "re" is better than "regex" in term of performance in 3 out of 
4 above micro-benchmarks.

Anyone searching for "regular expression python" will get a first hit at the 
Python documentation on "re".  Naturally, any new developer could start with 
"re" since day 1 and not bother to look elsewhere for alternatives later on.

We did a query for "import re" against the big cloud computing software 
application, OpenStack (with 3.7 million lines of source codes and majority of 
them written in Python), and got ~1000 hits.

With that being said, IMHO, it would be nice to capture ("borrow") the 
performance benefit from "regex" and merged into "re", without knowing or 
worrying about packaging/installing stuff.

Cheers,

Peter

 

-Original Message-
From: Python-Dev 
[mailto:python-dev-bounces+peter.xihong.wang=intel@python.org] On Behalf Of 
Nick Coghlan
Sent: Tuesday, January 31, 2017 1:54 AM
To: Barry Warsaw 
Cc: python-dev@python.org
Subject: Re: [Python-Dev] re performance

On 30 January 2017 at 15:26, Barry Warsaw  wrote:
> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
>>I think there are 3 main candidates that could fit that bill:
>>
>>- requests
>>- setuptools
>>- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  
> The setuptools crew is working on a branch that would allow for 
> setuptools and pkg_resources to be split, which would be great for 
> other reasons.  Splitting them may mean that pkg_resources could 
> eventually be added to the stdlib, but as an intermediate step, it 
> could also test out this idea.  It probably has a lot less of the baggage 
> that you outline.

Yep, if/when pkg_resources is successfully split out from the rest of 
setuptools, I agree it would also be a good candidate for stdlib bundling - 
version independent runtime access to the database of installed packages is a 
key capability for many use cases, and not currently something we support 
especially well.

It's also far more analogous to the existing pip bundling, since 
setuptools/pkg_resources are also maintained under the PyPA structure.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/peter.xihong.wang%40intel.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Steve Dower]

On 30Jan2017 1310, Christian Heimes wrote:

On 2017-01-30 21:50, Cory Benfield wrote:




On 30 Jan 2017, at 13:53, David Cournapeau  wrote:

Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it 
"acceptable" for downstream packagers to patch python's default cert locations ?


There *are* no default cert locations on Windows or macOS that can be accessed 
by OpenSSL.

I cannot stress this strongly enough: you cannot provide a platform-native 
certificate validation logic for Python *and* use OpenSSL for certificate 
validation on Windows or macOS. (macOS can technically do this when you link 
against the system OpenSSL, at the cost of using a catastrophically insecure 
version of OpenSSL.)


In theory it is possible for Python and OpenSSL, too. I looked into a
custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is
trying an alternative approach in https://bugs.python.org/issue28747. It
ain't pretty and we are not there yet, too. Native support for SChannel
and SecureTransport has some benefits.


My approach there is certainly not pretty, but IMHO it's the only 
feasible way to enable this on 3.6 (that is, until we get the proper TLS 
API).


In short, I want to allow Python code to set OpenSSL's certificate 
validation callback. Basically, given a raw certificate, return 
True/False based on whether it should be trusted. I then have separate 
code (yet to be published) implementing the callback on Windows by 
calling into the WinVerifyTrust API with the HTTPS provider, which 
(allegedly) behaves identically to browsers using the same API (again, 
allegedly - I have absolutely no evidence to support this other than 
some manual testing).


The big cons are that all the OpenSSL configuration you may do becomes 
useless (and as we know, there's essentially no configuration available 
for Windows's validation), and you're delegating the most 
interesting-to-exploit part of the process to someone else,perhaps 
unknowingly, if you import something that patches SSLContext... You 
still end up with an OpenSSL-wrapped socket, but I guess it could be 
with a certificate it doesn't like? (Also a big con is that Christian 
doesn't like this approach, and his risk assessment is certainly better 
than mine :) )


Native support for secure sockets is the best approach, but it won't 
land before 3.7.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Heads up: possible double-comments on bpo for commits

[Brett Cannon]

I've activated the webhook for receiving comments on issues when a commit
lands mentioning an issue, so if you see a commit from our hg integration
and another from GitHub, understand that's why (mention issues as "bpo
" in commit messages if you want to see it in action). If it becomes
too much of a hassle to see the duplicates before we migrate I can turn off
the notifications, but obviously more testing the better. :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Paul Moore]

On 31 January 2017 at 14:54, Cory Benfield  wrote:
>
> So C# applications are Windows-native safe on Windows, and are a crapshoot 
> elsewhere. For Java vs Python, I’d say we’re slightly ahead right now.

That's precisely the sort of answer I was after. Many thanks. The
additional detail is interesting, but starts being scary again. I
think the "advantage" languages like Java has is that no-one really
discusses the details - so it seems like things are fine - but it
devolves into a "how do we get this to work?" mess if you try to do
anything hard. That's not a real advantage, but unfortunately politics
often trumps technical accuracy in my area of work :-( (My job is
often to make technically correct politically acceptable - who knew
that's what "coding" really was?)

> Again, the long-term solution to this fix is to allow us to use SChannel and 
> SecureTransport to provide TLS on the relevant platforms. This will also let 
> people use GnuTLS or NSS or whatever other TLS implementations float their 
> boat on other Unices. I’ll be bringing a draft PEP in front of python-dev 
> sometime in the next month to start this work: if you’re interested, I 
> recommend helping out with that process!

That sounds fantastic. I'm not sure how much I'd be able to help,
beyond whining "but I don't care about all this, just make it work and
make it asyy" :-) but I'll certainly watch the discussions and
do what I can.

Thanks,
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 31 Jan 2017, at 09:56, Paul Moore  wrote:
> 
> On 31 January 2017 at 09:19, Cory Benfield  wrote:
>> 
>> In general, it is unwise to mix trust stores. If you want to use your OS’s
>> trust store, the best approach is to use the OS’s TLS stack as well. At
>> least that way when a user says “It works in my browser”, you know it should
>> work for you too.
> 
> As a bystander (and an "end user" of this stuff) the message I'm
> getting here is a bit worrying. To take a step back from the sysadmin
> issues here, is the statement
> 
>It's safe to use Python (either via the stdlib, or various 3rd
> party libraries like requests) to access https URLs
> 
> correct? I understand that "safe" is a complex concept here, but in
> terms of promoting Python, I'd be using the term in the sense of "at
> least as acceptable as using something like C# or Java" - in other
> words I'm not introducing any new vulnerabilities if I argue for
> Python over one of those languages?

My answer would be “Yes*”.

The asterisk is that it works best when you understand how you are rooting your 
trust stores. On Linux, if you use your distro provided OpenSSL and its trust 
store it is as safe as the rest of your system. On macOS, if you use the system 
Python it is unsafe. If you use a Python compiled against Homebrew’s OpenSSL, 
it is safe, though it is only safe based on a snapshot in time (this is for 
boring Homebrew-related reasons that are nothing to do with Python). On 
Windows, it is safe, though potentially less functional. Requests is basically 
safe, though it is definitely possible.

If you are trying to reach websites on the open web using Requests, that’s 
safe, subject to the criteria below.

The answer gets murkier if you do substantial configuration of a trust 
database. If you have added or removed root certificates from a system trust 
store, or you have configured the trust database on a Windows or Apple machine, 
then it all gets murky fast. If you’ve only added or expanded trust then Python 
is still safe on all those platforms, it is just less functional. If you’ve 
removed trust then it is possible that a Python process will trust something 
that other processes on your machine will not.

However, you should be aware that that’s basically the default state of being 
for TLS. Java is basically in the same place as Python today, but with the 
added fun of having a hand-rolled TLS implementation (JSSE). C# is a different 
story: the stdlib uses SSPI for certificate validation, which basically means 
SChannel *on Windows*. In the wider .NET ecosystem I have no idea how this is 
being done.

So C# applications are Windows-native safe on Windows, and are a crapshoot 
elsewhere. For Java vs Python, I’d say we’re slightly ahead right now.

Again, the long-term solution to this fix is to allow us to use SChannel and 
SecureTransport to provide TLS on the relevant platforms. This will also let 
people use GnuTLS or NSS or whatever other TLS implementations float their boat 
on other Unices. I’ll be bringing a draft PEP in front of python-dev sometime 
in the next month to start this work: if you’re interested, I recommend helping 
out with that process! 

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[David Cournapeau]

On Tue, Jan 31, 2017 at 9:19 AM, Cory Benfield  wrote:

>
> On 30 Jan 2017, at 21:00, David Cournapeau  wrote:
>
>
>
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  wrote:
>
>>
>>
>> > On 30 Jan 2017, at 13:53, David Cournapeau  wrote:
>> >
>> > Are there any official recommendations for downstream packagers beyond
>> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's
>> default cert locations ?
>>
>> There *are* no default cert locations on Windows or macOS that can be
>> accessed by OpenSSL.
>>
>> I cannot stress this strongly enough: you cannot provide a
>> platform-native certificate validation logic for Python *and* use OpenSSL
>> for certificate validation on Windows or macOS. (macOS can technically do
>> this when you link against the system OpenSSL, at the cost of using a
>> catastrophically insecure version of OpenSSL.)
>>
>
> Ah, thanks, that's already useful information.
>
> Just making sure I understand: this means there is no way to use python's
> SSL library to use the system store on windows, in particular private
> certifications that are often deployed by internal ITs in large orgs ?
>
>
> If only it were that simple!
>
> No, you absolutely *can* do that. You can extract the trust roots from the
> system trust store, convert them into PEM/DER-encoded files, and load them
> into OpenSSL. That will work.
>

Right, I guess it depends on what one means by "can". In my context, it was
to be taken as "can it work without the end user having to do anything". We
provide them a python-based tool, and it has to work with the system store
out of the box. If the system store is updated through e.g. group policy,
our python tool automatically get that update.

>From the sound of it, it looks like this is simply not possible ATM with
python, at least not without 3rd party libraries.

David


> The problem is that both SecureTransport and SChannel have got a number of
> differences from OpenSSL. In no particular order:
>
> 1. Their chain building logic is different. This means that, given a
> collection of certificates presented by a server and a bundle of
> already-trusted certs, each implementation may build a different trust
> chain. This may cause one implementation to refuse to validate where the
> others do, or vice versa. This is very common with older OpenSSLs.
> 2. SecureTransport and SChannel both use the system trust DB, which on
> both Windows and mac allows the setting of custom policies. OpenSSL won’t
> respect these policies, which means you can fail-open (that is, export and
> use a root certificate that the OS believes should not be trusted for a
> given use case). There is no way to export these trust policies into
> OpenSSL.
> 3. SecureTransport, SChannel, and OpenSSL all support different X.509
> extensions and understand them differently. This means that some certs may
> be untrusted for certain uses by Windows but trusted for those uses by
> OpenSSL, for example.
>
> In general, it is unwise to mix trust stores. If you want to use your OS’s
> trust store, the best approach is to use the OS’s TLS stack as well. At
> least that way when a user says “It works in my browser”, you know it
> should work for you too.
>
> Cory
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Paul Moore]

On 31 January 2017 at 09:19, Cory Benfield  wrote:
>
> In general, it is unwise to mix trust stores. If you want to use your OS’s
> trust store, the best approach is to use the OS’s TLS stack as well. At
> least that way when a user says “It works in my browser”, you know it should
> work for you too.

As a bystander (and an "end user" of this stuff) the message I'm
getting here is a bit worrying. To take a step back from the sysadmin
issues here, is the statement

It's safe to use Python (either via the stdlib, or various 3rd
party libraries like requests) to access https URLs

correct? I understand that "safe" is a complex concept here, but in
terms of promoting Python, I'd be using the term in the sense of "at
least as acceptable as using something like C# or Java" - in other
words I'm not introducing any new vulnerabilities if I argue for
Python over one of those languages?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] re performance

[Nick Coghlan]

On 30 January 2017 at 15:26, Barry Warsaw  wrote:
> On Jan 30, 2017, at 12:38 PM, Nick Coghlan wrote:
>
>>I think there are 3 main candidates that could fit that bill:
>>
>>- requests
>>- setuptools
>>- regex
>
> Actually, I think pkg_resources would make an excellent candidate.  The
> setuptools crew is working on a branch that would allow for setuptools and
> pkg_resources to be split, which would be great for other reasons.  Splitting
> them may mean that pkg_resources could eventually be added to the stdlib, but
> as an intermediate step, it could also test out this idea.  It probably has a
> lot less of the baggage that you outline.

Yep, if/when pkg_resources is successfully split out from the rest of
setuptools, I agree it would also be a good candidate for stdlib
bundling - version independent runtime access to the database of
installed packages is a key capability for many use cases, and not
currently something we support especially well.

It's also far more analogous to the existing pip bundling, since
setuptools/pkg_resources are also maintained under the PyPA structure.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Generator objects and list comprehensions?

[Nick Coghlan]

On 30 January 2017 at 19:05, Brett Cannon  wrote:
> On Sun, 29 Jan 2017 at 16:39 Craig Rodrigues  wrote:
>> I'm OK with either approach.  Leaving things the way they are in Python 3
>> is no good, IMHO.
>
> My vote is it be a SyntaxError since you're not getting what you expect from
> the syntax.

I'd agree that's a sensible place for us to end up, as any code
relying on the current behaviour is really too clever to be
maintainable.

In terms of getting there, we'll likely want:

- SyntaxWarning or DeprecationWarning in 3.7
- Py3k warning in 2.7.x
- SyntaxError in 3.8

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 31 Jan 2017, at 09:33, Christian Heimes  wrote:
> 
> One small correction, it is possible to export some of the trust
> settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works
> correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in
> X509_AUX extension after the actual certificate and signature. OpenSSL's
> default loaders for cert dir and cert file do load auxiliary trust
> information.

Ah, good spot.

I suspect the code you’d need to write to safely extract that functionality is 
pretty subtle. I definitely don’t trust myself to get it right.

Cory
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Christian Heimes]

On 2017-01-31 10:19, Cory Benfield wrote:
> 
>> On 30 Jan 2017, at 21:00, David Cournapeau > > wrote:
>>
>>
>>
>> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield > > wrote:
>>
>>
>>
>> > On 30 Jan 2017, at 13:53, David Cournapeau > > wrote:
>> >
>> > Are there any official recommendations for downstream packagers beyond 
>> PEP 476 ? Is it "acceptable" for downstream packagers to patch python's 
>> default cert locations ?
>>
>> There *are* no default cert locations on Windows or macOS that can
>> be accessed by OpenSSL.
>>
>> I cannot stress this strongly enough: you cannot provide a
>> platform-native certificate validation logic for Python *and* use
>> OpenSSL for certificate validation on Windows or macOS. (macOS can
>> technically do this when you link against the system OpenSSL, at
>> the cost of using a catastrophically insecure version of OpenSSL.)
>>
>>
>> Ah, thanks, that's already useful information.
>>
>> Just making sure I understand: this means there is no way to use
>> python's SSL library to use the system store on windows, in particular
>> private certifications that are often deployed by internal ITs in
>> large orgs ?
> 
> If only it were that simple!
> 
> No, you absolutely *can* do that. You can extract the trust roots from
> the system trust store, convert them into PEM/DER-encoded files, and
> load them into OpenSSL. That will work.
> 
> The problem is that both SecureTransport and SChannel have got a number
> of differences from OpenSSL. In no particular order:
> 
> 1. Their chain building logic is different. This means that, given a
> collection of certificates presented by a server and a bundle of
> already-trusted certs, each implementation may build a different trust
> chain. This may cause one implementation to refuse to validate where the
> others do, or vice versa. This is very common with older OpenSSLs.
> 2. SecureTransport and SChannel both use the system trust DB, which on
> both Windows and mac allows the setting of custom policies. OpenSSL
> won’t respect these policies, which means you can fail-open (that is,
> export and use a root certificate that the OS believes should not be
> trusted for a given use case). There is no way to export these trust
> policies into OpenSSL.

One small correction, it is possible to export some of the trust
settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works
correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in
X509_AUX extension after the actual certificate and signature. OpenSSL's
default loaders for cert dir and cert file do load auxiliary trust
information.

(Of course the feature is experimental and was broken in 1.0.2 for a
long time until I discovered the issue and fixed it...)


https://linux.die.net/man/1/x509

TRUST SETTINGS

Please note these options are currently experimental and may well change.

A trusted certificate is an ordinary certificate which has several
additional pieces of information attached to it such as the permitted
and prohibited uses of the certificate and an "alias".

Normally when a certificate is being verified at least one certificate
must be "trusted". By default a trusted certificate must be stored
locally and must be a root CA: any certificate chain ending in this CA
is then usable for any purpose.

Trust settings currently are only used with a root CA . They allow a
finer control over the purposes the root CA can be used for. For example
a CA may be trusted for SSL client but not SSL server use.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] SSL certificates recommendations for downstream python packagers

[Cory Benfield]

> On 30 Jan 2017, at 21:00, David Cournapeau  wrote:
> 
> 
> 
> On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield  > wrote:
> 
> 
> > On 30 Jan 2017, at 13:53, David Cournapeau  > > wrote:
> >
> > Are there any official recommendations for downstream packagers beyond PEP 
> > 476 ? Is it "acceptable" for downstream packagers to patch python's default 
> > cert locations ?
> 
> There *are* no default cert locations on Windows or macOS that can be 
> accessed by OpenSSL.
> 
> I cannot stress this strongly enough: you cannot provide a platform-native 
> certificate validation logic for Python *and* use OpenSSL for certificate 
> validation on Windows or macOS. (macOS can technically do this when you link 
> against the system OpenSSL, at the cost of using a catastrophically insecure 
> version of OpenSSL.)
> 
> Ah, thanks, that's already useful information.
> 
> Just making sure I understand: this means there is no way to use python's SSL 
> library to use the system store on windows, in particular private 
> certifications that are often deployed by internal ITs in large orgs ?

If only it were that simple!

No, you absolutely *can* do that. You can extract the trust roots from the 
system trust store, convert them into PEM/DER-encoded files, and load them into 
OpenSSL. That will work.

The problem is that both SecureTransport and SChannel have got a number of 
differences from OpenSSL. In no particular order:

1. Their chain building logic is different. This means that, given a collection 
of certificates presented by a server and a bundle of already-trusted certs, 
each implementation may build a different trust chain. This may cause one 
implementation to refuse to validate where the others do, or vice versa. This 
is very common with older OpenSSLs.
2. SecureTransport and SChannel both use the system trust DB, which on both 
Windows and mac allows the setting of custom policies. OpenSSL won’t respect 
these policies, which means you can fail-open (that is, export and use a root 
certificate that the OS believes should not be trusted for a given use case). 
There is no way to export these trust policies into OpenSSL.
3. SecureTransport, SChannel, and OpenSSL all support different X.509 
extensions and understand them differently. This means that some certs may be 
untrusted for certain uses by Windows but trusted for those uses by OpenSSL, 
for example.

In general, it is unwise to mix trust stores. If you want to use your OS’s 
trust store, the best approach is to use the OS’s TLS stack as well. At least 
that way when a user says “It works in my browser”, you know it should work for 
you too.

Cory

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com