Re: [Python-ideas] Unified TLS API for Python

[Christian Heimes]

On 2017-02-04 19:18, Cory Benfield wrote:
> 
>> On 3 Feb 2017, at 18:30, Steve Dower  wrote:
>>
>> On 02Feb2017 0601, Cory Benfield wrote:
>>>
>>> 4. Eventually, integrating the two backends above into the standard
>>> library so that it becomes possible to reduce the reliance on OpenSSL.
>>> This would allow future Python implementations to ship with all of their
>>> network protocol libraries supporting platform-native TLS
>>> implementations on Windows and macOS. This will almost certainly require
>>> new PEPs. I’ll probably volunteer to maintain a SecureTransport library,
>>> and I have got verbal suggestions from some other people who’d be
>>> willing to step up and help with that. Again, we’d need help with
>>> SChannel (looking at you, Steve).
>>
>> I'm always somewhat interested in learning a new API that I've literally 
>> never looked at before, so yeah, count me in :) (my other work was using the 
>> trust APIs directly, rather than the secure socket APIs).
>>
>> PyCon US sprints? It's not looking like I'll be able to set aside too much 
>> time before then, but I've already fenced off that time.
> 
> That might be a really good idea.
> 
> With feedback from Nathaniel and Glyph I’m going back to the drawing board a 
> bit with this PEP to see if we can reduce the amount of work needed from 
> backends, so this may shrink down to something that can feasibly be done in a 
> sprint.
> 
> For those who are interested, the refactoring proposed by Nathaniel and Glyph 
> is to drop the abstract TLSWrappedSocket class, and instead replace it with a 
> *concrete* TLSWrappedSocket class that is given a socket and an 
> implementation of TLSWrappedBuffers. This would essentially mean that 
> implementations only need to write a TLSWrappedBuffers implementation and 
> would get a TLSWrappedSocket essentially for free (with the freedom to 
> provide a complete reimplementation of TLSWrappedSocket if they need to).
> 
> I’m going to investigate the feasibility of that by writing a stub 
> TLSWrappedBuffers for SecureTransport and then writing the TLSWrappedSocket 
> implementation to validate what it looks like. Assuming the end result looks 
> generally suitable, I’ll come back with a new draft. But the TL;DR is that if 
> we do that all we need to implement on the Windows side is one-or-two 
> classes, and we’d be ready to go. That’d be really nice.

At first I was a bit worried that you were planning to chance the
semantic of socket wrapping. The PEP doesn't get into detail how
wrap_socket() works internally. How about you add a paragraph that the
function wraps an OS-level file descriptors (POSIX) or socket handles
(Windows).

For some TLS libraries it's an optimization to reduce memory copies and
overhead of extra GIL lock/unlock calls. Other TLS libraries (NSS, Linux
kernel TLS with AF_KTLS) can only operator on file descriptors. In fact
NSS can only operator on NSPR PRFileDesc, but NSPR has an API to wrap an
OS-level fd in a PRFileDesc.

By the way, how can a TLS implementation announce that it does not
provide buffer wrapping?

Christian



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Unified TLS API for Python

[Cory Benfield]

> On 3 Feb 2017, at 18:30, Steve Dower  wrote:
> 
> On 02Feb2017 0601, Cory Benfield wrote:
>> 
>> 4. Eventually, integrating the two backends above into the standard
>> library so that it becomes possible to reduce the reliance on OpenSSL.
>> This would allow future Python implementations to ship with all of their
>> network protocol libraries supporting platform-native TLS
>> implementations on Windows and macOS. This will almost certainly require
>> new PEPs. I’ll probably volunteer to maintain a SecureTransport library,
>> and I have got verbal suggestions from some other people who’d be
>> willing to step up and help with that. Again, we’d need help with
>> SChannel (looking at you, Steve).
> 
> I'm always somewhat interested in learning a new API that I've literally 
> never looked at before, so yeah, count me in :) (my other work was using the 
> trust APIs directly, rather than the secure socket APIs).
> 
> PyCon US sprints? It's not looking like I'll be able to set aside too much 
> time before then, but I've already fenced off that time.

That might be a really good idea.

With feedback from Nathaniel and Glyph I’m going back to the drawing board a 
bit with this PEP to see if we can reduce the amount of work needed from 
backends, so this may shrink down to something that can feasibly be done in a 
sprint.

For those who are interested, the refactoring proposed by Nathaniel and Glyph 
is to drop the abstract TLSWrappedSocket class, and instead replace it with a 
*concrete* TLSWrappedSocket class that is given a socket and an implementation 
of TLSWrappedBuffers. This would essentially mean that implementations only 
need to write a TLSWrappedBuffers implementation and would get a 
TLSWrappedSocket essentially for free (with the freedom to provide a complete 
reimplementation of TLSWrappedSocket if they need to).

I’m going to investigate the feasibility of that by writing a stub 
TLSWrappedBuffers for SecureTransport and then writing the TLSWrappedSocket 
implementation to validate what it looks like. Assuming the end result looks 
generally suitable, I’ll come back with a new draft. But the TL;DR is that if 
we do that all we need to implement on the Windows side is one-or-two classes, 
and we’d be ready to go. That’d be really nice.

Cory
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Unified TLS API for Python

[Steve Dower]

On 02Feb2017 0601, Cory Benfield wrote:


4. Eventually, integrating the two backends above into the standard
library so that it becomes possible to reduce the reliance on OpenSSL.
This would allow future Python implementations to ship with all of their
network protocol libraries supporting platform-native TLS
implementations on Windows and macOS. This will almost certainly require
new PEPs. I’ll probably volunteer to maintain a SecureTransport library,
and I have got verbal suggestions from some other people who’d be
willing to step up and help with that. Again, we’d need help with
SChannel (looking at you, Steve).


I'm always somewhat interested in learning a new API that I've literally 
never looked at before, so yeah, count me in :) (my other work was using 
the trust APIs directly, rather than the secure socket APIs).


PyCon US sprints? It's not looking like I'll be able to set aside too 
much time before then, but I've already fenced off that time.


Cheers,
Steve

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Unified TLS API for Python

[Cory Benfield]

> On 2 Feb 2017, at 11:17, Paul Moore  wrote:
> 
> One thing that wasn't clear to me was the backward compatibility
> implications. Will the existing ssl module and its API be retained, or
> will it be modified/deprecated? Will Windows/OSX distributions of
> Python ultimately stop shipping with OpenSSL included? From an end
> user POV, I won't care (high level modules like asyncio, urllib[1],
> http.client etc will be updated to transparently use the new
> infrastructure) so I accept that this is something of an "internal"
> and/or library author concern.

These are all good questions.

For the existing ssl module and its API, deprecating it is hard work: it’s used 
extremely widely, and I don’t think we’d ever remove it. I suspect we’d warn 
against using it in favour of programming to the new tls API, but we’d keep it 
in place partly because it is so heavily used today, and partly because it’ll 
form the basis of the OpenSSL implementation of the abstract API here.

As to stopping shipping with OpenSSL, that’s unlikely. OpenSSL doesn’t just 
back the ssl module, it backs hashlib as well (via libcrypto). While it’s 
potentially possible to replace most of hashlib with constructs from 
platform-native implementations (CommonCrypto and CryptoNG), that’s a lot of 
work and hard to do, as the cryptography.io  folks 
would tell you. So I suspect we’ll end up shipping with both, but the 
importance of OpenSSL for TLS would drop dramatically (and it would even 
potentially be possible to link only against libcrypto instead of libssl).

> [1] urllib wasn't actually a module noted as needing change. Was that
> an omission, or is it simply because it defers the TLS stuff to the
> lower levels like http.client?

That was an omission: I forgot to check whether it needed updating. It does. 
I’ve amended the PEP accordingly.

Cory___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Unified TLS API for Python

[Cory Benfield]

> On 2 Feb 2017, at 11:03, Robert Collins  wrote:
> 
> Nice. What's needed to move this forward?

In the short term, for the PEP, nothing more than review and sanity checking. 
Antione has kindly provided some detailed review on GitHub (thanks Antoine!), 
and the more of that we get the better. Once review has quieted down for a few 
days, I’ll take that as something approximating consensus from this list, and 
I’ll kick it over to python-dev and start the formal PEP process.

Longer term, the following chunks of work will be needed:

1. Providing a shim for the current ssl module. The ssl module has a number of 
objects that don’t quite cleave to this API, so we’d probably want to either 
add new objects that do that (likely by wrapping the current ones), or we’d 
want a whole-new module that cleaves to this API. I’ll discuss more in my 
response to Paul what I think we’d do with the ssl module longer-term.
2. Code needs to be written for the identified network libraries to have them 
use the new API. There’s no point doing this until (1) is done, as they’ll 
otherwise have no ability to use any TLS API at all, but transitioning over to 
the new API starts providing some real community value as third-party 
implementations can start being used with stdlib network modules.
3. Backends for SChannel and SecureTransport would need to be written. The most 
likely response I expect from Guido and others in python-dev is that they’d 
like to see example implementations for SChannel and SecureTransport that fit 
this API. I am likely to stub out a SecureTransport implementation sometime 
soon, as I have some experience with that API, and it’d make a good test-bed 
for the PEP. I’m a bad fit for doing SChannel though: I don’t know Windows well 
enough, and my ramp-up time would be high. I’ll scout out to see if there’s 
someone who’s willing to stub this out, though.
4. Eventually, integrating the two backends above into the standard library so 
that it becomes possible to reduce the reliance on OpenSSL. This would allow 
future Python implementations to ship with all of their network protocol 
libraries supporting platform-native TLS implementations on Windows and macOS. 
This will almost certainly require new PEPs. I’ll probably volunteer to 
maintain a SecureTransport library, and I have got verbal suggestions from some 
other people who’d be willing to step up and help with that. Again, we’d need 
help with SChannel (looking at you, Steve).

These are all valuable things to do, and I’d accept help with any and all of 
them. Certainly we need to do (1) and (2) to start getting real value out of 
this PEP, with (3) being something that could be done third-party initially 
(and so don’t have to land in Python 3.7).

Cory___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Unified TLS API for Python

[Cory Benfield]

All,

A recent change in the Fastly policy has led the distutils-sig to discover that 
there is a very real risk of many macOS installations of Python being entirely 
unable to access PyPI using pip, due to the fact that they are linked against 
the ancient system OpenSSL (see [1]). This problem ultimately stems from the 
fact that Python has never exposed the ability to use TLS backends other than 
OpenSSL in the standard library. As Christian and I discussed back at PyCon US 
2015, the ssl module would more properly be called the openssl module, due to 
exposing many OpenSSL-specific concepts and behaviours in its API. This has 
meant that the Python ecosystem is overwhelmingly also an OpenSSL ecosystem, 
which is problematic on Windows and macOS (and Unices that for whatever reason 
aren’t interested in shipping an OpenSSL), as it has meant that Python needs to 
bring its own OpenSSL, and that it is troublesome to interact with the system 
trust database.

The first step to resolving this would be to provide a new module that exposes 
TLS concepts in a much more generic manner. There are two possible approaches 
to this. The first is to have this module be a generic concrete implementation 
that can be compiled against multiple TLS backends (like curl). This would 
require that all the relevant bindings be built into the standard library from 
the get-go, which provides a substantial maintenance burden on a team of people 
that are already understaffed maintaining the ssl module alone. The second 
approach is to define a generic high-level TLS interface that provides a 
minimal usable interface that can be implemented by both first- and third-party 
modules. This would allow the standard library to continue to ship with only 
exactly what it needs (for example, OpenSSL, SecureTransport and SChannel), but 
would give those who want to use more esoteric TLS choices (NSS, GnuTLS, 
mbedTLS, and BoringSSL are some examples) an API that they can implement that 
will guarantee that complying modules can use the appropriate TLS backend.

To that end, I’ve composed a draft PEP that would define this API. The current 
copy can be found on GitHub[2] and is also provided below. This draft PEP has 
been under discussion in the Python Security-SIG since the start of January, 
and has gone through a number of revisions. This is the next step in moving 
this forward: exposing the wider Python development community to the PEP for 
further revision before it is proposed to python-dev.

Please note that this proposal is not intended to resolve the current problem 
pip has with TLSv1.2, and so is not subject to the tight time constraints 
associated with that problem. That problem will be solved by building a 
temporary shim into urllib3 that can use SecureTransport bindings on the Mac. 
This proposal is intended as a solution to the more-general problem of 
supporting different TLS backends in Python, and so is larger in scope and less 
urgent.

I should also mention that there will be a tendency to want to make this API 
all things to all people from the get-go. I’m going to strongly resist attempts 
to extend this API too much, because each additional bit of API surface makes 
it harder for us to encourage module authors to conform to this API. I will 
encourage people to extend this API over time as needed, but to begin with I 
think it is most important that basic TLS clients and servers can be 
constructed with this API. More specialised features should be considered 
future enhancements, rather than being tacked on to this initial PEP.

Please let me know what you think.

Cory

[1]: https://mail.python.org/pipermail/distutils-sig/2017-January/029970.html
[2]: https://github.com/Lukasa/peps/pull/1

—

PEP: XXX
Title: TLS Abstract Base Classes
Version: $Revision$
Last-Modified: $Date$
Author: Cory Benfield 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 17-Oct-2016
Python-Version: 3.7
Post-History: 30-Aug-2002


Abstract


This PEP would define a standard TLS interface in the form of a collection of
abstract base classes. This interface would allow Python implementations and
third-party libraries to provide bindings to TLS libraries other than OpenSSL
that can be used by tools that expect the interface provided by the Python
standard library, with the goal of reducing the dependence of the Python
ecosystem on OpenSSL.


Rationale
=

In the 21st century it has become increasingly clear that robust and
user-friendly TLS support is an extremely important part of the ecosystem of
any popular programming language. For most of its lifetime, this role in the
Python ecosystem has primarily been served by the `ssl module`_, which provides
a Python API to the `OpenSSL library`_.

Because the ``ssl`` module is distributed with the Python standard library, it
has become the overwhelmingly most-popular method for handling TLS in Python.
An extraordinary majority of Python