Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 07:45, Nick Coghlan wrote: > There's also the fact that most corporate Python users are > unlikely to know that PyPI exists, let alone that it contains a module > called "requests" that does SSL certificate validation by default. > Those of us in the corporate world that interact directly with > upstream are still the exception rather than the rule) I think this point deserves just a little bit more emphasis. This is why any solution that begins with 'use PyPI' is insufficient. I've worked on requests for 3 years now and most of my colleagues have never heard of it, and it's not because I don't talk about it (I talk about it all the time!). When building internal tools, corporate environments frequently restrict themselves to the standard library. This is because it's hard enough to get adoption of a tool when it requires a new language runtime, let alone if you have to get people ramped up on package distribution as well! I have enough trouble getting people to upgrade Python versions at work: trying to get them up to speed on pip and PyPI is worse. It is no longer tenable in the long term for Python to trust the network: you're right in this regard Nick. In the past, on this very list, I've been bullish about fixing up Python's network security position. I was an aggressive supporter of PEP 466 (and there are some corners of PEP 466 that I think didn't go far enough). However, I'm with you here: we should do this once and do it right. Corporate users *will* bump into it, and they will look to the docs to fix it. That fix needs to be easy and painless. A user-level cert store is a good start, and if cryptography.io aren't interested in it I might take a look at implementing it under the certifi.io umbrella instead. Cory ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 00:22, Antoine Pitrou wrote: > SSL_CERT_DIR and SSL_CERT_FILE are used, if set, when > SSLContext.load_verify_locations() is called. > > Actually, come to think of it, this allows us to write a better > test for that method. Patch welcome! The environment vars are used only when SSLContext.set_default_verify_paths() is called. load_verify_locations() loads certificates from a given file, directory or memory but it doesn't look at the env vars. create_default_context() calls SSLContext.load_default_certs() when neither cafile, capath nor cadata is given as an argument. SSLContext.load_default_certs() then calls SSLContext.set_default_verify_paths(). However there is a catch: SSLContext.set_default_verify_paths() is not called on Windows. In retrospective it was a bad decision by me to omit the call. http://hg.python.org/cpython/file/164a17eca081/Lib/ssl.py#l376 Christian PS: SSL_CERT_DIR and SSL_CERT_FILE are the default names. It's possible to change the names in OpenSSL. ssl.get_default_verify_paths() returns the names and paths to the default verify locations. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
HTML version:
http://legacy.python.org/dev/peps/pep-0475/
PEP: 475
Title: Retry system calls failing with EINTR
Version: $Revision$
Last-Modified: $Date$
Author: Charles-François Natali , Victor Stinner
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 29-July-2014
Python-Version: 3.5
Abstract
Retry system calls failing with the ``EINTR`` error and recompute
timeout if needed.
Rationale
=
Interrupted system calls
On POSIX systems, signals are common. Your program must be prepared to
handle them. Examples of signals:
* The most common signal is ``SIGINT``, signal sent when CTRL+c is
pressed. By default, Python raises a ``KeyboardInterrupt`` exception
when this signal is received.
* When running subprocesses, the ``SIGCHLD`` signal is sent when a
child process exits.
* Resizing the terminal sends the ``SIGWINCH`` signal to the
applications running in the terminal.
* Putting the application in background (ex: press CTRL-z and then
type the ``bg`` command) sends the ``SIGCONT`` signal.
Writing a signal handler is difficult, only "async-signal safe"
functions can be called. For example, ``printf()`` and ``malloc()``
are not async-signal safe. When a signal is sent to a process calling
a system call, the system call can fail with the ``EINTR`` error to
give the program an opportunity to handle the signal without the
restriction on signal safe functions. Depending on the platform, on
the system call and the ``SA_RESTART`` flag, the system call may or
may not fail with ``EINTR``.
If the signal handler was set with the ``SA_RESTART`` flag set, the
kernel retries some the system call instead of failing with
``EINTR``. For example, ``read()`` is retried, whereas ``select()`` is
not retried. The Python function ``signal.signal()`` clears the
``SA_RESTART`` flag when setting the signal handler: all system calls
should fail with ``EINTR`` in Python.
The problem is that handling ``EINTR`` should be done for all system
calls. The problem is similar to handling errors in the C language
which does not have exceptions: you must check all function returns to
check for error, and usually duplicate the code checking for errors.
Python does not have this issue, it uses exceptions to notify errors.
Current status
--
Currently in Python, the code to handle the ``InterruptedError``
exception (``EINTR`` error) is duplicated on case by case. Only a few
Python modules handle this exception, and fixes usually took several
years to cover a whole module. Example of code retrying
``file.read()`` on ``InterruptedError``::
while True:
try:
data = file.read(size)
break
except InterruptedError:
continue
List of Python modules of the standard library which handle
``InterruptedError``:
* ``asyncio``
* ``asyncore``
* ``io``, ``_pyio``
* ``multiprocessing``
* ``selectors``
* ``socket``
* ``socketserver``
* ``subprocess``
Other programming languages like Perl, Java and Go already retry
system calls failing with ``EINTR``.
Use Case 1: Don't Bother With Signals
-
In most cases, you don't want to be interrupted by signals and you
don't expect to get ``InterruptedError`` exceptions. For example, do
you really want to write such complex code for an "Hello World"
example?
::
while True:
try:
print("Hello World")
break
except InterruptedError:
continue
``InterruptedError`` can happen in unexpected places. For example,
``os.close()`` and ``FileIO.close()`` can raises ``InterruptedError``:
see the article `close() and EINTR
`_.
The `Python issues related to EINTR`_ section below gives examples of
bugs caused by "EINTR".
The expectation is that Python hides the ``InterruptedError``: retry
system calls failing with the ``EINTR`` error.
Use Case 2: Be notified of signals as soon as possible
--
Sometimes, you expect some signals and you want to handle them as soon
as possible. For example, you may want to quit immediatly a program
using the ``CTRL+c`` keyboard shortcut.
Some signals are not interesting and should not interrupt the the
application. There are two options to only interrupt an application
on some signals:
* Raise an exception in the signal handler, like ``KeyboardInterrupt`` for
``SIGINT``
* Use a I/O multiplexing function like ``select()`` with the Python
signal "wakeup" file descriptor: see the function
``signal.set_wakeup_fd()``.
Proposition
===
If a system call fails with ``EINTR``, Python must call signal
handlers: call ``PyErr_CheckSignals()``. If a signal handler raises
an exception, the Python function fails with the exception.
Otherwise, the system call is retried. If the system call takes a
timeout parameter, the timeout is recomputed.
Modified functions
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Sun, 31 Aug 2014 16:45:42 +1000, Nick Coghlan wrote: > On 31 August 2014 16:16, Donald Stufft wrote: > > > > On Aug 31, 2014, at 2:09 AM, Nick Coghlan wrote: > > > > At the same time, we need to account for the fact that most existing > > organisations still trust in perimeter defence for their internal > > network security, and hence tolerate (or even actively encourage) the > > use of unsecured connections, or skipping certificate validation, > > internally. This is actually a really terrible idea, but it's still > > incredibly common due to the general failure of the technology > > industry to take usability issues seriously when we design security > > systems (at least until recently) - doing the wrong "unsafe" thing is > > genuinely easier than doing things right. > > > > > > Just a quick clarification in order to be a little clearer, this change will > > (obviously) only effect those who trust perimeter security *and* decided to > > install an invalid certificate instead of just using HTTP. I'm not saying > > that > > this doesn't happen, just being specific (I'm not actually sure why they > > would > > install a TLS certificate at all if they are trusting perimeter security, > > but > > I'm sure folks do). > > It's the end result when a company wide edict to use HTTPS isn't > backed up by the necessary documentation and training on how to get a > properly signed cert from your internal CA (or, even better, when such > an edict comes down without setting up an internal CA first). Folks > hit the internet instead, find instructions on creating a self-signed > cert, install that, and tell their users to ignore the security > warning and accept the cert. Historically, Python clients have "just > worked" in environments that required a click-through on the browser > side, since you had to opt in to checking the certificates properly. > > Self-signed certificates can also be really handy for doing local > testing - you're not really aiming to authenticate the connection in > that case, you're just aiming to test that the secure connection > machinery is all working properly. Self-signed certificates are not crazy in an internal corporate environment even when properly playing the defense in depth game. Once you've acked the cert the first time, you will be warned if it changes (like an ssh host key). Sure, as Nick says the corp could set up an internal signing authority and make sure everyone has their CA...and they *should*...but realistically, that is probably relatively rare at the moment, because it is not particularly easy to accomplish (distributing the CA everywhere it needs to go is still a Hard Problem, though it has gotten a lot better). Given the reality of human nature, even when the documentation accompanying the HTTPS initiative is good, there will *still* be someone who hasn't followed the internal rules, yet you really need to talk to the piece of infrastructure they are maintaining. At least that one is short term problem (for some definition of "short" that may be several months long), but it does exist. In addition, as has been mentioned before, self-signed certs are often embedded in *devices* from vendors (I'm looking at you, Cisco). This is another area where security conciousness has gotten better (the cert exists) but isn't good yet (the cert is self-signed and replacing it isn't trivial when it is even possible; and, because the self-signed cert happens by defaultit gets left in place). And in the case of those embedded certs, the cert can wind up *invalid* (expired) as well as self-signed. (This last item is where my concern about being able to talk to invalid certs comes from.) And yes, I have encountered all of this in the wild. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [libmpdec] mpdecimal-2.4.1 released
Hi, I've released mpdecimal-2.4.1: http://www.bytereef.org/mpdecimal/changelog.html da74d3cfab559971a4fbd4fb506e1b4498636eb77d0fd09e44f8e546d18ac068 mpdecimal-2.4.1.tar.gz Starting with Python 3.4.2, this version should be used for an external libmpdec. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner : > Proposition > === > > If a system call fails with ``EINTR``, Python must call signal > handlers: call ``PyErr_CheckSignals()``. If a signal handler raises > an exception, the Python function fails with the exception. > Otherwise, the system call is retried. If the system call takes a > timeout parameter, the timeout is recomputed. Signals are tricky and easy to get wrong, to be sure, but I think it is dangerous for Python to unconditionally commandeer signal handling. If the proposition is accepted, there should be a way to opt out. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 16:16, R. David Murray wrote: > Self -signed certificates are not crazy in an internal corporate > environment even when properly playing the defense in depth game. Once > you've acked the cert the first time, you will be warned if it changes > (like an ssh host key). Sure, as Nick says the corp could set up an > internal signing authority and make sure everyone has their CA...and > they *should*...but realistically, that is probably relatively rare at > the moment, because it is not particularly easy to accomplish > (distributing the CA everywhere it needs to go is still a Hard Problem, > though it has gotten a lot better). It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. That's all. A self-signed certificate acts as its own root CA (so to speak). But there is a downside, too. The certificate is trusted for any and all connections. Python's SSL module has no way to trust a specific certificate for a host. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 17:27, Christian Heimes wrote: > It's very simple to trust a self-signed certificate: just download it > and stuff it into the trust store. "Stuff it into the trust store" is the hard bit, though. I have honestly no idea how to do that. Or if it's temporary (which it likely is) how to manage it - delete it when I no longer need it, list what junk I've added over time, etc. And equally relevantly, no idea how to do that in a way that won't clash with my company's policies... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:24, Nick Coghlan wrote: > To answer David's specific question, the existing knobs at the OpenSSL > level (SSL_CERT_DIR and SSL_CERT_FILE ) let people add an internal CA, > opt out of the default CA system, and trust *specific* self-signed > certs. This works only on Unix platforms iff SSL_CERT_DIR and SSL_CERT_FILE are both set to a non-empty string that points to non-existing files or something like /dev/null. On Windows my enhancement will always cause the system trust store to kick in. There is currently no way to disable the Windows system store for ssl.create_default_context() and ssl._create_stdlib_context() with the functions' default arguments. On Mac OS X the situation is even more obscure. Apple's OpenSSL binaries are using Apple's Trust Evaluation Agent. You have to set OPENSSL_X509_TEA_DISABLE=1 in order to prevent the agent from adding trusted certs from OSX key chain. Hynek Schlawack did a deep dive into it. https://hynek.me/articles/apple-openssl-verification-surprises/ > A Python-specific user level cert store is something that could be > developed as a PyPI library for Python 2.7.9+ and 3.4+ (Is cert > management considered in scope for cryptography.io? If so, that could > be a good home). Python's SSL module is lacking some functionalities in order to implement a fully functional cert store. * no verify hook to verify each certificate in the chain like https://www.openssl.org/docs/ssl/SSL_CTX_set_cert_verify_callback.html http://linux.die.net/man/3/x509_store_ctx_set_verify_cb /api/ssl.html#OpenSSL.SSL.Context.set_verify * no way to get the full cert chain including the root certificate. * no API to get the subject public key information (SPKI). The SPKI hash can be used to identify a certificate. For example it's used in Google's CRLSet. http://dev.chromium.org/Home/chromium-security/crlsets * the cert validation exception could use some additional information. There are probably some more things mising. An X509 object would help, too. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 19:03, Paul Moore a écrit : On 31 August 2014 17:27, Christian Heimes wrote: It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. "Stuff it into the trust store" is the hard bit, though. I have honestly no idea how to do that. You certainly shouldn't do so. If an application has special needs that require trusting a self-signed certificate, then it should expose a configuration setting to let users specify the cert's location. Stuffing self-signed certs into the system trust store is really a measure of last resort. There's another case which isn't solved by this, though, which is when a cert is invalid. The common situation being that it has expired (renewing certs is a PITA and therefore expired certs are more common than it sounds they should be). In this case, there is no way to whitelist it: you have to disable certificate checking altogether. This can be exposed by the application as configuration option if necessary, as well. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 18:29, Antoine Pitrou wrote: > If an application has special needs that require trusting a self-signed > certificate, then it should expose a configuration setting to let users > specify the cert's location. I can't see how that would be something the application would know. For example, pip allows me to specify an "alternate cert bundle" but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions. Or did you mean that Python should provide such a setting that would cover all applications written in Python? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 20:28, Paul Moore a écrit : I can't see how that would be something the application would know. For example, pip allows me to specify an "alternate cert bundle" but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions. Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? pip knows about the use cases here, Python doesn't. (perhaps you want to serve your local index using http, though) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 19:37, Antoine Pitrou wrote: > Well, it's certainly pip's responsibility more than Python's. What would > Python do? Provide a setting that would blindly add a cert for all uses of > httplib? That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case. You said that doing so was bad, but I don't see why. It seems to me that you're saying that I should raise a feature request for pip instead, which seems unreasonable. Am I missing something? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 21:12, Paul Moore a écrit : On 31 August 2014 19:37, Antoine Pitrou wrote: Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case. What do you call your local cert store? If you mean the system cert store, then that will affect all users. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 19:29, Antoine Pitrou wrote: > You certainly shouldn't do so. If an application has special needs that > require trusting a self-signed certificate, then it should expose a > configuration setting to let users specify the cert's location. Stuffing > self-signed certs into the system trust store is really a measure of > last resort. Correct! I merely wanted to state that OpenSSL can verify a self-signed certificate easily. The certificate 'just' have to be added to the SSLContext's store of trusted root certs. Somebody has to figure out how Python can accomplish the task. > There's another case which isn't solved by this, though, which is when a > cert is invalid. The common situation being that it has expired > (renewing certs is a PITA and therefore expired certs are more common > than it sounds they should be). In this case, there is no way to > whitelist it: you have to disable certificate checking altogether. This > can be exposed by the application as configuration option if necessary, > as well. It's possible to ignore errors with a verify callback. OpenSSL's wiki has an example for the expired certs http://wiki.openssl.org/index.php/Manual:X509_STORE_CTX_set_verify_cb%283%29#EXAMPLES Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 21:15, Antoine Pitrou wrote: > What do you call your local cert store? I was referring to Christian's comment > It's very simple to trust a self-signed certificate: just download it and > stuff it into the trust store. >From his recent response, I guess he meant the system store, and he agrees that this is a bad option. OK, that's fair, but: a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. b) I doubt my confusion over Christian's response is atypical. Based on what he said, if we hadn't had the subsequent discussion, I would probably have found a way to add a cert to "the store" without understanding the implications. While it's not Python's job to educate users, it would be a shame if its default behaviour led people to make ill-informed decisions. Maybe an SSL HOWTO would be a useful addition to the docs, if anyone feels motivated to write one. Regardless, thanks for the education! Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Hi, Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? Can you please elaborate? What do you mean by "get wrong"? Victor Le dimanche 31 août 2014, Marko Rauhamaa a écrit : > Victor Stinner >: > > > Proposition > > === > > > > If a system call fails with ``EINTR``, Python must call signal > > handlers: call ``PyErr_CheckSignals()``. If a signal handler raises > > an exception, the Python function fails with the exception. > > Otherwise, the system call is retried. If the system call takes a > > timeout parameter, the timeout is recomputed. > > Signals are tricky and easy to get wrong, to be sure, but I think it is > dangerous for Python to unconditionally commandeer signal handling. If > the proposition is accepted, there should be a way to opt out. > > > Marko > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner : > Sorry but I don't understand your remark. What is your problem with > retrying syscall on EINTR? The application will often want the EINTR return (exception) instead of having the function resume on its own. > Can you please elaborate? What do you mean by "get wrong"? Proper handling of signals is difficult and at times even impossible. For example it is impossible to wake up reliably from the select(2) system call when a signal is generated (which is why linux now has pselect). Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 08/31/2014 02:19 PM, Marko Rauhamaa wrote: Victor Stinner : Sorry but I don't understand your remark. What is your problem with retrying syscall on EINTR? The application will often want the EINTR return (exception) instead of having the function resume on its own. Examples? As an ignorant person in this area, I do not know why I would ever want to have EINTR raised instead just getting the results of, say, my read() call. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le dimanche 31 août 2014, Marko Rauhamaa a écrit : > Victor Stinner >: > > > Sorry but I don't understand your remark. What is your problem with > > retrying syscall on EINTR? > > The application will often want the EINTR return (exception) instead of > having the function resume on its own. This case is described as the use case #2 in the PEP, so it is supported. As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. For example the default signal handler for SIGINT raises KeyboardInterrupt. > > Can you please elaborate? What do you mean by "get wrong"? > > Proper handling of signals is difficult and at times even impossible. > For example it is impossible to wake up reliably from the select(2) > system call when a signal is generated (which is why linux now has > pselect). In my experience, using signal.set_wakeup_fd() works well with select(), even on Windows. The PEP promotes this. It is even thread safe. I don't know issues of signals with select() (and without a file descriptor used to wake up it). Python now exposes signal.pthread_sigmask(), I don't know if it helps. In my experience, signals don't play well with multithreading. On FreeBSD, the signal is send to a "random" thread. So you must have the same signal mask on all threads if you want to rely on them. But I don't get you point. How does this PEP make the situation worse? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 06:32, "Paul Moore" wrote: > > On 31 August 2014 21:15, Antoine Pitrou wrote: > > What do you call your local cert store? > > I was referring to Christian's comment > > It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. > > From his recent response, I guess he meant the system store, and he > agrees that this is a bad option. > > OK, that's fair, but: > > a) Is there really no OS-level personal trust store? I'm thinking of > Windows here for my own personal use, but the same question applies > elsewhere. > b) I doubt my confusion over Christian's response is atypical. Based > on what he said, if we hadn't had the subsequent discussion, I would > probably have found a way to add a cert to "the store" without > understanding the implications. While it's not Python's job to educate > users, it would be a shame if its default behaviour led people to make > ill-informed decisions. Right, this is why I came to the conclusion we need to follow the browser vendors lead here and support a per-user Python specific supplementary certificate cache before we can start validating certs by default at the *Python* level. There are still too many failure modes for cert management on private networks for us to safely ignore the use case of needing to force connections to services with invalid certs. We don't need to *solve* that problem here today - we can push it back to Alex (and anyone else interested) as a building block to investigate providing as part of cryptography.io or certi.fi, with a view to making a standard library version of that (along with any SSL module updates) part of PEP 476. In the meantime, we can update the security considerations for the ssl module to make it clearer that the defaults are set up for trusted networks and that using it safely on the public internet may mean you're better off with a third party library like requests or Twisted. (I'll start another thread shortly that is highly relevant to that topic) Regards, Nick. > > Maybe an SSL HOWTO would be a useful addition to the docs, if anyone > feels motivated to write one. > > Regardless, thanks for the education! > > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:09, Nick Coghlan wrote: > As Antoine says here, I'm also opposed to adding more Python specific > configuration options. However, I think there may be something > worthwhile we can do that's closer to the way browsers work, and has > the significant benefit of being implementable as a PyPI module first > (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Le 31/08/2014 23:41, Nick Coghlan a écrit : Right, this is why I came to the conclusion we need to follow the browser vendors lead here and support a per-user Python specific supplementary certificate cache before we can start validating certs by default at the *Python* level. There are still too many failure modes for cert management on private networks for us to safely ignore the use case of needing to force connections to services with invalid certs. We are not ignoring that use case. The proper solution is simply to disable cert validation in the application code (or, for more sophisticated needs, provide an application configuration setting for cert validation). > In the meantime, we can update the security considerations for the ssl > module to make it clearer that the defaults are set up for trusted networks > and that using it safely on the public internet may mean you're better off > with a third party library like requests or Twisted. No, you simply have to select the proper validation settings. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 maintenance release in addition to including it with Python 3.4. That part of the proposal proved to be controversial, so we dropped it from the original PEP in order to focus on meeting the Python 3.4 specific release deadlines. This also had the benefit of working out the kinks in the bootstrapping processing as part of the Python 3.4 release cycle. However, we still think we should start providing pip by default to Python 2.7 users as well, at least as part of the Windows and Mac OS X installers. One notable difference from PEP 453 is that because there is no venv module in 2.7, and hence no integration between venv and ensurepip, we can give redistributors the option of just disabling ensurepip entirely and redirecting users to platform specific installation tools. Regards, Nick. == PEP: 477 Title: Backport ensurepip (PEP 453) to Python 2.7 Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft Nick Coghlan Status: Active Type: Process Content-Type: text/x-rst Created: 26-Aug-2014 Post-History: 1-Sep-2014 Abstract This PEP proposes that the ``ensurepip`` module, added to Python 3.4 by PEP 453, be backported to Python 2.7. It also proposes that automatic invocation of ``ensurepip`` be added to the Python 2.7 Windows and OSX installers. However it does **not** propose that automatic invocation be added to the ``Makefile``. It also proposes that the documentation changes for the package distribution and installation guides be updated to match that in 3.4, which references using the ``ensurepip`` module to bootstrap the installer. Rationale = Python 2.7 is effectively a LTS release of Python which represents the end of the 2.x series and there is still a very large contingent of users whom are still using Python 2.7 as their primary version. These users, in order to participate in the wider Python ecosystem, must manually attempt to go out and find the correct way to bootstrap the packaging tools. It is the opinion of this PEP that making it as easy as possible for end users to participate in the wider Python ecosystem is important for 3 primary reasons: 1. The Python 2.x to 3.x migration has a number of painpoints that are eased by a number of third party modules such as six [#six]_, modernize [#modernize]_, or future [#future]_. However relying on these tools requires that everyone who uses the project have a tool to install these packages. 2. In addition to tooling to aid in migration from Python 2.x to 3.x, there are also a number of modules that are *new* in Python 3 for which there are backports available on PyPI. This can also aid in the ability for people to write 2.x and 3.x compatible software as well as enable them to use some of the newer features of Python 3 on Python 2. 3. Users also will need a number of tools in order to create python packages that conform to the newer standards that are being proposed. Things like setuptools [#setuptools]_, Wheel [#wheel]_, and twine [#twine]_ are enabling a safer, faster, and more reliable packaging tool chain. These tools can be difficult for people to use if first they must be told how to go out and install the package manager. 4. One of Pythons biggest strengths is in the huge ecosystem of libraries and projects that have been built on top of it, most of which are distributed through PyPI. However in order to benefit from this wide ecosystem meaningfully requires end users, some of which are going to be new, to make a decision on which package manager they should get, how to get it, and then finally actually installing it first. Furthermore, alternative implementations of Python are recognizing the benefits of PEP 453 and both PyPy and Jython have plans to backport ensurepip to their 2.7 runtimes. Automatic Invocation PEP 453 has ``ensurepip`` automatically invoked by default in the ``Makefile`` and the Windows and OSX Installers. This allowed it to ensure that, by default, all users would get Python with pip already installed. This PEP however believes that while this is fine for the Python 2.7 Windows and Mac OS X installers it is *not* ok for the Python 2.7 ``Makefile`` in general. The primary consumers of the ``Makefile`` are downstream package managers which distribute Python themselves. These downstream distributors typically do not want pip to be installed via ``ensurepip`` and would prefer that end users install it with their own package manager. Not invoking ``ensurepip`` automatically from the ``Makefile`` would allow these distributors to simply ignore the fact that ``ensurepip`` has been backported and still not end up with pip installed via it. The primary consumers of the OSX and Windows installers are end users who are attempting to install Python on their own machine. There is not a package manager available where these u
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 22:30, Paul Moore wrote: > On 31 August 2014 21:15, Antoine Pitrou wrote: >> What do you call your local cert store? > > I was referring to Christian's comment >> It's very simple to trust a self-signed certificate: just download it and >> stuff it into the trust store. I was referring to the the trust store of the SSLContext object and not to any kind of cert store of the operating system. Sorry for the confusion. > a) Is there really no OS-level personal trust store? I'm thinking of > Windows here for my own personal use, but the same question applies > elsewhere. Windows and OSX have superior cert stores compared to Linux and BSD. They have means for user and system wide cert stores and trust settings Linux just have one central directory or file with all trusted certs. My KDE has some options to disable certs but I don't know how to make use of the configuration. Even worse: Linux distros doesn't make a different between purposes. On Windows a user can trust a certificate for S/MIME but not for server auth or client auth. Ubuntu just puts all certification in one directory but it's wrong. :( https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1207004 Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner : > But I don't get you point. How does this PEP make the situation worse? Did I say it would? I just wanted to make sure the system call resumption doesn't become mandatory. Haven't thought through what the exception raising technique would entail. It might be perfectly ok apart from being a change to the signal handler API. > I don't know issues of signals with select() (and without a file > descriptor used to wake up it). A signal handler often sets a flag, which is inspected when select() returns. The problem is when a signal arrives between testing the flag and calling select(). The pselect() system call allows you to block signals and have the system call unblock them correctly to avoid the race. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 07:43, "Christian Heimes" wrote: > > On 31.08.2014 08:09, Nick Coghlan wrote: > > As Antoine says here, I'm also opposed to adding more Python specific > > configuration options. However, I think there may be something > > worthwhile we can do that's closer to the way browsers work, and has > > the significant benefit of being implementable as a PyPI module first > > (more on that in a separate reply). > > I'm on your and Antoine's side and strictly against any additional > environment variables or command line arguments. That would make the > whole validation process even more complex and harder to understand. > > There might be a better option to give people and companies the option > to tune the SSL module to their needs. Python already have a > customization hook for the site module called sitecustomize. How about > another module named sslcustomize? Such a module could be used to tune > the ssl module to the needs of users, e.g. configure a different default > context, add certificates to a default context etc. > > Companies could install them in a system global directory on their > servers. Users could put them in their own user site directory and even > each virtual env can have one sslcustomize of its own. It's fully > backward compatible, doesn't add any flags and developers have the full > power of Python for configuration and customization. And means a user specific store (if one became available) could be configured there. Yes, I think this would address my concerns, especially if combined with a clear recipe in the documentation on how to optionally disable cert validation at the application layer. Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration. Cheers, Nick. > > Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Ethan Furman : > On 08/31/2014 02:19 PM, Marko Rauhamaa wrote: >> The application will often want the EINTR return (exception) instead >> of having the function resume on its own. > > Examples? > > As an ignorant person in this area, I do not know why I would ever > want to have EINTR raised instead just getting the results of, say, my > read() call. Say you are writing data into a file and it takes a long time (because there is a lot of data or the medium is very slow or there is a hardware problem). You might have designed in a signaling scheme to address just this possibility. Then, the system call had better come out right away without trying to complete the full extent of the call. If a signal is received when read() or write() has completed its task partially (> 0 bytes), no EINTR is returned but the partial count. Obviously, Python should take that possibility into account so that raising an exception in the signal handler (as mandated by the PEP) doesn't cause the partial result to be lost on os.read() or os.write(). Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
> On Aug 31, 2014, at 5:43 PM, Christian Heimes wrote: > > On 31.08.2014 08:09, Nick Coghlan wrote: >> As Antoine says here, I'm also opposed to adding more Python specific >> configuration options. However, I think there may be something >> worthwhile we can do that's closer to the way browsers work, and has >> the significant benefit of being implementable as a PyPI module first >> (more on that in a separate reply). > > I'm on your and Antoine's side and strictly against any additional > environment variables or command line arguments. That would make the > whole validation process even more complex and harder to understand. > > There might be a better option to give people and companies the option > to tune the SSL module to their needs. Python already have a > customization hook for the site module called sitecustomize. How about > another module named sslcustomize? Such a module could be used to tune > the ssl module to the needs of users, e.g. configure a different default > context, add certificates to a default context etc. > > Companies could install them in a system global directory on their > servers. Users could put them in their own user site directory and even > each virtual env can have one sslcustomize of its own. It's fully > backward compatible, doesn't add any flags and developers have the full > power of Python for configuration and customization. > > This may be a dumb question, but why can’t sitecustomize do this already? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:04, "Marko Rauhamaa" a écrit : > > Victor Stinner : > > > But I don't get you point. How does this PEP make the situation worse? > > Did I say it would? I just wanted to make sure the system call > resumption doesn't become mandatory. The syscall is only retried on EINTR if the signal handler didn't raise an exception. So it is not always retried: "Proposition If a system call fails with ``EINTR``, Python must call signalhandlers: call ``PyErr_CheckSignals()``. If a signal handler raisesan exception, the Python function fails with the exception.Otherwise, the system call is retried. If the system call takes atimeout parameter, the timeout is recomputed." > Haven't thought through what the exception raising technique would > entail. It might be perfectly ok apart from being a change to the signal > handler API. I don't think that it is safe to expect an InterruptedError if the signal handler doesn't raise an exception. Many Python module already retry the syscall on EINTR. So I'm not sure that the PEP is really a major change. It's just to make Python more homogeneous, always have the same reliable and portable behaviour. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Mon, 01 Sep 2014 01:15:12 +0300 Marko Rauhamaa wrote: > > If a signal is received when read() or write() has completed its task > partially (> 0 bytes), no EINTR is returned but the partial count. > Obviously, Python should take that possibility into account so that > raising an exception in the signal handler (as mandated by the PEP) > doesn't cause the partial result to be lost on os.read() or os.write(). If the signal handler is called, the exception *will* be raised. There's no guarantee at which point in the Python code it will be raised (it's implementation-dependent), but it's near impossible to protect regular Python code against such asynchronous exceptions. Which is why you should switch to a wakeup fd scheme as mentioned by Victor, if you want to rely on signals at all. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On 8/31/2014 6:00 PM, Nick Coghlan wrote: Earlier versions of PEP 453 proposed bootstrapping pip into a Python 2.7 maintenance release in addition to including it with Python 3.4. That part of the proposal proved to be controversial, so we dropped it from the original PEP in order to focus on meeting the Python 3.4 specific release deadlines. This also had the benefit of working out the kinks in the bootstrapping processing as part of the Python 3.4 release cycle. However, we still think we should start providing pip by default to Python 2.7 users as well, at least as part of the Windows and Mac OS X installers. Having used pip for 3.4 on two windows computers, the only reason I see to *not* install it with Python would be 'push' people to 3.4+ by keeping 2.7 harder to use. > One notable difference from PEP 453 is that because there is no venv > module in 2.7, and hence no integration between venv and ensurepip, > we can give redistributors the option of just disabling ensurepip > entirely and redirecting users to platform specific installation > tools. This also suggests that ensurepip does not have to be left on a Win/OS install either, after installation is done. Perhaps after ensuring that pip is installed, ensurepip could remove itself. This would remove any objection that its presence in /lib constitutes a new feature (which it is, even if not very useful). It would also keep /lib uniform across platforms. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Le 1 sept. 2014 00:17, "Marko Rauhamaa" a écrit : > If a signal is received when read() or write() has completed its task > partially (> 0 bytes), no EINTR is returned but the partial count. > Obviously, Python should take that possibility into account so that > raising an exception in the signal handler (as mandated by the PEP) > doesn't cause the partial result to be lost on os.read() or os.write(). This case is unrelated to the PEP, the PEP only changes the behaviour when a syscall fails with EINTR. (When Python gets a signal, the C signal handler is immediatly called. The handler sets a flag which is cheched before executing an instruction. The Python signal handler can be called between two Python instructions. In some cases, it may be called earlier in functions checking manually the flag. IMO the exact behaviour is undefined. Python tries to call the Python signal handler as soon as possible, with a low performance overhead.) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On Mon, 1 Sep 2014 08:00:14 +1000 Nick Coghlan wrote: > > That part of the proposal proved to be controversial, so we dropped it from > the original PEP in order to focus on meeting the Python 3.4 specific > release deadlines. This also had the benefit of working out the kinks in > the bootstrapping processing as part of the Python 3.4 release cycle. > > However, we still think we should start providing pip by default to Python > 2.7 users as well, at least as part of the Windows and Mac OS X installers. I don't agree with this. pip is simply not part of the 2.7 feature set. If you add pip to a bugfix version, then you have bugfix versions which are more featureful than others, which makes things more complicated to explain. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > On Mon, 1 Sep 2014 08:00:14 +1000 > Nick Coghlan wrote: > > > > That part of the proposal proved to be controversial, so we dropped it from > > the original PEP in order to focus on meeting the Python 3.4 specific > > release deadlines. This also had the benefit of working out the kinks in > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > However, we still think we should start providing pip by default to Python > > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > > I don't agree with this. pip is simply not part of the 2.7 feature set. > If you add pip to a bugfix version, then you have bugfix versions which > are more featureful than others, which makes things more complicated to > explain. 2.7.x has been and will be alive for so long that will already have to explain that sort thing; i.e. PEP 466 and why different bugfix releases support different versions of dependency libraries. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
On 1 Sep 2014 09:23, "Benjamin Peterson" wrote: > > On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > > On Mon, 1 Sep 2014 08:00:14 +1000 > > Nick Coghlan wrote: > > > > > > That part of the proposal proved to be controversial, so we dropped it from > > > the original PEP in order to focus on meeting the Python 3.4 specific > > > release deadlines. This also had the benefit of working out the kinks in > > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > > > However, we still think we should start providing pip by default to Python > > > 2.7 users as well, at least as part of the Windows and Mac OS X installers. > > > > I don't agree with this. pip is simply not part of the 2.7 feature set. > > If you add pip to a bugfix version, then you have bugfix versions which > > are more featureful than others, which makes things more complicated to > > explain. > > 2.7.x has been and will be alive for so long that will already have to > explain that sort thing; i.e. PEP 466 and why different bugfix releases > support different versions of dependency libraries. Exactly. LTS is genuinely different from stopping maintenance after the next feature release - it requires considering the "stability risk" and "user experience improvement" questions separately. In this case, the problem is that the Python 2 platform *is* still evolving, but the centre of that evolution has moved to PyPI. For "standard library only" users, Python 2 stopped evolving back in 2010. For PyPI users, by contrast, it's still evolving at a rapid pace. For our Python 3 transition story to be coherent, we need to ensure tools like six, modernize and future are readily available, while still remaining free to evolve independently of the standard library. That means providing a package management utility as easily and as painlessly as possible. Embracing pip upstream for Python 2 as well as Python 3 also sends a powerful signal to redistributors that says "your users are going to need this" and makes them do additional work to *avoid* providing it. Some of them *will* choose that path, but that becomes a matter for discussion between them and their user base, rather than a problem we need to worry about upstream. Cheers, Nick. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 Sep 2014 08:15, "Donald Stufft" wrote: > > >> On Aug 31, 2014, at 5:43 PM, Christian Heimes wrote: >> >> Companies could install them in a system global directory on their >> servers. Users could put them in their own user site directory and even >> each virtual env can have one sslcustomize of its own. It's fully >> backward compatible, doesn't add any flags and developers have the full >> power of Python for configuration and customization. > > This may be a dumb question, but why can’t sitecustomize do this already? It can. The advantage of a separate file is that it won't conflict with existing sitecustomize modules, so (for example) redistributors can add a default sslcustomize, and you can add one to your virtual environments that are integrated with the system Python environment without needing to worry about whether or not there's a global sitecustomize (you'd only have trouble if there was a global sslcustomize). Cheers, Nick. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
Victor Stinner wrote: As written in the PEP, if you want to be notified of the signal, set a signal handler which raises an exception. I'm not convinced that this covers all possible use cases. It might be all right if you have control over the signal handler, but what if you don't? I think it's best if the functions in the os module remain thin wrappers that expose the OS functionality as fully and directly as possible. Anything else should be provided by higher-level facilities. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan wrote: > On 1 Sep 2014 07:43, "Christian Heimes" wrote: > > > > On 31.08.2014 08:09, Nick Coghlan wrote: > > > As Antoine says here, I'm also opposed to adding more Python specific > > > configuration options. However, I think there may be something > > > worthwhile we can do that's closer to the way browsers work, and has > > > the significant benefit of being implementable as a PyPI module first > > > (more on that in a separate reply). > > > > I'm on your and Antoine's side and strictly against any additional > > environment variables or command line arguments. That would make the > > whole validation process even more complex and harder to understand. > > > > There might be a better option to give people and companies the option > > to tune the SSL module to their needs. Python already have a > > customization hook for the site module called sitecustomize. How about > > another module named sslcustomize? Such a module could be used to tune > > the ssl module to the needs of users, e.g. configure a different default > > context, add certificates to a default context etc. > > > > Companies could install them in a system global directory on their > > servers. Users could put them in their own user site directory and even > > each virtual env can have one sslcustomize of its own. It's fully > > backward compatible, doesn't add any flags and developers have the full > > power of Python for configuration and customization. > > And means a user specific store (if one became available) could be > configured there. > > Yes, I think this would address my concerns, especially if combined with a > clear recipe in the documentation on how to optionally disable cert > validation at the application layer. > > Assuming sslcustomize was in site-packages rather than the standard library > directories, you would also be able to use virtual environments with an > appropriate sslcustomize module to disable cert checking even if the > application you were running didn't support direct configuration. It sounds like this would address my concerns as well (I don't really care *how* it is implemented as long as I don't have to touch the code of a third party application when I upgrade my python version to 3.5...remember, the context here is backward compatibility concerns). Does it address the issue of accepting an invalid cert, though? --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing wrote: > Victor Stinner wrote: >> >> As written in the PEP, if you want to be notified of the signal, set a >> signal handler which raises an exception. > > I'm not convinced that this covers all possible use cases. > It might be all right if you have control over the signal > handler, but what if you don't? > > I think it's best if the functions in the os module remain > thin wrappers that expose the OS functionality as fully and > directly as possible. Anything else should be provided > by higher-level facilities. I'm inclined to agree about keeping the os module thin. If we were to recreate Python today, from scratch, it might make sense to hide this by default, but now there's almost certainly code out there that depends on the current behavior. But I also agree that it's hard to pin down which higher level Python library calls are going to be using system calls. My http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem with window resizing for a while (SIGWINCH), and although I use it pretty much daily now without problems, I'm still not sure I got 100% of the possibilities covered. Fortunately, wrapping a system call can be as simple as: def retry_on_eintr(function, *args, **kw): ''' Retry a system call until one of the following happens: 1) It succeeds 2) It errors with something other than EINTR ''' while True: try: return function(*args, **kw) except OSError: nothing, extra, nothing2 = sys.exc_info() dummy = nothing dummy = nothing2 if extra.errno == errno.EINTR: continue else: raise Note that os.read() and os.write() need different handling. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On Sun, 31 Aug 2014 20:14:50 -0700, Dan Stromberg wrote: > On Sun, Aug 31, 2014 at 3:28 PM, Greg Ewing > wrote: > > Victor Stinner wrote: > >> > >> As written in the PEP, if you want to be notified of the signal, set a > >> signal handler which raises an exception. > > > > I'm not convinced that this covers all possible use cases. > > It might be all right if you have control over the signal > > handler, but what if you don't? > > > > I think it's best if the functions in the os module remain > > thin wrappers that expose the OS functionality as fully and > > directly as possible. Anything else should be provided > > by higher-level facilities. > > I'm inclined to agree about keeping the os module thin. If we were to > recreate Python today, from scratch, it might make sense to hide this > by default, but now there's almost certainly code out there that > depends on the current behavior. > > But I also agree that it's hard to pin down which higher level Python > library calls are going to be using system calls. My > http://stromberg.dnsalias.org/~strombrg/pypty/ program had a problem > with window resizing for a while (SIGWINCH), and although I use it > pretty much daily now without problems, I'm still not sure I got 100% > of the possibilities covered. > > Fortunately, wrapping a system call can be as simple as: > > def retry_on_eintr(function, *args, **kw): > ''' > Retry a system call until one of the following happens: > 1) It succeeds > 2) It errors with something other than EINTR > ''' > > while True: > try: > return function(*args, **kw) > except OSError: > nothing, extra, nothing2 = sys.exc_info() > dummy = nothing > dummy = nothing2 > if extra.errno == errno.EINTR: > continue > else: > raise > > Note that os.read() and os.write() need different handling. Personally, I really want Python to handle EINTR for me. And indeed, that has been what we have been doing for a while now, piece by piece, bug by bug. Victor just wants to systematize and document that, and I think that's a good idea. We've been consistently treating lack of handling of EINTR as a bug. If there are *real* cases where that causes a backward compatibility problem, then we need to know. But so far, we have gotten zero complaints about the cases that we have fixed. --David PS: I recently switched from using selectors to using a timeout on a socket because in that particular application I could, and because reading a socket with a timeout handles EINTR (in recent python versions), whereas reading a non-blocking socket doesn't. Under the hood, a socket with a timeout is a non-blocking socket. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
"R. David Murray" : > PS: I recently switched from using selectors to using a timeout on a > socket because in that particular application I could, and because > reading a socket with a timeout handles EINTR (in recent python > versions), whereas reading a non-blocking socket doesn't. Under the > hood, a socket with a timeout is a non-blocking socket. Under what circumstances would a nonblocking socket generate an EINTR? I believe the biggest EINTR problem child is file I/O, which is always blocking in linux. Marko ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 September 2014 11:10, R. David Murray wrote:
>
> It sounds like this would address my concerns as well (I don't really
> care *how* it is implemented as long as I don't have to touch the
> code of a third party application when I upgrade my python version to
> 3.5...remember, the context here is backward compatibility concerns).
>
> Does it address the issue of accepting an invalid cert, though?
That's actually an interesting question, as the PEP doesn't currently
propose adding any new global configuration knobs to the ssl or
httplib modules - it just proposes switching httplib from the legacy
(but fully backwards compatible) ssl._create_stdlib_context() API to
the newer (but potentially backwards incompatible in some
environments) ssl.create_default_context() API.
Having the ssl module import an sslcustomize module at the end
wouldn't be enough unless the appropriate APIs were put in place to
allow things to be configured at a process global level.
One possible way to do that would be to provide a central context
factory mapping that provide a module specific SSL context creator.
We'd seed it appropriately for the stdlib modules where we wanted to
use the legacy context definition, but it would default to using
ssl.create_default_context.
Under that kind of model, the first change we would actually make is
to make ssl._create_stdlib_context() public under a suitable name,
let's say ssl.create_legacy_context()
Independenting of any other changes, exposing
ssl.create_legacy_context() like that would also make it
straightforward for folks to opt in to the old behaviour as an interim
hack in a way that is easy to grep for and fix later (it's also
something a linter can easily disallow).
The second change would be to provide a mapping from arbitrary names
to context factories in the ssl module that defaults to
ssl.create_default_context:
named_contexts = defaultdict((lambda name: create_default_context))
(A more accurate name would be "named_context_factory", but I think
"named_contexts" reads better. Folks will learn quickly enough that it
actually stores context factories rather than prebuilt context
objects)
The third change would be to replace all calls to
"ssl._create_stdlib_context()" with calls to
"ssl.named_contexts[__name__]()" instead.
The final change would be to seed the context factory map
appropriately for the standard library modules where we wanted to keep
the *old* default:
for modname in ("nntplib", "poplib", "imaplib", "ftplib",
"smtplib", "asyncio.selector_events", "urllib.request",
"http.client"):
named_contexts[modname] = create_legacy_context
The list I have above is for *all* current uses of
"sss._create_stdlib_context". The backwards incompatible part of PEP
476 would then just be about removing names from that list (currently
just "http.client", but I'd suggest "asyncio.selector_events" as
another candidate, taking advantage of asyncio's provisional API
status).
The "revert to 3.4 behaviour" content for sslcustomize.py would then just be:
import ssl
ssl.named_contexts["http.client"] = ssl.create_legacy_context
However, someone that wanted to also enforce SSL properly for other
standard library modules could go the other way:
import ssl
for modname in ("nntplib", "poplib", "imaplib", "ftplib",
"smtplib", "urllib.request"):
ssl.named_contexts[modname] = ssl.create_default_context
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31 August 2014 23:10, Nick Coghlan wrote: > Assuming sslcustomize was in site-packages rather than the standard library > directories, you would also be able to use virtual environments with an > appropriate sslcustomize module to disable cert checking even if the > application you were running didn't support direct configuration. Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 475, Retry system calls failing with EINTR
On 31 August 2014 22:38, Victor Stinner wrote: > This case is described as the use case #2 in the PEP, so it is supported. As > written in the PEP, if you want to be notified of the signal, set a signal > handler which raises an exception. For example the default signal handler > for SIGINT raises KeyboardInterrupt. Wait - sigint? Does this mean that (unless the application adds a signal handler) keyboard interrupts will be in effect ignored while in a system call? I'm not sure I like that - I'd rather Ctrl-C always interrupted the program. Specifically, in one-off scripts that *don't* take care to handle all errors appropriately and just show the traceback... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 477: selected ensurepip backports for Python 2.7
In article , Nick Coghlan wrote: > On 1 Sep 2014 09:23, "Benjamin Peterson" wrote: > > On Sun, Aug 31, 2014, at 16:17, Antoine Pitrou wrote: > > > On Mon, 1 Sep 2014 08:00:14 +1000 > > > Nick Coghlan wrote: > > > > > > > > That part of the proposal proved to be controversial, so we dropped > it from > > > > the original PEP in order to focus on meeting the Python 3.4 specific > > > > release deadlines. This also had the benefit of working out the kinks > in > > > > the bootstrapping processing as part of the Python 3.4 release cycle. > > > > > > > > However, we still think we should start providing pip by default to > Python > > > > 2.7 users as well, at least as part of the Windows and Mac OS X > installers. A much bigger than average +1 > > > I don't agree with this. pip is simply not part of the 2.7 feature set. > > > If you add pip to a bugfix version, then you have bugfix versions which > > > are more featureful than others, which makes things more complicated to > > > explain. > > 2.7.x has been and will be alive for so long that will already have to > > explain that sort thing; i.e. PEP 466 and why different bugfix releases > > support different versions of dependency libraries. And that is a minor complication compared with the confusion and difficulty of trying to explain to users (stuck with 2.7 for the time being) of how to install third-party packages on each platform (especially Windows) versus the simplicity of the 3.4.x story, thanks to ensurepip. Having pip available as a documented, batteries-included tool for all current releases would be a huge usability improvement. > Exactly. LTS is genuinely different from stopping maintenance after the > next feature release - it requires considering the "stability risk" and > "user experience improvement" questions separately. > > In this case, the problem is that the Python 2 platform *is* still > evolving, but the centre of that evolution has moved to PyPI. For "standard > library only" users, Python 2 stopped evolving back in 2010. For PyPI > users, by contrast, it's still evolving at a rapid pace. > > For our Python 3 transition story to be coherent, we need to ensure tools > like six, modernize and future are readily available, while still remaining > free to evolve independently of the standard library. That means providing > a package management utility as easily and as painlessly as possible. > > Embracing pip upstream for Python 2 as well as Python 3 also sends a > powerful signal to redistributors that says "your users are going to need > this" and makes them do additional work to *avoid* providing it. Some of > them *will* choose that path, but that becomes a matter for discussion > between them and their user base, rather than a problem we need to worry > about upstream. FTR, I'm willing to backport the pieces I did for 3.4 and I could do the ensurepip backport, as well. I'll leave the Windows installer changes for someone else, though. -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 1 September 2014 16:07, Paul Moore wrote: > On 31 August 2014 23:10, Nick Coghlan wrote: >> Assuming sslcustomize was in site-packages rather than the standard library >> directories, you would also be able to use virtual environments with an >> appropriate sslcustomize module to disable cert checking even if the >> application you were running didn't support direct configuration. > > Would this mean that a malicious package could install a custom > sslcustomize.py and so add unwanted certs to the system? I guess we > have to assume that installed packages are trusted, but I just wanted > to be explicit. Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The "-S" and "-I" switches would need to disable the implied "sslcustomize", just as they disable "import site". Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
