Re: [Python-Dev] surrogatepass - she's a witch, burn 'er!
Greg Ewing writes: > M.-A. Lemburg wrote: > > we needed > > a way to make sure that Python 3 also optionally supports working > > with lone surrogates in such UTF-8 streams (nowadays called CESU-8: > > http://en.wikipedia.org/wiki/CESU-8). Besides what Greg says, CESU-8 is an UTF, and therefore encodes valid Unicode. Speaking imprecisely, CESU-8 is UTF-16 with variable-width code units (ie, each 16-bit code point is represented using the UTF-8 variable-width representation).[1] I think you are thinking of Markus Kuhn's utf-8b (which I believe is exactly what is implemented by the surrogateescape handler). As far as the goal of "working with lone surrogates in such UTF-8 streams", the surrogateescape handler already permits that, and does so consistently across streams in the sense that lone surrogates in the UTF-8 stream cannot be mixed with garbage bytes decoded by surrogateescape in another stream, which produces an unencodable mess. I still don't see a justification for the surrogatepass handler. What applications are producing (not merely passing through) UTF-8-encoded surrogates these days? Footnotes: [1] For the curious, it's imprecise because in Unicode code units are fixed-width by definition. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Thanks for the rapid feedback everyone! I want to summarize the action items and discussion points that have come up so far: To add to the PEP: * Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root Discussion points: * Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside. Cheers, Alex ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]
M.-A. Lemburg wrote: we needed a way to make sure that Python 3 also optionally supports working with lone surrogates in such UTF-8 streams (nowadays called CESU-8: http://en.wikipedia.org/wiki/CESU-8). I don't think CESU-8 is the same thing. According to the wiki page, CESU-8 *requires* all code points above 0x to be split into surrogate pairs before encoding. It also doesn't say that lone surrogates are valid -- it doesn't mention lone surrogates at all, only pairs. Neither does the linked technical report. The technical report also says that CESU-8 forbids any UTF-8 sequences of more than three bytes, so it's definitely not "UTF-8 plus lone surrogates". -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Fri, 29 Aug 2014 18:00:50 -0400, Donald Stufft wrote: > > On Aug 29, 2014, at 5:42 PM, R. David Murray wrote: > > Especially if you want an accelerated change, there must be a way to > > *easily* get back to the previous behavior, or we are going to catch a > > lot of flack. There may be only 7% of public certs that are problematic, > > but I'd be willing to bet you that there are more not-really-public ones > > that are critical to day to day operations *somewhere* :) > > > > wget and curl have 'ignore validation' as a command line flag for a reason. > > > > Right, thatâs why Iâm on the fence :) > > On one hand, itâs going to break things for some people, (arguably they are > already broken, just silently so, but weâll leave that argument aside) and a > way to get back the old behavior is good. There are already ways within > the Python code itself, so thatâs covered. From outside of the Python code > there are ways if the certificate is untrusted but otherwise valid which are > pretty easy to do. The major âgapâ is when you have an actual invalid > certificate due to expiration or hostname or some other such thing. > > On the other hand Python is not wget/curl and the people who are most > likely to be the target for a âI canât change the code but I need to get > the > old behavior backâ are people who are likely to not be invoking Python > itself but using something written in Python which happens to be using > Python. IOW they might be executing âfoobarâ not âpython -m foobarâ. Right, so an environment variable is better than a command line switch, for Python. > Like I said though, Iâm personally fine either way so donât take this as > being against that particular change! Ack. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Fri, 29 Aug 2014 18:08:19 -0400 Donald Stufft wrote: > > > > Are you sure that's possible ? Python doesn't load the > > openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env > > vars only work for the openssl command line binary, AFAIK. > > I’m not 100% sure on that. I know they are not limited to the command > line binary as ruby uses those environment variables in the way I > described above. SSL_CERT_DIR and SSL_CERT_FILE are used, if set, when SSLContext.load_verify_locations() is called. Actually, come to think of it, this allows us to write a better test for that method. Patch welcome! Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
> On Aug 29, 2014, at 5:58 PM, M.-A. Lemburg wrote: > > On 29.08.2014 23:11, Donald Stufft wrote: >> >> Sorry I was on my phone and didn’t get to fully reply to this. >> >>> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg wrote: >>> >>> On 29.08.2014 21:47, Alex Gaynor wrote: Hi all, I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think. PEP text follows. >>> >>> Thanks for the PEP. I think this is generally a good idea, >>> but some important parts are missing from the PEP: >>> >>> * transition plan: >>> >>> I think starting with warnings in Python 3.5 and going >>> for exceptions in 3.6 would make a good transition >>> >>> Going straight for exceptions in 3.5 is not in line with >>> our normal procedures for backwards incompatible changes. >> >> As far as a transition plan, I think that this is an important >> enough thing to have an accelerated process. If we need >> to provide a warning than let’s add it to the next 3.4 otherwise >> it’s going to be 2.5+ years until we stop being unsafe by >> default. > > Fine with me; we're still early in the Python 3.4 > patch level releases. > >> Another problem with this is that I don’t think it’s actually >> possible to do. Python itself isn’t validating the TLS certificates, >> OpenSSL is doing that. To my knowledge OpenSSL doesn’t >> have a way to say “please validate these certificates and if >> they don’t validate go ahead and keep going and just let me >> get a warning from it”. It’s a 3 way switch, no validation, validation >> if a certificate is provided, and validation always. >> >> Now that’s strictly for the “verify the certificate chain” portion, >> the hostname verification is done entirely on our end and we >> could do something there… but I’m not sure it makes sense >> to do so if we can’t do it for invalid certificates too. > > OpenSSL provides a callback for certificate validation, > so it is possible to issue a warning and continue with > accepting the certificate. Ah right, I forgot about that. I was thinking in terms of CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED. I think it’s fine to add a warning if possible to Python 3.4, I just couldn’t think off the top of my head a way of doing it. > >>> * configuration: >>> >>> It would be good to be able to switch this on or off >>> without having to change the code, e.g. via a command >>> line switch and environment variable; perhaps even >>> controlling whether or not to raise an exception or >>> warning. >> >> I’m on the fence about this, if someone provides a certificate >> that we can validate against (which can be done without >> touching the code) then the only thing that really can’t be >> “fixed” without touching the code is if someone has a certificate >> that is otherwise invalid (expired, not yet valid, wrong hostname, >> etc). I’d say if I was voting on this particular thing I’d be -0, I’d >> rather it didn’t exist but I wouldn’t cry too much if it did. > > If you're testing code or trying out some new stuff, you > don't want to get a valid cert first, but instead go ahead > with a self signed one. That's the use case. > >>> * choice of trusted certificate: >>> >>> Instead of hard wiring using the system CA roots into >>> Python it would be good to just make this default and >>> permit the user to point Python to a different set of >>> CA roots. >>> >>> This would enable using self signed certs more easily. >>> Since these are often used for tests, demos and education, >>> I think it's important to allow having more control of >>> the trusted certs. >> >> >> Like my other email said, the Python API has everything needed >> to easily specify your own CA roots and/or disable the validations. >> The OpenSSL library also allows you to specify either a directory >> or a file to change the root certificates without code changes. The >> only real problems with the APIs are that the default is bad and >> an unrelated thing where you can’t pass in an in memory certificate. > > Are you sure that's possible ? Python doesn't load the > openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env > vars only work for the openssl command line binary, AFAIK. I’m not 100% sure on that. I know they are not limited to the command line binary as ruby uses those environment variables in the way I described above. I do not believe that Ruby has done anything special to enable the use of those variables. It’s possible we’re doing something differently that bypasses those variables though. If that is the case then yes let’s add it, ideally doing whatever it needs to be to make OpenSSL respect those variables, or else respecting them ourselves. > > In any case, Python will have to tap into the OS CA root > provider using special code and this code could easily be > made to check other dirs/files as well. > > The point is that it should be possi
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
> On Aug 29, 2014, at 5:42 PM, R. David Murray wrote: > > On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft wrote: >> Sorry I was on my phone and didn’t get to fully reply to this. >>> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg wrote: >>> >>> * configuration: >>> >>> It would be good to be able to switch this on or off >>> without having to change the code, e.g. via a command >>> line switch and environment variable; perhaps even >>> controlling whether or not to raise an exception or >>> warning. >> >> I’m on the fence about this, if someone provides a certificate >> that we can validate against (which can be done without >> touching the code) then the only thing that really can’t be >> “fixed” without touching the code is if someone has a certificate >> that is otherwise invalid (expired, not yet valid, wrong hostname, >> etc). I’d say if I was voting on this particular thing I’d be -0, I’d >> rather it didn’t exist but I wouldn’t cry too much if it did. > > Especially if you want an accelerated change, there must be a way to > *easily* get back to the previous behavior, or we are going to catch a > lot of flack. There may be only 7% of public certs that are problematic, > but I'd be willing to bet you that there are more not-really-public ones > that are critical to day to day operations *somewhere* :) > > wget and curl have 'ignore validation' as a command line flag for a reason. > Right, that’s why I’m on the fence :) On one hand, it’s going to break things for some people, (arguably they are already broken, just silently so, but we’ll leave that argument aside) and a way to get back the old behavior is good. There are already ways within the Python code itself, so that’s covered. From outside of the Python code there are ways if the certificate is untrusted but otherwise valid which are pretty easy to do. The major “gap” is when you have an actual invalid certificate due to expiration or hostname or some other such thing. On the other hand Python is not wget/curl and the people who are most likely to be the target for a “I can’t change the code but I need to get the old behavior back” are people who are likely to not be invoking Python itself but using something written in Python which happens to be using Python. IOW they might be executing “foobar” not “python -m foobar”. Like I said though, I’m personally fine either way so don’t take this as being against that particular change! --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Fri, 29 Aug 2014 17:42:34 -0400 "R. David Murray" wrote: > > Especially if you want an accelerated change, there must be a way to > *easily* get back to the previous behavior, or we are going to catch a > lot of flack. There may be only 7% of public certs that are problematic, > but I'd be willing to bet you that there are more not-really-public ones > that are critical to day to day operations *somewhere* :) Actually, by construction, there are certs which will always fail verification, for example because they are embedded in telco equipments which don't have a predefined hostname or IP address. (I have encountered some of those) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 29.08.2014 23:11, Donald Stufft wrote: > > Sorry I was on my phone and didn’t get to fully reply to this. > >> On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg wrote: >> >> On 29.08.2014 21:47, Alex Gaynor wrote: >>> Hi all, >>> >>> I've just submitted PEP 476, on enabling certificate validation by default >>> for >>> HTTPS clients in Python. Please have a look and let me know what you think. >>> >>> PEP text follows. >> >> Thanks for the PEP. I think this is generally a good idea, >> but some important parts are missing from the PEP: >> >> * transition plan: >> >> I think starting with warnings in Python 3.5 and going >> for exceptions in 3.6 would make a good transition >> >> Going straight for exceptions in 3.5 is not in line with >> our normal procedures for backwards incompatible changes. > > As far as a transition plan, I think that this is an important > enough thing to have an accelerated process. If we need > to provide a warning than let’s add it to the next 3.4 otherwise > it’s going to be 2.5+ years until we stop being unsafe by > default. Fine with me; we're still early in the Python 3.4 patch level releases. > Another problem with this is that I don’t think it’s actually > possible to do. Python itself isn’t validating the TLS certificates, > OpenSSL is doing that. To my knowledge OpenSSL doesn’t > have a way to say “please validate these certificates and if > they don’t validate go ahead and keep going and just let me > get a warning from it”. It’s a 3 way switch, no validation, validation > if a certificate is provided, and validation always. > > Now that’s strictly for the “verify the certificate chain” portion, > the hostname verification is done entirely on our end and we > could do something there… but I’m not sure it makes sense > to do so if we can’t do it for invalid certificates too. OpenSSL provides a callback for certificate validation, so it is possible to issue a warning and continue with accepting the certificate. >> * configuration: >> >> It would be good to be able to switch this on or off >> without having to change the code, e.g. via a command >> line switch and environment variable; perhaps even >> controlling whether or not to raise an exception or >> warning. > > I’m on the fence about this, if someone provides a certificate > that we can validate against (which can be done without > touching the code) then the only thing that really can’t be > “fixed” without touching the code is if someone has a certificate > that is otherwise invalid (expired, not yet valid, wrong hostname, > etc). I’d say if I was voting on this particular thing I’d be -0, I’d > rather it didn’t exist but I wouldn’t cry too much if it did. If you're testing code or trying out some new stuff, you don't want to get a valid cert first, but instead go ahead with a self signed one. That's the use case. >> * choice of trusted certificate: >> >> Instead of hard wiring using the system CA roots into >> Python it would be good to just make this default and >> permit the user to point Python to a different set of >> CA roots. >> >> This would enable using self signed certs more easily. >> Since these are often used for tests, demos and education, >> I think it's important to allow having more control of >> the trusted certs. > > > Like my other email said, the Python API has everything needed > to easily specify your own CA roots and/or disable the validations. > The OpenSSL library also allows you to specify either a directory > or a file to change the root certificates without code changes. The > only real problems with the APIs are that the default is bad and > an unrelated thing where you can’t pass in an in memory certificate. Are you sure that's possible ? Python doesn't load the openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env vars only work for the openssl command line binary, AFAIK. In any case, Python will have to tap into the OS CA root provider using special code and this code could easily be made to check other dirs/files as well. The point is that it should be possible to change this default at the Python level, without needing application code changes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duessel
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Fri, 29 Aug 2014 17:11:35 -0400 Donald Stufft wrote: > > Another problem with this is that I don’t think it’s actually > possible to do. Python itself isn’t validating the TLS certificates, > OpenSSL is doing that. To my knowledge OpenSSL doesn’t > have a way to say “please validate these certificates and if > they don’t validate go ahead and keep going and just let me > get a warning from it”. Actually, there may be a solution. In client mode, OpenSSL always verifies the server cert chain and stores the verification result in the SSL structure. It will then only report an error if the verify mode is not SSL_VERIFY_NONE. (see ssl3_get_server_certificate() in s3_clnt.c) The verification result should then be readable using SSL_get_verify_result(), even with SSL_VERIFY_NONE. (note this is only from reading the source code and needs verifying) Then we could have the following transition phase: - define a new CERT_WARN value for SSLContext.verify_mode - use that value as the default in the HTTP stack (people who want the old silent default will have to set verify_mode explicitly to VERIFY_NONE) - with CERT_WARN, SSL_VERIFY_NONE is passed to OpenSSL and Python manually calls SSL_get_verify_result() after a handshake; if there was a verification error, a warning is printed out And in the following version we switch the HTTP default to CERT_REQUIRED. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft wrote: > Sorry I was on my phone and didnât get to fully reply to this. > > On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg wrote: > > > > * configuration: > > > > It would be good to be able to switch this on or off > > without having to change the code, e.g. via a command > > line switch and environment variable; perhaps even > > controlling whether or not to raise an exception or > > warning. > > Iâm on the fence about this, if someone provides a certificate > that we can validate against (which can be done without > touching the code) then the only thing that really canât be > âfixedâ without touching the code is if someone has a certificate > that is otherwise invalid (expired, not yet valid, wrong hostname, > etc). Iâd say if I was voting on this particular thing Iâd be -0, Iâd > rather it didnât exist but I wouldnât cry too much if it did. Especially if you want an accelerated change, there must be a way to *easily* get back to the previous behavior, or we are going to catch a lot of flack. There may be only 7% of public certs that are problematic, but I'd be willing to bet you that there are more not-really-public ones that are critical to day to day operations *somewhere* :) wget and curl have 'ignore validation' as a command line flag for a reason. --David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Sorry I was on my phone and didn’t get to fully reply to this. > On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg wrote: > > On 29.08.2014 21:47, Alex Gaynor wrote: >> Hi all, >> >> I've just submitted PEP 476, on enabling certificate validation by default >> for >> HTTPS clients in Python. Please have a look and let me know what you think. >> >> PEP text follows. > > Thanks for the PEP. I think this is generally a good idea, > but some important parts are missing from the PEP: > > * transition plan: > > I think starting with warnings in Python 3.5 and going > for exceptions in 3.6 would make a good transition > > Going straight for exceptions in 3.5 is not in line with > our normal procedures for backwards incompatible changes. As far as a transition plan, I think that this is an important enough thing to have an accelerated process. If we need to provide a warning than let’s add it to the next 3.4 otherwise it’s going to be 2.5+ years until we stop being unsafe by default. Another problem with this is that I don’t think it’s actually possible to do. Python itself isn’t validating the TLS certificates, OpenSSL is doing that. To my knowledge OpenSSL doesn’t have a way to say “please validate these certificates and if they don’t validate go ahead and keep going and just let me get a warning from it”. It’s a 3 way switch, no validation, validation if a certificate is provided, and validation always. Now that’s strictly for the “verify the certificate chain” portion, the hostname verification is done entirely on our end and we could do something there… but I’m not sure it makes sense to do so if we can’t do it for invalid certificates too. > > * configuration: > > It would be good to be able to switch this on or off > without having to change the code, e.g. via a command > line switch and environment variable; perhaps even > controlling whether or not to raise an exception or > warning. I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did. > > * choice of trusted certificate: > > Instead of hard wiring using the system CA roots into > Python it would be good to just make this default and > permit the user to point Python to a different set of > CA roots. > > This would enable using self signed certs more easily. > Since these are often used for tests, demos and education, > I think it's important to allow having more control of > the trusted certs. Like my other email said, the Python API has everything needed to easily specify your own CA roots and/or disable the validations. The OpenSSL library also allows you to specify either a directory or a file to change the root certificates without code changes. The only real problems with the APIs are that the default is bad and an unrelated thing where you can’t pass in an in memory certificate. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
> On Aug 29, 2014, at 4:00 PM, "M.-A. Lemburg" wrote: > > * choice of trusted certificate: > > Instead of hard wiring using the system CA roots into > Python it would be good to just make this default and > permit the user to point Python to a different set of > CA roots. > > This would enable using self signed certs more easily. > Since these are often used for tests, demos and education, > I think it's important to allow having more control of > the trusted certs. If I recall OpenSSL already allows this to be configured via envvar and the python API already allows it to be configured via API. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 08/29/2014 01:00 PM, M.-A. Lemburg wrote: On 29.08.2014 21:47, Alex Gaynor wrote: I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think. Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP: * transition plan: I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes. * configuration: It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning. * choice of trusted certificate: Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots. This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs. +1 for PEP with above changes. -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
Alex Gaynor gmail.com> writes: > > Hi all, > > I've just submitted PEP 476, on enabling certificate validation by default for > HTTPS clients in Python. Please have a look and let me know what you think. Yes please. The two most commons answers I get to "Why did you switch to go?" are "Concurrency" and "The stdlib HTTP client verifies TLS by default." In a work related survey of webhook providers I found that only ~7% of HTTPS URLs would be affected by a change like this. -David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 29.08.2014 21:47, Alex Gaynor wrote: > Hi all, > > I've just submitted PEP 476, on enabling certificate validation by default for > HTTPS clients in Python. Please have a look and let me know what you think. > > PEP text follows. Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP: * transition plan: I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes. * configuration: It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning. * choice of trusted certificate: Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots. This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 476: Enabling certificate validation by default!
Hi all, I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think. PEP text follows. Alex --- PEP: 476 Title: Enabling certificate verification by default for stdlib http clients Version: $Revision$ Last-Modified: $Date$ Author: Alex Gaynor Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-August-2014 Abstract Currently when a standard library http client (the ``urllib`` and ``http`` modules) encounters an ``https://`` URL it will wrap the network HTTP traffic in a TLS stream, as is necessary to communicate with such a server. However, during the TLS handshake it will not actually check that the server has an X509 certificate is signed by a CA in any trust root, nor will it verify that the Common Name (or Subject Alternate Name) on the presented certificate matches the requested host. The failure to do these checks means that anyone with a privileged network position is able to trivially execute a man in the middle attack against a Python application using either of these HTTP clients, and change traffic at will. This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. Rationale = The "S" in "HTTPS" stands for secure. When Python's users type "HTTPS" they are expecting a secure connection, and Python should adhere to a reasonable standard of care in delivering this. Currently we are failing at this, and in doing so, APIs which appear simple are misleading users. When asked, many Python users state that they were not aware that Python failed to perform these validations, and are shocked. The popularity of ``requests`` (which enables these checks by default) demonstrates that these checks are not overly burdensome in any way, and the fact that it is widely recommended as a major security improvement over the standard library clients demonstrates that many expect a higher standard for "security by default" from their tools. The failure of various applications to note Python's negligence in this matter is a source of *regular* CVE assignment [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_ [#]_. .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4340 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-3533 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5822 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-5825 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-1909 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2037 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2073 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-2191 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-4111 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6396 .. [#] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2013-6444 Technical Details = Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it. This can be achieved by simply replacing the use of ``ssl._create_stdlib_context`` with ``ssl.create_default_context`` in ``http.client``. Trust database -- This PEP proposes using the system-provided certificate database. Previous discussions have suggested bundling Mozilla's certificate database and using that by default. This was decided against for several reasons: * Using the platform trust database imposes a lower maintenance burden on the Python developers -- shipping our own trust database would require doing a release every time a certificate was revoked. * Linux vendors, and other downstreams, would unbundle the Mozilla certificates, resulting in a more fragmented set of behaviors. * Using the platform stores makes it easier to handle situations such as corporate internal CAs. Backwards compatibility --- This change will have the appearance of causing some HTTPS connections to "break", because they will now raise an Exception during handshake. This is misleading however, in fact these connections are presently failing silently, an HTTPS URL indicates an expectation of confidentiality and authentication. The fact that Python does not actually verify that the user's request has been made is a bug, further: "Errors should never pass silently." Nevertheless, users who have a need to access servers with self-signed or incorrect certificates would be able to do so by providing a context with custom trust roots or which disables validation (documentation should strongly recommend the former where possible). Users will also be able to add necessary certificates to system trust stores in order to trust them globally. Twisted's 14.0 r
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2014-08-22 - 2014-08-29) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open4638 (+17) closed 29431 (+32) total 34069 (+49) Open issues with patches: 2193 Issues opened (41) == #17095: Modules/Setup *shared* support broken http://bugs.python.org/issue17095 reopened by haypo #22200: Remove distutils checks for Python version http://bugs.python.org/issue22200 reopened by Arfrever #22232: str.splitlines splitting on non-\r\n characters http://bugs.python.org/issue22232 reopened by terry.reedy #22252: ssl blocking IO errors http://bugs.python.org/issue22252 opened by h.venev #22253: ConfigParser does not handle files without sections http://bugs.python.org/issue22253 opened by kernc #22255: Multiprocessing freeze_support raises RuntimeError http://bugs.python.org/issue22255 opened by Michael.McAuliffe #22256: pyvenv should display a progress indicator while creating an e http://bugs.python.org/issue22256 opened by ncoghlan #22257: PEP 432: Redesign the interpreter startup sequence http://bugs.python.org/issue22257 opened by ncoghlan #22258: set_inheritable(): ioctl(FIOCLEX) is available but fails with http://bugs.python.org/issue22258 opened by igor.pashev #22260: Rearrange tkinter tests, use test discovery http://bugs.python.org/issue22260 opened by zach.ware #22261: Document how to use Concurrent Build when using MsBuild http://bugs.python.org/issue22261 opened by sbspider #22263: Add a resource for CLI tests http://bugs.python.org/issue22263 opened by serhiy.storchaka #22264: Add wsgiref.util helpers for dealing with "WSGI strings" http://bugs.python.org/issue22264 opened by ncoghlan #22268: mrohasattr and mrogetattr http://bugs.python.org/issue22268 opened by Gregory.Salvan #22269: Resolve distutils option conflicts with priorities http://bugs.python.org/issue22269 opened by minrk #22270: cache version selection for documentation http://bugs.python.org/issue22270 opened by thejj #22271: Deprecate PyUnicode_AsUnicode(): emit a DeprecationWarning http://bugs.python.org/issue22271 opened by haypo #22273: abort when passing certain structs by value using ctypes http://bugs.python.org/issue22273 opened by weeble #22274: subprocess.Popen(stderr=STDOUT) fails to redirect subprocess s http://bugs.python.org/issue22274 opened by akira #22275: asyncio: enhance documentation of OS support http://bugs.python.org/issue22275 opened by haypo #22276: pathlib glob issues http://bugs.python.org/issue22276 opened by joca.bt #22277: webbrowser.py add parameters to suppress output on stdout and http://bugs.python.org/issue22277 opened by CristianCantoro #22278: urljoin duplicate slashes http://bugs.python.org/issue22278 opened by demian.brecht #22279: read() vs read1() in asyncio.StreamReader documentation http://bugs.python.org/issue22279 opened by oconnor663 #22281: ProcessPoolExecutor/ThreadPoolExecutor should provide introspe http://bugs.python.org/issue22281 opened by dan.oreilly #22282: ipaddress module accepts octal formatted IPv4 addresses in IPv http://bugs.python.org/issue22282 opened by xZise #22283: "AMD64 FreeBSD 9.0 3.x" fails to build the _decimal module: #e http://bugs.python.org/issue22283 opened by haypo #22284: decimal module contains less symbols when the _decimal module http://bugs.python.org/issue22284 opened by haypo #22285: The Modules/ directory should not be added to sys.path http://bugs.python.org/issue22285 opened by haypo #22286: Allow backslashreplace error handler to be used on input http://bugs.python.org/issue22286 opened by ncoghlan #22289: support.transient_internet() doesn't catch timeout on FTP test http://bugs.python.org/issue22289 opened by haypo #22290: "AMD64 OpenIndiana 3.x" buildbot: assertion failed in PyObject http://bugs.python.org/issue22290 opened by haypo #22292: pickle whichmodule RuntimeError http://bugs.python.org/issue22292 opened by attilio.dinisio #22293: unittest.mock: use slots in MagicMock to reduce memory footpri http://bugs.python.org/issue22293 opened by james-w #22294: 2to3 consuming_calls: len, min, max, zip, map, reduce, filter http://bugs.python.org/issue22294 opened by eddygeek #22295: Clarify available commands for package installation http://bugs.python.org/issue22295 opened by ncoghlan #22296: cookielib uses time.time(), making incorrect checks of expirat http://bugs.python.org/issue22296 opened by regu0004 #22297: 2.7 json encoding broken for enums http://bugs.python.org/issue22297 opened by eddygeek #22298: Lib/warnings.py _show_warning does not protect against being c http://bugs.python.org/issue22298 opened by Julius.Lehmann-Richter #22299: resolve() on Windows makes some pathological paths unusable http://bugs.python.org/issue22299 opened by Kevin.Norris #22300: PEP 446 What's New Updates for 2.
Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]
On Fri, 29 Aug 2014, M.-A. Lemburg wrote:
On 29.08.2014 02:41, Stephen J. Turnbull wrote:
Since Python allows working with lone surrogates in Unicode (they
are valid code points) and we're using UTF-8 for marshal, we needed
a way to make sure that Python 3 also optionally supports working
with lone surrogates in such UTF-8 streams (nowadays called CESU-8:
http://en.wikipedia.org/wiki/CESU-8).
If I want that wouldn't I specify "cesu-8" as the encoding?
i.e., instead of .decode ('utf-8') I would use .decode ('cesu-8'). Right
now, trying this I get that cesu-8 is an unknown encoding but that could
be changed without affecting the behaviour of the utf-8 codec.
It seems to me that .decode ('utf-8') should decode exactly and only valid
utf-8, including the non-use of surrogate pairs as an intermediate
encoding step.
Isaac Morland CSCF Web Guru
DC 2554C, x36650WWW Software Specialist
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]
On 29.08.2014 13:22, Isaac Morland wrote:
> On Fri, 29 Aug 2014, M.-A. Lemburg wrote:
>
>> On 29.08.2014 02:41, Stephen J. Turnbull wrote:
>> Since Python allows working with lone surrogates in Unicode (they
>> are valid code points) and we're using UTF-8 for marshal, we needed
>> a way to make sure that Python 3 also optionally supports working
>> with lone surrogates in such UTF-8 streams (nowadays called CESU-8:
>> http://en.wikipedia.org/wiki/CESU-8).
>
> If I want that wouldn't I specify "cesu-8" as the encoding?
>
> i.e., instead of .decode ('utf-8') I would use .decode ('cesu-8'). Right
> now, trying this I get
> that cesu-8 is an unknown encoding but that could be changed without
> affecting the behaviour of the
> utf-8 codec.
Why write a new codec that's almost identical to the utf-8 codec,
if you can get the same functionality by explicitly using a
special error handler ?
>From a maintenance POV that does not sound like a good approach.
> It seems to me that .decode ('utf-8') should decode exactly and only valid
> utf-8, including the
> non-use of surrogate pairs as an intermediate encoding step.
It does in Python 3.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Aug 29 2014)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62
2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go
2014-09-27: PyDDF Sprint 2014 ... 29 days to go
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bytes path related questions for Guido
On 28 Aug 2014, at 19:54, Glenn Linderman wrote:
On 8/28/2014 10:41 AM, R. David Murray wrote:
On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman
wrote:
[...]
Also for
cases where the data stream is *supposed* to be in a given encoding,
but
contains undecodable bytes. Showing the stuff that incorrectly
decodes
as whatever it decodes to is generally what you want in that case.
Sure, people can learn to recognize mojibake for what it is, and maybe
even learn to recognize it for what it was intended to be, in limited
domains. But suppressing/replacing the surrogates doesn't help with
that... would it not be better to replace the surrogates with an
escape sequence that shows the original, undecodable, byte value?
Like \xNN ?
For that we could extend the "backslashreplace" codec error callback, so
that it can be used for decoding too, not just for encoding. I.e.
b"a\xffb".decode("utf-8", "backslashreplace")
would return
"a\\xffb"
Servus,
Walter
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] surrogatepass - she's a witch, burn 'er! [was: Cleaning up ...]
On 29.08.2014 02:41, Stephen J. Turnbull wrote: > In the process of booking up for my other post in this thread, I > noticed the 'surrogatepass' handler. > > Is there a real use case for the 'surrogatepass' error handler? It > seems like a horrible break in the abstraction. IMHO, if there's a > need, the application should handle this. Python shouldn't provide > it on encoding as the resulting streams are not Unicode conformant, > nor on decoding UTF-16, as conversion of surrogate pairs is a > requirement of all Unicode versions since about 1995. This error handler allows applications to reactivate the Python 2 style behavior of the UTF codecs in Python 3, which allow reading lone surrogates on input. Since Python allows working with lone surrogates in Unicode (they are valid code points) and we're using UTF-8 for marshal, we needed a way to make sure that Python 3 also optionally supports working with lone surrogates in such UTF-8 streams (nowadays called CESU-8: http://en.wikipedia.org/wiki/CESU-8). See http://bugs.python.org/issue3672 http://bugs.python.org/issue12892 for discussions. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 29 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ 2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ...21 days to go 2014-09-27: PyDDF Sprint 2014 ... 29 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
