[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

i believe new work will be done via the new issue.  marking this closed.  if 
there is something not covered by issue38576 that remains, please open a new 
issue for it.  new discussion on this long issue is easy to get lost in.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Ned Deily]

Ned Deily  added the comment:

What is the status of this issue?  Now that Issue38576 has been opened to cover 
the host address part, can this issue be closed or downgraded?  Should 
Issue38576 be a release blocker?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Riccardo Schirone]

Riccardo Schirone  added the comment:

I have created https://bugs.python.org/issue38576 to address CVE-2019-18348.

@gregory.p.smith if you have particular complains about these CVEs feel free to 
let me know (even privately). I think the security impact of these flaws is: an 
application that relies on urlopen/HTTPConnection/etc. where either the query 
part, the path part or the host part are user-controlled, could be exploited to 
send unintended HTTP headers to other hosts (maybe services that would not be 
directly reachable by the user).

FYI, there were some good replies to that CVE talk, one of which is 
https://grsecurity.net/reports_of_cves_death_greatly_exaggerated .

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Can you please open a separate issue for CVE-2019-18348?  It is easier to track 
that way.

(META: In general I think the CVE process is being abused and that these really 
did not deserve that treatment.  https://lwn.net/Articles/801157/  is good 
reading and food for thought.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Riccardo Schirone]

Riccardo Schirone  added the comment:

CVE-2019-18348 has been assigned to the issue explained in 
https://bugs.python.org/issue30458#msg347282 . Maybe a separate bug for it 
would be better though. CVE-2019-18348 is about injecting CRLF in HTTP requests 
through the *host* part of a URL.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Jason R. Coombs]

Change by Jason R. Coombs :


--
pull_requests: +15900
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/16321

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Ned Deily]

Ned Deily  added the comment:

With the breaking out of the portential and/or actual regression (e.g. invalid 
requests can no longer be crafted) into Issue38216, itself a potential release 
blocker, we are still left here with the as-yet unresolved issue identified 
above in msg34728 (e.g. not checking for control characters in the "host" part 
of the URL, only the "path" part).  Since this also affects so many 
branches/releases and has external components (CVE's, third-party impacts), it 
probably would have made sense to break it out into a separate issue (and maybe 
it still does).  But since this problem has been present for many releases 
(apparently), I would rather not further hold the 3.7.5 release for a 
resolution (though that would be a good thing) so I'm going to change the 
priority for the moment to "deferred blocker".

But we need someone (preferably a core dev already involved) to take charge of 
this and push it to a resolution.  Thanks for everyone's help so far!

--
priority: release blocker -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I've created issue38216 to address the (perceived) regression.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Jason R. Coombs]

Jason R. Coombs  added the comment:

> Should we open a separate issue to track (fixing) the regression?

Yes, I think so. The ticket I referenced mainly addresses an incompatibility 
that was introduced with Python 3.0, so is much less urgent than the one 
introduced more recently, so I believe it deserves a proper, independent 
description and discussion. I'll gladly file that ticket, tonight most likely.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Larry Hastings]

Larry Hastings  added the comment:

Should we open a separate issue to track fixing the regression?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Ned Deily]

Ned Deily  added the comment:

If I understand Jason's message correctly, the changes for Issue30458 
introduced a regression in 3.7.4 and will introduce the same regression in 
other branches as they are released, including 3.5.8 whose rc1 is now in 
testing.  3.7.5rc1 is scheduled to be tagged later today.  Is this regression 
serious enough that we should hold 3.7.5 and/or 3.5.8 for a fix?  If so, there 
should probably be a separate issue for it unless it is necessarily intertwined 
with the resolution of Issue36274.

I'm provisionally setting the status of this issue to "release blocker".

--
nosy: +benjamin.peterson, lukasz.langa, ned.deily
priority: normal -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Jason R. Coombs]

Jason R. Coombs  added the comment:

This change caused a regression or two captured in issue36274. Essentially, by 
blocking invalid requests, it's now not possible for a system intentionally to 
generate invalid requests for testing purposes. As these point releases of 
Python start making it into the wild, the impact of this change will likely 
increase.

I think this patch was applied at too low a level. That is, instead of 
protecting the user inputs, the change protects the programmer's inputs.

I mention this here so those interested can follow the mitigation work 
happening in issue36274.

--
nosy: +jaraco

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

I'm not a fan of CVE numbers in general, people have been creating too many of 
those.  But that also means I just don't care if someone does.  Having a CVE 
entry is not a way to claim something is important.

This issue is still open and can be used to track dealing with the host.

--
assignee: larry -> 
keywords: +security_issue
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Riccardo Schirone]

Riccardo Schirone  added the comment:

Will the flaw outlined in https://bugs.python.org/issue30458#msg347282 be fixed 
in python itself? If so, I think a CVE for python should be requested to MITRE 
(I can request one, in that case).

Moreover, does it make sense to create a new bug to track the new issue? This 
bug already references 3 CVEs and it would probably just create more confusion 
to reference a 4th. What do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Larry Hastings]

Larry Hastings  added the comment:


New changeset afe3a4975cf93c97e5d6eb8800e48f368011d37a by larryhastings (Miro 
Hrončok) in branch '3.5':
bpo-30458: Disallow control chars in http URLs. (GH-12755) (#13207)
https://github.com/python/cpython/commit/afe3a4975cf93c97e5d6eb8800e48f368011d37a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Riccardo Schirone]

Riccardo Schirone  added the comment:

> > A second problem comes into the game. Some C libraries like glibc strip the 
> > end of the hostname (strip at the first newline character) and so HTTP 
> > Header injection is still possible is this case: 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1673465

> The bug link raises permission error. Does fixing the host part fix this 
> issue too since there won't be any socket connection made? Is it possible to 
> have a Python reproducer of this issue?

I think this was supposed to refer to CVE-2016-10739 
(https://bugzilla.redhat.com/show_bug.cgi?id=1347549)

--
nosy: +rschiron

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Okay, the url variable against which the regex check is made is not the full 
url but the path. The HTTPConnection class sets self.host [0] in the 
constructor which is used to send the Host header. Perhaps the regex check 
could be done for the host too given the path check is already done in the 
previous commit. With that the reported host also throws a 
http.client.InvalidURL exception.

> A second problem comes into the game. Some C libraries like glibc strip the 
> end of the hostname (strip at the first newline character) and so HTTP Header 
> injection is still possible is this case: 
> https://bugzilla.redhat.com/show_bug.cgi?id=1673465

The bug link raises permission error. Does fixing the host part fix this issue 
too since there won't be any socket connection made? Is it possible to have a 
Python reproducer of this issue?

[0] 
https://github.com/python/cpython/blob/7f41c8e0dd237d1f3f0a1d2ba2f3ee4e4bd400a7/Lib/http/client.py#L829

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

The commit b7378d77289c911ca6a0c0afaf513879002df7d5 is incomplete: it doesn't 
seem to check for control characters in the "host" part of the URL, only in the 
"path" part of the URL. Example:
---
try:
from urllib import request as urllib_request
except ImportError:
import urllib2 as urllib_request
import socket
def bug(*args):
raise Exception(args)
# urlopen() must not call create_connection()
socket.create_connection = bug
urllib_request.urlopen('http://127.0.0.1\r\n\x20hihi\r\n :11211')
---

The URL comes from the first message of this issue:
https://bugs.python.org/issue30458#msg294360

Development branches 2.7 and master produce a similar output:
---
Traceback (most recent call last):
 ...
Exception: (('127.0.0.1\r\n hihi\r\n ', 11211), ..., None)
---

So urllib2/urllib.request actually does a real network connection (DNS query), 
whereas it should reject control characters in the "host" part of the URL.

***

A second problem comes into the game. Some C libraries like glibc strip the end 
of the hostname (strip at the first newline character) and so HTTP Header 
injection is still possible is this case:
https://bugzilla.redhat.com/show_bug.cgi?id=1673465

***

According to the RFC 3986, the "host" grammar doesn't allow any control 
character, it looks like:

   host  = IP-literal / IPv4address / reg-name

   ALPHA (letters)
   DIGIT (decimal digits)
   unreserved= ALPHA / DIGIT / "-" / "." / "_" / "~"
  pct-encoded = "%" HEXDIG HEXDIG
  sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
  / "*" / "+" / "," / ";" / "="
   reg-name  = *( unreserved / pct-encoded / sub-delims )

   IP-literal= "[" ( IPv6address / IPvFuture  ) "]"
   IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" )
   IPv6address   =6( h16 ":" ) ls32
 /   "::" 5( h16 ":" ) ls32
 / [   h16 ] "::" 4( h16 ":" ) ls32
 / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
 / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
 / [ *3( h16 ":" ) h16 ] "::"h16 ":"   ls32
 / [ *4( h16 ":" ) h16 ] "::"  ls32
 / [ *5( h16 ":" ) h16 ] "::"  h16
 / [ *6( h16 ":" ) h16 ] "::"
   h16   = 1*4HEXDIG
   ls32  = ( h16 ":" h16 ) / IPv4address
   IPv4address   = dec-octet "." dec-octet "." dec-octet "." dec-octet

--
versions: +Python 2.7, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

Note for myself: Python 2 urllib.urlopen(url) always quotes the URL and so is 
not vulnerable to HTTP Header Injection (at least, not to this issue ;-)).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Roundup Robot]

Change by Roundup Robot :


--
pull_requests: +13655
pull_request: https://github.com/python/cpython/pull/13771

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Sihoon Lee]

Change by Sihoon Lee :


--
pull_requests: +13545
pull_request: https://github.com/python/cpython/pull/12524

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
pull_requests: +13546
pull_request: https://github.com/python/cpython/pull/11768

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

Assigning to Larry to decide if he wants to merge that PR into 3.5 or not.

--
assignee:  -> larry
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
versions:  -Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
versions:  -Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:


New changeset bb8071a4cae5ab3fe321481dd3d73662ffb26052 by Victor Stinner in 
branch '2.7':
bpo-30458: Disallow control chars in http URLs (GH-12755) (GH-13154) (GH-13315)
https://github.com/python/cpython/commit/bb8071a4cae5ab3fe321481dd3d73662ffb26052


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

I backported the fix from Python 3.7 to Python 2.7: PR 13315.

Please review it carefully, I had to make multiple changes to adapt the fix to 
Python 2:

* non-ASCII characters are explicitly rejected
* urllib doesn't reject control characters: they are quoted properly, so I 
addapted test_urllib
* urllib2 doesn't quote the URL and so reject control characters, I added tests 
to test_urllib2
* I replaced http.client with httplib
* I replaced urllib.request with urllib or urllib2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

Change by STINNER Victor :


--
pull_requests: +13225

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Charalampos Stratakis]

Charalampos Stratakis  added the comment:

A small clarification on the differences of those two CVE's.

CVE-2019-9740: CLRF sequences are not properly handled in python built-in 
modules urllib/urllib2 in the query part of the url parameter of urlopen() 
function

CVE-2019-9947: CLRF sequences are not properly handled in python built-in 
modules urllib/urllib2 in the path part of the url parameter of urlopen() 
function

--
nosy: +cstratak -hroncok, koobs, ned.deily
versions: +Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Miro Hrončok]

Change by Miro Hrončok :


--
pull_requests: +13118

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Ned Deily]

Ned Deily  added the comment:


New changeset c50d437e942d4c4c45c8cd76329b05340c02eb31 by Ned Deily (Miro 
Hrončok) in branch '3.6':
bpo-30458: Disallow control chars in http URLs. (GH-12755) (GH-13155)
https://github.com/python/cpython/commit/c50d437e942d4c4c45c8cd76329b05340c02eb31


--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
versions:  -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset 7e200e0763f5b71c199aaf98bd5588f291585619 by Gregory P. Smith 
(Miro Hrončok) in branch '3.7':
bpo-30458: Disallow control chars in http URLs. (GH-12755) (GH-13154)
https://github.com/python/cpython/commit/7e200e0763f5b71c199aaf98bd5588f291585619


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Miro Hrončok]

Change by Miro Hrončok :


--
pull_requests: +13072

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Miro Hrončok]

Change by Miro Hrončok :


--
pull_requests: +13071

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Kubilay Kocak]

Change by Kubilay Kocak :


--
nosy: +koobs

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Miro Hrončok]

Miro Hrončok  added the comment:

I'll work on 3.7 backport.

--
nosy: +hroncok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

> One thing to note though is that they claim URLs with spaces embedded in them 
> are apparently somewhat common in the world, we might want to relax our check 
> to not include space (\x20) in the rejected characters for that reason.

Guess I missed it in the PR discussion and read your comment [0] now about the 
change from golang's fix that excluded space as a problematic character. Is it 
worth documenting this change somewhere like a versionchanged directive in 
http.client?

[0] https://github.com/python/cpython/pull/12755#discussion_r279888496

Thanks for the details.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

A note from the urllib3 fixes to this:  They chose to go the route of 
auto-%-encoding the offending characters in URLs instead.  I do not think the 
stdlib should do this.

One thing to note though is that they claim URLs with spaces embedded in them 
are apparently somewhat common in the world, we might want to relax our check 
to not include space (\x20) in the rejected characters for that reason.

A space alone cannot be used for injection.  Someone could append an incorrect 
HTTP protocol version to a request using it " HTTP/1.0" but that would be 
followed by the actual " HTTP/x.y" generated by our library which at that point 
is up to the server to parse and or reject as odd.  Without the ability to 
inject \r\n the headers to go with the protocol cannot be modified; so a change 
in protocol version could at most alter how some headers may be treated.  Worst 
case: they upgrade/downgrade the http version in a non-pedantic server - i 
believe this to be low impact (feel free to prove me wrong with a working 
example against a common server).  Best case: The server rejects the 
unparseable request or considers their " HTTP/1.0" to be part of their URL path.

In a world where unescaped spaces in URLs are common, some servers _might_ 
already take the strategy of splitting only on the first and last spaces in the 
request line anyways, considering everything in the middle to be the url with 
embedded spaces.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

IMO it does qualify as a security issue. In case of urllib to be lenient and 
can be exploited it's good to document like tarfile and xml modules that have a 
warning about untrusted data potentially causing issues and perhaps link to a 
url validator that adheres to RFC in pypi. I would expect stdlib to handle this 
but in case it's not handled due to backwards compatibility and potential 
regressions a warning could be made about the same in the docs noting down the 
responsibility of the functions and that they are not always safe against 
malicious data.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset b7378d77289c911ca6a0c0afaf513879002df7d5 by Gregory P. Smith in 
branch 'master':
bpo-30458: Use InvalidURL instead of ValueError. (GH-13044)
https://github.com/python/cpython/commit/b7378d77289c911ca6a0c0afaf513879002df7d5


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
pull_requests: +12964
stage: backport needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
stage: patch review -> backport needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[miss-islington]

miss-islington  added the comment:


New changeset 2fc936ed24cf04ed32f6015a8aa78c8ea40da66b by Miss Islington (bot) 
(Xtreak) in branch 'master':
bpo-30458: Disable https related urllib tests on a build without ssl (GH-13032)
https://github.com/python/cpython/commit/2fc936ed24cf04ed32f6015a8aa78c8ea40da66b


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
pull_requests: +12953
stage: backport needed -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

Sorry, I will toggle back the issue status. Not sure why bpo didn't warn in 
this case.

--
assignee: gregory.p.smith -> 
stage: patch review -> backport needed
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

This causes buildbot failure (AMD64 FreeBSD 10-STABLE Non-Debug 3.x and AMD64 
Debian root 3.x). I tried debugging and it's reproducible on my mac machine 
that has python not built with ssl and not reproducible on Ubuntu machine built 
with ssl. 

The failed tests use https scheme and as I can see from the file there is one 
another test (test_cafile_and_context) which is skipped and has skip test if 
ssl is absent @unittest.skipUnless(ssl, "ssl module required"). It seems 
perhaps wrapping these machines don't have ssl built with skip test might help. 
Since primary CI has ssl built it would have been caught.

On trying to add a print statement for lookup variable at 
https://github.com/python/cpython/blob/c4e671eec20dfcb29b18596a89ef075f826c9f96/Lib/urllib/request.py#L485
 I can see the below output where httpshandler was not defined for machines 
without built ssl. HTTPSConnection was not present as part of http.client due 
to import ssl causing ImportError.

Ubuntu with ssl

{'unknown': [], 'http': 
[], 'ftp': 
[], 'file': 
[], 'data': 
[], 'https': 
[]}

Mac without ssl (https handler missing causing unknown to be taken up for the 
test)

{'unknown': [], 'http': 
[], 'ftp': 
[], 'file': 
[], 'data': 
[]}

Gregory, I can create a PR with below patch if my analysis right to see if it 
helps or you can try a buildbot-custom branch to see if this works with 
buildbots since my PR won't have any effect on primary CI which have ssl built 
version of Python. I am not sure I have privileges to trigger a custom buildbot 
run. I tested it on my Mac without ssl and it has no failures since the tests 
are skipped.

diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
index e87c85b928..c5b23f935b 100644
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -329,6 +329,7 @@ class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin, 
FakeFTPMixin):
 finally:
 self.unfakehttp()

+@unittest.skipUnless(ssl, "ssl module required")
 def test_url_with_control_char_rejected(self):
 for char_no in list(range(0, 0x21)) + [0x7f]:
 char = chr(char_no)
@@ -354,6 +355,7 @@ class urlopen_HttpTests(unittest.TestCase, FakeHTTPMixin, 
FakeFTPMixin):
 finally:
 self.unfakehttp()

+@unittest.skipUnless(ssl, "ssl module required")
 def test_url_with_newline_header_injection_rejected(self):
 self.fakehttp(b"HTTP/1.1 200 OK\r\n\r\nHello.")
 host = "localhost:?a=1 HTTP/1.1\r\nX-injected: header\r\nTEST: 123"

--
assignee:  -> gregory.p.smith
stage: backport needed -> patch review
versions: +Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Change by Gregory P. Smith :


--
assignee: gregory.p.smith -> 
stage: patch review -> backport needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

backports to older releases will need to be done manually and take care 
depending on how much of a concern tightening the existing abusive lenient 
behavior of the http.client API to enforce what characters are allowed in URLs 
is to stable releases.

I question if this is _really_ worthy of a "security" tag and a CVE (thus its 
non-high ranking)... it is a bug in the calling program if it blindly uses 
untrusted data as a URL.  What this issue addresses is that we catch that more 
often and raise an error; a good thing to do for sure, but the stdlib should be 
the last line of defense.

--
versions:  -Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:


New changeset c4e671eec20dfcb29b18596a89ef075f826c9f96 by Gregory P. Smith in 
branch 'master':
bpo-30458: Disallow control chars in http URLs. (GH-12755)
https://github.com/python/cpython/commit/c4e671eec20dfcb29b18596a89ef075f826c9f96


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

> urllib3 now vendors a copy of the rfc3986 library:
> https://pypi.org/project/rfc3986/

There are multiple Python projects to validate URI:

* https://github.com/python-hyper/rfc3986/ -> https://pypi.org/project/rfc3986/
* https://github.com/dgerber/rfc3987 -> https://pypi.org/project/rfc3987/ (the 
name is confusing: the library implements the RFC 3986, not the RFC 3987)
* https://github.com/tkem/uritools/ -> https://pypi.org/project/uritools/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

"wave Hi! I've noticed that CVE-2019-11236 has been assigned to the CRLF 
injection issue described here. It seems that the library has been patched in 
GitHub, but no new release has been made to pypi. (...)"

This urllib3 change:
https://github.com/urllib3/urllib3/commit/0aa3e24fcd75f1bb59ab159e9f8adb44055b2271

urllib3 now vendors a copy of the rfc3986 library:

https://pypi.org/project/rfc3986/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

It seems like a change has been pushed into urllib3 to fix this issue, but that 
there is an issue with international URLs and that maybe RFC 3986 should be 
updated.

RFC 3986: "Uniform Resource Identifier (URI): Generic Syntax" (January 2005)
https://www.ietf.org/rfc/rfc3986.txt

"Without #1531 or IRI support in rfc3986 releasing master in it's current state 
will break backwards compatibility with international URLs."

https://github.com/urllib3/urllib3/issues/1553#issuecomment-474046652

=> where 1531 means https://github.com/urllib3/urllib3/pull/1531

"wave Hi! I've noticed that CVE-2019-11236 has been assigned to the CRLF 
injection issue described here. It seems that the library has been patched in 
GitHub, but no new release has been made to pypi. Will a new release containing 
the fix be made to pypi soon? Based on @theacodes comment it seems like a 
release was going to be made, but I also see her status has her perhaps 
unavailable. Is someone else perhaps able to cut a new release into pypi?"

https://github.com/urllib3/urllib3/issues/1553#issuecomment-484113222

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

> Will this break something in the world other than our own test_xmlrpc test?  
> Probably. Do they have a right to complain about it?  Not one we need listen 
> to.

I understand. But. Can we consider that for old Python versions like Python 2.7 
and 3.5?

This change will be applied to all supported Python versions.

I recall that when Python 2.7 started to validate TLS certificate, the change 
broke some applications. Are these applications badly written? Yes! But well, 
"it worked well before". Sometimes, when you work in a private network, the 
security matters less, whereas it might be very expensive to fix a legacy 
application. At Red Hat, we developed a solution to let customers to opt-out 
from this fix (to no validate TLS certificates), because it is just too 
expensive for customers to fix their legacy code but they would like to be able 
to upgrade RHEL.

One option to not validate URLs is to downgrade Python, but I'm not sure that 
it's the best compromise :-/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Gregory P. Smith]

Gregory P. Smith  added the comment:

> *Maybe* we need to provide a way to allow to pass junk characters in an URL? 
> (disable URL validation)

We should not do this in our http protocol stack code.  Anyone who _wants_ that 
is already intentionally violating the http protocol which defeats the entire 
purpose of our library and the parameter named "url".

Will this break something in the world other than our own test_xmlrpc test?  
Probably.  Do they have a right to complain about it?  Not one we need listen 
to.  Such code is doing something that was clearly an abuse of the API.  The 
parameter was named url not 
raw_data_to_stuff_subversively_into_the_binary_protocol.  Its intent was clear.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

Since this issue has a long history and previously attempts to fix it failed, 
it seems like the Internet is a black or white world, more like a scale of 
gray... *Maybe* we need to provide a way to allow to pass junk characters in an 
URL? (disable URL validation)

Idea: add an optional parameter to urllib, httplib, maybe also ftplib, to allow 
arbitrary "invalid" URLs / FTP commands. It would be a parameter *per request*, 
not a global option.

I don't propose to have a global configuration option like an environment 
variable, urllib attribute or something else. A global option would be hard to 
control and would impact just too much code.

My PEP 433 has been rejected because of the sys.setdefaultcloexec(cloexec: 
bool) function which allowed to change globally the behavior of Python. The PEP 
446 has been accepted with no *global* option to opt-in for the old behavior, 
but only "local" *per file descriptor*: os.set_inheritable(fd, inheritable).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

> According to the following message, urllib3 is also vulnerable to HTTP Header 
> Injection: (...)

And the issue has been reported to urllib3:
https://github.com/urllib3/urllib3/issues/1553

Copy of the first message:

"""
At https://bugs.python.org/issue36276 there's an issue in Python's urllib that 
an attacker controlling the request parameter can inject headers by injecting 
CR/LF chars.

A commenter mentions that the same bug is present in urllib3:
https://bugs.python.org/issue36276#msg337837

So reporting it here to make sure it gets attention.
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

STINNER Victor  added the comment:

bpo-36276 has been marked as a duplicate of this issue.

According to the following message, urllib3 is also vulnerable to HTTP Header 
Injection:
https://bugs.python.org/issue36276#msg337837

Copy of Alvin Chang's msg337837:

"""
I am also seeing the same issue with urllib3 

import urllib3

pool_manager = urllib3.PoolManager()

host = "localhost:?a=1 HTTP/1.1\r\nX-injected: header\r\nTEST: 123"
url = "http://; + host + ":8080/test/?test=a"

try:
info = pool_manager.request('GET', url).info()
print(info)
except Exception:
pass

nc -l localhost 
GET /?a=1 HTTP/1.1
X-injected: header
TEST: 123:8080/test/?test=a HTTP/1.1
Host: localhost:
Accept-Encoding: identity
"""

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Martin Panter]

Martin Panter  added the comment:

Gregory, I haven’t tried recent Python code, but I expect the problem with 
percent decoding is still there. If you did try my example, what results did 
you see? Be aware that these techniques only work if the OS co-operates and 
connects to localhost when you give it the longer host string. At the moment I 
have glibc 2.26 on x86-64 Linux.

In the Python 3 master branch, the percent-encoding should be decoded in 
“urllib.request.Request._parse”:

def _parse(self):
...
self.host, self.selector = _splithost(rest)
if self.host:
self.host = unquote(self.host)

Then in “AbstractHTTPHandler.do_request_” the decoded host string becomes the 
“Host” header field value, without any encoding:

def do_request_(self, request):
host = request.host
...
sel_host = host
...
if not request.has_header('Host'):
request.add_unredirected_header('Host', sel_host)

Perhaps one solution to both my version and Orange’s original version is to 
encode the “Host” header field value properly. This might also apply to the 
“http.client” code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[Karthikeyan Singaravelan]

Karthikeyan Singaravelan  added the comment:

As @gregory.p.smith noted in GitHub [0] this fixes only protocol level bugs. 
There are some parsing ambiguities in urllib that are potential security issues 
still to be fixed.

issue20271 - urllib.urlparse('http://benign.com\[attacker.com]') returns 
attacker.com as hostname . A slightly related issue 
https://bugs.python.org/issue20271
issue35748 - urllib.urlparse(r'http://spam\eggs!cheese@evil.com') 
returns evil.com as hostname
issue23505 - Urlparse insufficient validation leads to open redirect
issue33661 - urllib may leak sensitive HTTP headers to a third-party web site 
(Redirecting from https to http might also pass some headers in plain text. 
This behavior was changed in requests, golang, Curl that had their own 
respective CVEs)

As a fun side note this vulnerability was used by one of our own tests as a 
feature from 2012 to test another security issue (issue14001) [1] :) 

[0] https://github.com/python/cpython/pull/12755#issuecomment-481599611
[1] https://github.com/python/cpython/pull/12755#issuecomment-481618741

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30458] [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of CVE-2016-5699)

[STINNER Victor]

Change by STINNER Victor :


--
title: [CVE-2019-9740][CVE-2019-9947][security] CRLF Injection in httplib -> 
[security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection (follow-up of 
CVE-2016-5699)

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com