[issue11662] Redirect vulnerability in urllib/urllib2

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:

I think this is another patch that needs to be cross-ported to the 2.6 svn 
branch (which I'll do).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


--
nosy: +jcea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +Arfrever

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

Ok, merged into the central repo. Let me know where I screwed up.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Roundup Robot]

Roundup Robot  added the comment:

New changeset 5937d2119a20 by guido in branch '3.1':
Issue 11662: Fix vulnerability in urllib/urllib2.
http://hg.python.org/cpython/rev/5937d2119a20

New changeset 96a6c128822b by guido in branch '3.2':
Merge Issue 11662 from 3.1 branch.
http://hg.python.org/cpython/rev/96a6c128822b

New changeset a778b963eae3 by guido in branch 'default':
Merge Issue 11662 from 3.2 branch.
http://hg.python.org/cpython/rev/a778b963eae3

New changeset 9eeda8e3a13f by Guido van Rossum in branch '2.6':
Merge issue 11662 from 2.5.
http://hg.python.org/cpython/rev/9eeda8e3a13f

New changeset b2934d98dac1 by Guido van Rossum in branch '2.7':
Merge issue 11662 from 2.6.
http://hg.python.org/cpython/rev/b2934d98dac1

New changeset 3dc90ebc540a by Guido van Rossum in branch '3.1':
Merge issue 11662.
http://hg.python.org/cpython/rev/3dc90ebc540a

New changeset 968bca2cab60 by Guido van Rossum in branch '3.2':
Merge issue 11662.
http://hg.python.org/cpython/rev/968bca2cab60

New changeset c8701b9256cf by Guido van Rossum in branch 'default':
Merge issue 11662.
http://hg.python.org/cpython/rev/c8701b9256cf

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Benjamin Peterson]

Benjamin Peterson  added the comment:

2011/3/29 Guido van Rossum :
>
> Guido van Rossum  added the comment:
>
> The fix is now also in the 3.1, 3.2 and default branches of my repo
> (http://hg.python.org/sandbox/guido).
>
> Maybe I should just merge the whole bunch into the root repo and be
> done with it?

Sounds good.

BTW, you should probably put your name your hg username messages by
setting username in .hgrc to

username = "Guido van Rossum "

or so

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

The fix is now also in the 3.1, 3.2 and default branches of my repo
(http://hg.python.org/sandbox/guido).

Maybe I should just merge the whole bunch into the root repo and be
done with it?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

Also for the Python 3 family it's best to backport Senthil's patch. I will try 
that in my tree as well.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

I have the final version of the patch for Python 2 in the 2.5, 2.6 and 2.7 
branches in my repo (http://hg.python.org/sandbox/guido).

What's the next step?  Just push this to the central repo?  There are a few 
separate changes:

summary: Merge urllib/urllib2 security fix from 2.6 branch.
summary: Merge urllib/urllib2 security fix from 2.5 branch.
summary: Adding .hgignore (copied from default branch).
summary: Add CVE number to urllib/urllib2 news item.
summary: Add tests for the urllib[2] vulnerability. Change to raise 
exceptions.
summary: Add FTP to the allowed url schemes. Add Misc/NEWS.
summary: Issue 22663: fix redirect vulnerability in urllib/urllib2.

--
nosy:  -serdar.dalgic
versions: +Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Antoine Pitrou]

Antoine Pitrou  added the comment:

I don't have a 2.5 checkout to test but the patch looks ok to me.
Under 2.7 I get a test failure, I suppose you'll have some merging work to do:

test test_urllib2 failed -- Traceback (most recent call last):
  File "/home/antoine/cpython/27/Lib/test/test_urllib2.py", line 990, in 
test_invalid_redirect
MockHeaders({"location": valid_url}))
  File "/home/antoine/cpython/27/Lib/urllib2.py", line 616, in http_error_302
return self.parent.open(new, timeout=req.timeout)
  File "/home/antoine/cpython/27/Lib/urllib2.py", line 219, in __getattr__
raise AttributeError, attr
AttributeError: timeout

--
versions:  -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

This issue was first reported by Niels Heinen from the Google Security Team.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Serdar Dalgic]

Changes by Serdar Dalgic :


--
nosy: +serdar.dalgic

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Changes by Guido van Rossum :


Removed file: http://bugs.python.org/file21441/9d06d5eb1a7e.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Changes by Guido van Rossum :


Added file: http://bugs.python.org/file21442/f03e2acb9826.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

Aha. I now see the point of raising an exception instead of just returning 
None. I have backported Senthil's patch to the 2.5 branch. Please review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Changes by Guido van Rossum :


Added file: http://bugs.python.org/file21441/9d06d5eb1a7e.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Henri Salo]

Henri Salo  added the comment:

CVE-2011-1521 has been assigned to this issue.

--
nosy: +Henri.Salo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Changes by Senthil Kumaran :


Added file: http://bugs.python.org/file21388/ff71c4416cde.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

On Thu, Mar 24, 2011 at 05:32:42PM +, Guido van Rossum wrote:
> I still don't think we should raise URLError on the bad redirect; we
> should treat it the same as a missing URI/Location header, and it
> will raise HTTPError.

Agreed. Updated the hg repository by raising HTTPError instead of
URLError.

Thing to note - HTTPError does not change anything from the
redirection. It would still give the code as 302 with an additional
message saying that Redirection to 'newurl' is not allowed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Changes by Guido van Rossum :


Added file: http://bugs.python.org/file21377/ca3b117c40f3.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

I am okay with adding FTP to the list.

I still don't think we should raise URLError on the bad redirect; we should 
treat it the same as a missing URI/Location header, and it will raise HTTPError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> > Senthil's patch allows a redirect to ftp while Guido's doesn't.
> 
> That is a good question. Should we? It doesn't look like ftp:
> participates in the vulnerability, but I'm not sure how useful it is
> either.

I would say accept it anyway. That way we minimize potential for
compatibility breakage.
(do we support "ftps" as well? I don't think so)

> > Senthil's patch doesn't seem to fix urllib-inherited code, only
> urllib2- (see FancyURLopener.redirect_internal()).
> 
> Right, that's for Python 3.

FancyURLopener is still present in Python 3 (even though we would like
to deprecate it in 3.3).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

Here is a more complete patch with tests. Please review this. Yes, it is 
against the default branch (3.x codeline). We can backport this behavior to 2.x 
codeline.

I have raised an URLError exception when the direct to invalid_schemes is 
detected.

Also, ftp redirection should be allowed. It is common to see ISO download 
mirrors which will redirect itself to an ftp url. Also the security report says 
about allowing to http, https and ftp.

Thanks.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Changes by Senthil Kumaran :


Added file: http://bugs.python.org/file21376/3c07ea6a176a.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

> Which patch should be reviewed? They seem to be different.

Both. Mine's for the Python 2 line while Senthil seems to deal with
Python 3. (However the presence of Senthil's patch somehow overrode my
patch in Rietveld. It looks like Martin didn't think of this use
case.) I'd like to have agreement over the Python 2 patch first, then
we can think about forward porting.

> Senthil's patch allows a redirect to ftp while Guido's doesn't.

That is a good question. Should we? It doesn't look like ftp:
participates in the vulnerability, but I'm not sure how useful it is
either.

> Senthil's patch doesn't seem to fix urllib-inherited code, only urllib2- (see 
> FancyURLopener.redirect_internal()).

Right, that's for Python 3.

> Guido's patch doesn't close the file (fp.close()) when the redirect is denied.

But the calling code does. Note that when there is no URI or Location
header, redirect_internal() also returns without closing the file; if
the error handler returns no result, http_error() will call
http_error_default() which closes the file.

> Both patches apparently return silently (?), while it might be better to 
> raise an exception.

This follows the tradition of returning silently when no URI or
Location header is found. The 302 error will be treated the same as
any other error.

> Both would deserve a test :)

If someone would contribute one I'd appreciate it. Otherwise I will
get on it myself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[R. David Murray]

R. David Murray  added the comment:

Yes there is a delay.  The cron job that creates the link runs every two 
minutes.  Not sure why the delay seems to be longer than that, though.

--
nosy: +r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[STINNER Victor]

STINNER Victor  added the comment:

c6a4d267fe88.diff: This patch doesn't explain why other scheme are not allowed. 
I like Guido's comment:

# For security reasons we do not allow redirects to protocols
# other than HTTP or HTTPS.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Which patch should be reviewed? They seem to be different. Senthil's patch 
allows a redirect to ftp while Guido's doesn't.

Senthil's patch doesn't seem to fix urllib-inherited code, only urllib2- (see 
FancyURLopener.redirect_internal()).

Guido's patch doesn't close the file (fp.close()) when the redirect is denied.

Both patches apparently return silently (?), while it might be better to raise 
an exception.
Both would deserve a test :)

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

Oddly, I now see a review link for my own diff but not for orsenthil's. Maybe 
there's a delay?

I could use help with the tests.

I suppose orsenthil's patch is for Python 3?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[STINNER Victor]

STINNER Victor  added the comment:

The patch has no test. You may read our new "Python Developer’s Guide" for new 
contributors:
http://docs.python.org/devguide/runtests.html#writing

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Changes by Senthil Kumaran :


Added file: http://bugs.python.org/file21374/c6a4d267fe88.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

>> why doesn't it have a "review" link?

Perhaps, as it is not against the 'default'?

Let's try my hg sandbox link which has a fix committed. Let's see if it gives 
the review link.

--
hgrepos: +7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Guido van Rossum  added the comment:

Please review the patch that I created. (Now why doesn't it have a "review" 
link?) Note that the patch currently only allows http and https.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[STINNER Victor]

STINNER Victor  added the comment:

Repository URL is incorrect (missing http:/ prefix). The commit:
http://hg.python.org/sandbox/guido/rev/dd852a0f92d6

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

Changes by Guido van Rossum :


--
keywords: +patch
Added file: http://bugs.python.org/file21372/dd852a0f92d6.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Senthil Kumaran]

Senthil Kumaran  added the comment:

>> HTTPRedirectHandler behaviour can be changed
>> to only allow redirects to HTTP, HTTPS and FTP by checking the scheme
>> of the location URL (this seems to be a common practise in browsers)

This would be the way to go.

--
nosy: +orsenthil

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11662] Redirect vulnerability in urllib/urllib2

[Guido van Rossum]

New submission from Guido van Rossum :

We received the following on the security list. With the OP's permission I am 
now filing a public bug with a patch, with the intent to submit the patch ASAP 
(in time for MvL's planned April security release of Python 2.5).

The OP's description is below; I will attach a patch to this issue as soon as I 
have figured out how.


description:

The Python urllib and urllib2 modules are typically used to fetch web
pages but by default also contains handlers for ftp:// and file:// URL
schemes.

Now unfortunately it appears that it is possible for a web server to
redirect (HTTP 302) a urllib request to any of the supported
schemes. Examples on how this could turn bad:

 1) File disclosure: A web application, that normally fetches and
 displays a web page, is redirected to file:///etc/passwd and
 discloses it.

 2) Denial of Service: An application is redirected to a system device
 (e.g. file:///dev/zero) which will result in excessive CPU/memory/disk
 usage.

Affected versions:
--
The urllib and urllib2 modules of python 2.4.6 and 2.6.5 where tested
but this likely affects all versions.

Possible solution:
--
The default handlers could be reduced but this will probably break
existing python scripts.

Alternatively the default HTTPRedirectHandler behaviour can be changed
to only allow redirects to HTTP, HTTPS and FTP by checking the scheme
of the location URL (this seems to be a common practise in browsers)

--
assignee: gvanrossum
components: Library (Lib)
hgrepos: 6
messages: 131981
nosy: barry, benjamin.peterson, georg.brandl, gvanrossum
priority: release blocker
severity: normal
stage: patch review
status: open
title: Redirect vulnerability in urllib/urllib2
type: security
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 
3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com