[issue14826] urllib2.urlopen fails to load URL

[Martin Panter]

Martin Panter added the comment:

FWIW urlopen() already handles space characters in the Location target of 
redirects; see HTTPRedirectHandler.redirect_request(). So I think it is 
reasonable to handle space characters in user-supplied URLs also, if it is done 
properly.

--
nosy: +martin.panter
resolution: fixed -> 
stage: resolved -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Ross Lagerwall]

Ross Lagerwall rosslagerw...@gmail.com added the comment:

It looks like this broke the build bots:
http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%202.7/builds/66/steps/test/logs/stdio

--
assignee:  - orsenthil
nosy: +rosslagerwall
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Stephen Thorne]

Stephen Thorne step...@thorne.id.au added the comment:

Here's a followup patch that fixes the trunk build for me.

This will unbreak the builds as well as fixing this bug, but it should be 
investigated why URLopener calls to_bytes() and Request does not. Ideally this 
interface should be consistent.

--
Added file: http://bugs.python.org/file26314/urllib-request.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Éric Araujo]

Éric Araujo mer...@netwok.org added the comment:

I’m not sure urllib should accept invalid (non-escaped) URLs; a higher-level 
application can do so, but for the low-level stdlib module it is more debatable.

--
nosy: +eric.araujo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

It seems to me that toBytes in urllib was introduce to restrict the
allowance of urls which were sent as unicode strings. We wanted urls
to be ascii strings in Python2.

http://mail.python.org/pipermail/python-bugs-list/2000-November/002779.html

And quoting to toBytes / to_bytes is actually the problem here, as
cookielib test cases is sending a unicode character which ascii
encoding fails to operate on. I am thinking that we should arrive at a
solution which brings consistency and fixes any previous mistakes.  In
3.3, I think, rework of to_bytes may also be a good solution, in 2.7
and 3.2, I think stephen's attached patch is in good lines.
Practically, the quote is more important than the failure at toBytes
by sending an unicode url.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

Yeah, I am thinking so as well in that case, the test_cookielib.py
test case may need a change.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset ee1828dc3bf6 by Senthil Kumaran in branch '3.2':
issue 14826 - Address the buildbot failure ( explanation msg164973)
http://hg.python.org/cpython/rev/ee1828dc3bf6

New changeset dc30111a5d7e by Senthil Kumaran in branch 'default':
issue 14826 - Address the buildbot failure quote of url is the required change 
( explanation msg164973)
http://hg.python.org/cpython/rev/dc30111a5d7e

New changeset 224b27a8d9be by Senthil Kumaran in branch '2.7':
revert the changes done in d931a3b64fd6  - buildbot failure.
http://hg.python.org/cpython/rev/224b27a8d9be

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

The last change should settle the buildbots, But I would like to come
back to this issue again tomorrow with focus - 3.3to see if we can
deal with removing to_bytes and then in 2.7 to see if something can
done to test_cookielib.py test case.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Senthil, do you read python-dev? I think this change was prematurate from the 
start (nevermind the fact that you didn't run the test suite before committing).

For example, if you have an URL with a non-ASCII domain name such as 
http://وزارة-الأتصالات.مصر/;, the domain name should IDNA-encoded, not 
%-encoded like the rest.

Furthermore, some people are certainly already quoting their URLs to workaround 
this issue, so fixing it will break their code by double-escaping the URLs. 
You've got to be more careful.

--
nosy: +pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Christian Heimes]

Christian Heimes li...@cheimes.de added the comment:

The docs [1] state that `url should be a string containing a valid URL.` An URL 
with a space ' ' is not a valid URL as the space must be quoted as %20. The 
brackets may also cause problems as they are not valid xs:anyURI chars.

I vote for reverting the chances as they break the API. You could improve the 
docs and emphasize that URLs must be quoted correctly as the module doesn't 
implement browser magic.

[1] 
http://docs.python.org/py3k/library/urllib.request.html#urllib.request.Request

--
nosy: +christian.heimes

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

On Sun, Jul 8, 2012 at 2:30 AM, Antoine Pitrou rep...@bugs.python.org wrote:

 Senthil, do you read python-dev? I think this change was prematurate from the 
 start (nevermind the fact that you didn't run the test suite before 
 committing).

I  thought that the other legacy URLOpen was quoting it correct and
then I wanted to see it can be made consistent.
It did get me thinking that why it was different for so long. I
realize that committing soon was a mistake.

 For example, if you have an URL with a non-ASCII domain name such as 
 http://وزارة-الأتصالات.مصر/;, the domain name should IDNA-encoded, not 
 %-encoded like the rest.

Agreed and understood.

 Furthermore, some people are certainly already quoting their URLs to 
 workaround this issue, so fixing it will break their code by 
 double-escaping the URLs. You've got to be more careful.

Oh. yes, the change may break an already quoted URL. I think, I shall
revert this back.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

On Sun, Jul 8, 2012 at 9:42 AM, Christian Heimes rep...@bugs.python.org wrote:

 I vote for reverting the chances as they break the API. You could improve the 
 docs and emphasize that URLs must be quoted correctly as the module doesn't 
 implement browser magic.

Okay. But I do realize that in 3.3, we may have a FancyURLOpener /
URLOpener 's open method, which is not directly called by the apis,
but they seem to have quote behavior. I guess, I approached this
change as to making them consistent, but realize it is mistake, for
the reasons that you state and Antoine state.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset ebd37273e0fe by Senthil Kumaran in branch '3.2':
revert the changes done for issue14826 - quoting witin Request is not desirable.
http://hg.python.org/cpython/rev/ebd37273e0fe

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset a4bdb637d818 by Senthil Kumaran in branch 'default':
revert the changes done for issue14826 - quoting witin Request is not desirable.
http://hg.python.org/cpython/rev/a4bdb637d818

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Stephen Thorne]

Stephen Thorne step...@thorne.id.au added the comment:

Here is a patch that uses the same quoting logic in 
urllib.request.Request.__init__ as is used by urllib.request.URLopener.open()

--
keywords: +patch
nosy: +jerub
versions: +Python 3.3 -Python 2.7
Added file: http://bugs.python.org/file26311/urllib-quote-14826.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset 01c8d800efd2 by Senthil Kumaran in branch '3.2':
Fix issue14826 - make urllib.request.Request quoted url consistent with 
URLOpener open method.
http://hg.python.org/cpython/rev/01c8d800efd2

New changeset e6bb919b2623 by Senthil Kumaran in branch 'default':
Fix issue14826 - make urllib.request.Request quoted url consistent with 
URLOpener open method.
http://hg.python.org/cpython/rev/e6bb919b2623

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Roundup Robot]

Roundup Robot devn...@psf.upfronthosting.co.za added the comment:

New changeset d931a3b64fd6 by Senthil Kumaran in branch '2.7':
Fix issue14826 - make urllib.request.Request quoted url consistent with 
URLOpener open method.
http://hg.python.org/cpython/rev/d931a3b64fd6

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Changes by Senthil Kumaran sent...@uthcode.com:


--
resolution:  - fixed
stage:  - committed/rejected
status: open - closed
versions: +Python 2.7, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Senthil Kumaran]

Senthil Kumaran sent...@uthcode.com added the comment:

Thanks for the patch, Stephen.

--
nosy: +orsenthil

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Wichert Akkerman]

New submission from Wichert Akkerman wich...@wiggy.net:

There appears to be an odd networking issue with how urllib2 sends HTTP 
requests. Downloading an image from maw.liquifire.com gives an error:

$ python -c 'import urllib2 ; 
urllib2.urlopen(http://maw.liquifire.com/maw?set=image[2302.000.13314 
a]call=url[file:325x445])'
Traceback (most recent call last):
  File string, line 1, in module
  File /usr/lib/python2.7/urllib2.py, line 126, in urlopen
return _opener.open(url, data, timeout)
  File /usr/lib/python2.7/urllib2.py, line 400, in open
response = self._open(req, data)
  File /usr/lib/python2.7/urllib2.py, line 418, in _open
'_open', req)
  File /usr/lib/python2.7/urllib2.py, line 378, in _call_chain
result = func(*args)
  File /usr/lib/python2.7/urllib2.py, line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
  File /usr/lib/python2.7/urllib2.py, line 1180, in do_open
r = h.getresponse(buffering=True)
  File /usr/lib/python2.7/httplib.py, line 1030, in getresponse
response.begin()
  File /usr/lib/python2.7/httplib.py, line 407, in begin
version, status, reason = self._read_status()
  File /usr/lib/python2.7/httplib.py, line 365, in _read_status
line = self.fp.readline()
  File /usr/lib/python2.7/socket.py, line 447, in readline
data = self._sock.recv(self._rbufsize)
socket.error: [Errno 104] Connection reset by peer

Downloading the same image using wget works fine:

$ wget 'http://maw.liquifire.com/maw?set=image[2302.000.13314 
a]call=url[file:325x445]' 
--2012-05-16 10:53:27--  
http://maw.liquifire.com/maw?set=image[2302.000.13314%20a]call=url[file:325x445]
Resolving maw.liquifire.com (maw.liquifire.com)... 184.169.78.6
Connecting to maw.liquifire.com (maw.liquifire.com)|184.169.78.6|:80... 
connected.
HTTP request sent, awaiting response... 200 OK
Length: 11393 (11K) [image/jpeg]
Saving to: `maw?set=image[2302.000.13314 a]call=url[file:325x445]'

100%[==] 11,393  --.-K/s   in 0.003s  

2012-05-16 10:53:27 (3.49 MB/s) - `maw?set=image[2302.000.13314 
a]call=url[file:325x445]' saved [11393/11393]

--
components: Library (Lib)
messages: 160811
nosy: wichert
priority: normal
severity: normal
status: open
title: urllib2.urlopen fails to load URL
versions: Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14826] urllib2.urlopen fails to load URL

[Anthony Long]

Anthony Long antl...@gmail.com added the comment:

http://maw.liquifire.com/maw?set=image[2302.000.13314%20a]call=url[file:325x445]
 works properly. Notice the %20 instead of ' '

--
nosy: +antlong

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14826
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com