[issue10116] Sporadic failures in test_urllibnet

[Irit Katriel]

Irit Katriel  added the comment:

The www.example.com url was replaced by a local one in Issue36019, so this 
seems to complete what remained to do here.

--
nosy: +iritkatriel
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Martin Panter]

Martin Panter added the comment:

It looks like Antoine’s second failure (test_getcode) should be fixed now that 
transient_internet() is used (Issue 12138).

Looking at Antoine’s first failure, it will always be possible for it to fail 
depending on the conditions of the buildbot, internet, and the www.example.com 
server:

def setUp(self):
socket.setdefaulttimeout(self.TIMEOUT)

def testURLread(self):
f = _open_with_retry(urllib.urlopen, "http://www.example.com/;)
x = f.read()  # Timed out!

I would suggest changing this to connect to a server running on localhost in a 
background thread, and moving it into test_urllib.py because it would no longer 
need the “network” resource. That should eliminate enough causes of a timeout, 
the remaining one being a very slow buildbot, which I don’t think is worth 
worrying about.

Also see revision aa46bdbe641d in Python 3, where the read() call was shifted 
under the transient_internet() handler, and seems to defeat the purpose of the 
test.

--
nosy: +martin.panter
versions: +Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Berker Peksag]

Changes by Berker Peksag berker.pek...@gmail.com:


--
nosy: +berker.peksag

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Mark Lawrence]

Mark Lawrence added the comment:

Is it safe to assume that these failures have been resolved?

--
nosy: +BreamoreBoy
versions: +Python 3.4, Python 3.5 -Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

There are still sporadic failures:

http://www.python.org/dev/buildbot/all/builders/AMD64%20Leopard%203.x/builds/684/steps/test/logs/stdio

Of course, it seems to be because of stupid urllib wrapping the socket.error 
inside an IOError:

Traceback (most recent call last):
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/test/test_urllibnet.py,
 line 97, in test_getcode
open_url = urllib.request.FancyURLopener().open(URL)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/urllib/request.py,
 line 1516, in open
raise IOError('socket error', msg).with_traceback(sys.exc_info()[2])
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/urllib/request.py,
 line 1512, in open
return getattr(self, name)(url)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/urllib/request.py,
 line 1684, in open_http
return self._open_generic_http(http.client.HTTPConnection, url, data)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/urllib/request.py,
 line 1664, in _open_generic_http
http_conn.request(GET, selector, headers=headers)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 962, in request
self._send_request(method, url, body, headers)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 1000, in _send_request
self.endheaders(body)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 958, in endheaders
self._send_output(message_body)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 803, in _send_output
self.send(msg)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 741, in send
self.connect()
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/http/client.py,
 line 719, in connect
self.timeout, self.source_address)
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/socket.py, 
line 398, in create_connection
raise err
  File 
/Users/pythonbuildbot/buildarea/3.x.hansen-osx-x86-2/build/Lib/socket.py, 
line 389, in create_connection
sock.connect(sa)
IOError: [Errno socket error] [Errno 60] Operation timed out

--
resolution: fixed - 
stage: patch review - 
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

On Sat, Nov 13, 2010 at 01:17:47PM +, Antoine Pitrou wrote:
 There are still sporadic failures such as:

I have made a small change for this in r87260.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Senthil Kumaran]

Changes by Senthil Kumaran orsent...@gmail.com:


--
status: pending - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

There are still sporadic failures such as:

==
ERROR: testURLread (test.test_urllibnet.URLTimeoutTest)
--
Traceback (most recent call last):
  File 
/Users/pythonbuildbot/buildarea/2.7.hansen-osx-x86/build/Lib/test/test_urllibnet.py,
 line 40, in testURLread
x = f.read()
  File 
/Users/pythonbuildbot/buildarea/2.7.hansen-osx-x86/build/Lib/socket.py, line 
351, in read
data = self._sock.recv(rbufsize)
timeout: timed out

which seems to show that not everything has been properly converted.

(http://www.python.org/dev/buildbot/all/builders/AMD64%20Snow%20Leopard%
202.7/builds/156/steps/test/logs/stdio )

Le samedi 13 novembre 2010 à 12:34 +, Senthil Kumaran a écrit :
 Changes by Senthil Kumaran orsent...@gmail.com:
 
 
 --
 status: pending - closed
 
 ___
 Python tracker rep...@bugs.python.org
 http://bugs.python.org/issue10116
 ___

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

Stephen Hansen me+pyt...@ixokai.io added the comment:

New patch, sans trailing whitespace. Ahem.

--
Added file: http://bugs.python.org/file19398/issue10116-nowhitespace.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

Changes by Stephen Hansen me+pyt...@ixokai.io:


Removed file: http://bugs.python.org/file19390/issue10116.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
stage: committed/rejected - patch review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Georg Brandl]

Georg Brandl ge...@python.org added the comment:

Let's try with the patch. r85893.

(I'm afraid the pre-commit hook still rejected the file after applying the 
patch.)

--
nosy: +georg.brandl
resolution:  - fixed
status: open - pending

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

Stephen Hansen me+pyt...@ixokai.io added the comment:

The attached patch wraps all the calls to the internet in 
support.transient_internet; I ran it against 3.x and it passed, and then I ran 
it for quite awhile with the -F option, and encountered one event that I 
believe would previously had resulted in one of these sporadic failures, and it 
resulted in a skipped 'resource not available' message. 

I left in the previous 'retry' code, just by virtue of changing as little as 
possible. I can adjust if its desired.

I believe that transient_internet won't capture EBADF: so if that particular 
sporadic failure happens again, I'll post up a new issue about it.

--
keywords: +patch
Added file: http://bugs.python.org/file19390/issue10116.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

ixokai, A change made as part of issue10119 should have resolved this issue 
too. Please let me know if this can be closed.

--
assignee:  - orsenthil
resolution:  - fixed
stage:  - committed/rejected
type:  - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

Stephen Hansen me+pyt...@ixokai.io added the comment:

I'll run the test in -F mode for a few hours to see if it comes up or not: but 
its hard for me to say one way or the other if anything has fixed or not fixed 
it, as the failure only came up every once in awhile. But I'll look.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

Stephen Hansen me+pyt...@ixokai.io added the comment:

Okay, at -r85630 on branches/py3k, I ran:

./python.exe -m test.regrtest -uall -F test_urllibnet

And after 158 retries, got the same error I had before:

test test_urllibnet failed -- Traceback (most recent call last):
  File /Users/pythonbuildbot/test/build/Lib/urllib/request.py, line 1504, in 
open
return getattr(self, name)(url)
  File /Users/pythonbuildbot/test/build/Lib/urllib/request.py, line 1676, in 
open_http
return self._open_generic_http(http.client.HTTPConnection, url, data)
  File /Users/pythonbuildbot/test/build/Lib/urllib/request.py, line 1659, in 
_open_generic_http
response = http_conn.getresponse()
  File /Users/pythonbuildbot/test/build/Lib/http/client.py, line 1027, in 
getresponse
response.begin()
  File /Users/pythonbuildbot/test/build/Lib/http/client.py, line 347, in begin
version, status, reason = self._read_status()
  File /Users/pythonbuildbot/test/build/Lib/http/client.py, line 303, in 
_read_status
line = str(self.fp.readline(), iso-8859-1)
  File /Users/pythonbuildbot/test/build/Lib/socket.py, line 267, in readinto
return self._sock.recv_into(b)
socket.error: [Errno 9] Bad file descriptor

--
resolution: fixed - 

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Stephen Hansen]

New submission from Stephen Hansen me+pyt...@ixokai.io:

Ever since running the snow leopard buildslave, I've had sporadic failures in 
test_urllibnet. At first I thought it was just a net glitch on my machine or 
something, as immediately re-running the tests made it go away: but this most 
recent one:

http://www.python.org/dev//buildbot/builders/x86%20Snow%20Leopard%203.1/builds/20/steps/test/logs/stdio

happened while I was very much monitoring and using the network on the machine 
for other purposes, and everything was fine in general.

So, I went and looked into test_urllibnet to try to figure out why, and I 
notice that some of the tests use code to retry on IOErrors, and some don't-- 
and this test that failed in particular is one that doesn't. 

So: anyone have a better idea of what's going wrong, or is it just that hey, 
the active network tests are a bit flaky and all should use _open_with_retry 
instead of just some as is the case now?

[If the latter, I can do a patch]

FWIW, I've only seen this on the 3.1 and 3.x buildslaves, but have seen it on 
both of those more then once. But I don't know that its a 3.x specific issue: 
those builds get run more often then the 2.7 one, so have more chances to run 
into a sporadic issue.

--
components: Tests
messages: 118772
nosy: ixokai
priority: normal
severity: normal
status: open
title: Sporadic failures in test_urllibnet
versions: Python 3.1, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

Rather than the hand-made _open_with_retry, I think it would be better to use 
support.transient_internet() (it's already used in other tests). Retrying is 
not very helpful if the other end is down.

Also, EBADF (bad file descriptor) looks like a bug. httplib should not use 
closed/non-existent file descriptors.

--
nosy: +orsenthil, pitrou

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10116] Sporadic failures in test_urllibnet

[Senthil Kumaran]

Senthil Kumaran orsent...@gmail.com added the comment:

Issue10119 is related too and both I guess, is boiling down to httplib either 
not properly using an open socket or closing it prematurely.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10116
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com