[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: Terry: the slowdown might be IPV6 handling, which was a bit broken on windows. In any case, I think the slowdown should be treated as a separate issue if you care about it, so I'm going to close this. (I think it would be great if someone would do a review of the test suite looking for things to speed up...maybe we could add that to the devguide as a what-can-I-work-on project...) -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Roundup Robot added the comment: New changeset 3ffa43e8ab47 by Terry Jan Reedy in branch '2.7': Issue #20155: Fix non-buildbot test failure on Windows. Patch by Claudiu Popa, https://hg.python.org/cpython/rev/3ffa43e8ab47 New changeset 5c0f17063fb8 by Terry Jan Reedy in branch '3.4': Issue #20155: Fix non-buildbot test failure on Windows. Patch by Claudiu Popa, https://hg.python.org/cpython/rev/5c0f17063fb8 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment: Patch fixes failures for me. Let's hope that change does not fail on other systems or buildbots (starting at 21:17 bb time). I will check bbs later. There seems to be a 2.7 speed issue which looks like a hang until running with -v. On my machine, 2.7 runs 34 tests 40 seconds (consistently), with test_get taking 6 seconds. 3.4 runs 45 tests in under 3 seconds, a 15x speedup. I attached difflib.Differ output. David, can you see anything responsible for the slowdown. Should this be ignored? -- stage: needs patch - resolved ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment:
I copied the example profile code at the top and bottom of test_get and nearly
all (7.155 / 7.210) of the time is in _socket.socket.connect, which takes .5
sec per call.
147.1550.5117.1550.511 {method 'connect' of _socket.socket'
objects}
I am guessing that socket, and thence _socket, is imported in HTTPServer. I
tested whether this diff in class TestServerThread is responsible
- self.server = HTTPServer(('', 0), self.request_handler)
+ self.server = HTTPServer(('localhost', 0), self.request_handler)
but it is not. Anyone else have other ideas?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20155
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment:
If Python itself works correctly on a machine, it is a bug for a test to say
that is it not, by failing. Should we change
-response = self.request('/', method='get')
to
+response = self.request('/', method='gets')
or
+response = self.request('/', method='custom')
or conditionally, on type(status), change the test from assertEqual to (after
substitutions) assertIn(response.status, (200,501))? David, do you have a
preference? or should I flip a coin?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20155
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: It should be 'custom', since that will actually test case sensitivity, which 'gets' would not. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: If deactivating the AV/firewall doesn't change the behavior, it is presumably a different problem. But let's see what Claudiu has to say, since he was able to reproduce and then circumvent the original problem. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Claudiu Popa added the comment:
Terry is right, this patch doesn't completely work. But with this fix the
problem is solved:
diff -r 394e6bda5a70 Lib/test/test_httpservers.py
--- a/Lib/test/test_httpservers.py Sun Jun 29 15:56:21 2014 +0300
+++ b/Lib/test/test_httpservers.py Sun Jun 29 19:27:16 2014 +0300
@@ -306,7 +306,7 @@
response = self.request('/', method='FOO')
self.check_status_and_reason(response, 501)
# requests must be case sensitive,so this should fail too
-response = self.request('/', method='get')
+response = self.request('/', method='gets')
self.check_status_and_reason(response, 501)
response = self.request('/', method='GETs')
self.check_status_and_reason(response, 501)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue20155
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Claudiu Popa added the comment: Although it is incorrect, because the test specifically tests case sensitivity. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: What if you use 'custom' instead? -- resolution: fixed - stage: resolved - needs patch status: closed - open ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Claudiu Popa added the comment: It works with 'custom'. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment: Both 'gets' and 'custom' work on my machine. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Jeff Allen added the comment: Disabling the AV/firewall did not stop the symptoms when I was investigating originally. In order to get the unmodified test to pass, I had to stop the BFE (base filtering engine), which I think may have been given new rules or behaviours as a result of installing the AV solution ... or maybe it was a Windows upgrade that did it. I did wonder if this might be a moving target, as the test deliberately includes server abuse, while the products want to stop that. If I try test_httpservers.py as amended (http://hg.python.org/cpython/file/ffdd2d0b0049/Lib/test/test_httpservers.py) on my machine with CPython 3.4.1, I do not get the error Terry reports. (test_urlquote_decoding_in_cgi_check fails but it should.) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment: After updating and rebuilding (32 bit VC express, on 64 bit Win 7), I get the same error as in msg210926, with or without -uall, even after turning my antivirus off. File F:\Python\dev\4\py34\lib\test\test_httpservers.py, line 310, in test_invalid_requests self.check_status_and_reason(response, 501) File F:\Python\dev\4\py34\lib\test\test_httpservers.py, line 265, in check_status_and_reason self.assertEqual(response.status, status) AssertionError: 200 != 501 Is this a failure of the patch or a different issue? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Claudiu Popa added the comment: It would be nice if this could be committed. It's cumbersome to always have to deactivate the AV solution when running the tests on Windows, in order to avoid a failure of test_httpservers. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Roundup Robot added the comment: New changeset ffdd2d0b0049 by R David Murray in branch '3.4': #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/ffdd2d0b0049 New changeset e67ad57eed26 by R David Murray in branch 'default': merge: #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/e67ad57eed26 New changeset b0526da56c54 by R David Murray in branch '2.7': #20155: use fake HTTP method names so windows doesn't hang the tests. http://hg.python.org/cpython/rev/b0526da56c54 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: Done. Thanks Jeff. -- resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Jeff Allen added the comment: I worked out that the essence of the test is to insert an extra \n at the end of a GET request line. The request is syntactically invalid for HTTP. The \n\r\n appears like two blank lines, implying no headers, but the headers then follow where no data should be. The server is supposed to respond with status 501, because it does not, in fact, define a GET operation. To find the replacement test plausible you have to accept that, with a server that doesn't define GET, the verb may as well be XYZBOGUS. Since the security filter doesn't understand that verb either (unlike GET), it doesn't interfere in the test. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: Ah, OK. I was thinking that BaseHTTPRequestHandler defined GET, but I see that it doesn't. So yes, this patch looks fine to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
R. David Murray added the comment: The second test looks correct to me. I don't understand what tset_request_line_trimming is testing (I haven't investigated), so I don't know if that one is correct, but it is certainly plausible. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Claudiu.Popa added the comment: Terry, I had the same problem with that failing test_httpservers.test_invalid_request using the latest build on Windows. After debugging, I found out that the problem was caused by my antivirus solution. Its http scanning engine caught malformed http requests, like the one in the test (gEt / hTTP/1.0\r\nhost: 127.0.0.1:50340\r\nAccept-Encoding: identity\r\n\r\n; for instance) and it modified the first line by uppercasing it, thus making the perfect condition for a failing test. -- nosy: +Claudiu.Popa ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Jeff Allen added the comment: Thanks for adding to the evidence here. As discussed above, disabling the security product (which is Bitdefender) on my PC didn't stop the problem for me, and I'm reluctant to uninstall. I narrowed it to the Windows Base Filtering Engine, but perhaps the behaviour of the BFE is extended by installing BD. If so, you could say this is not a Python problem, it is caused by BD normalising the HTTP. Or BD could say it is caused by expecting a defined result from abnormal HTTP. I took the view it were best fixed at our end. I found I could test the same thing (AFAICT), but modify the tests so they don't get interfered with. http://bugs.jython.org/issue2109 http://hg.python.org/jython/rev/6441fcfd940b Would a patch made from this be applicable to CPython? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment: Except where specifically indicated otherwise, the Python test suite should be the same, and correct, for all implementations. If that change is correct for Jython, it should be correct for CPython. David, Ezio, or Senthil: does the change in the Jython patch look correct to any of you, enough that we should apply it? -- components: +Tests nosy: +ezio.melotti, orsenthil, r.david.murray stage: - patch review type: - behavior ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Jeff Allen added the comment: Actual patch for your convenience. I'm not set up to build CPython from source, so I've tested this with my installed CPython 2.7.6, and it's clean. [As for keeping the tests in sync, yes that's our aim. Jython's Lib contains only the customised versions, and everything else comes from a copy of CPython's in lib-python/2.7. I'm always looking for a chance to delete one (i.e. use the common file).] -- keywords: +patch nosy: +fwierzbicki Added file: http://bugs.python.org/file34202/issue20155_py.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
Terry J. Reedy added the comment: Installed 3.4.0c1 on Windows: test test_httpservers failed -- Traceback (most recent call last): File C:\Programs\Python34.32\lib\test\test_httpservers.py, line 309, in test_invalid_requests self.check_status_and_reason(response, 501) File C:\Programs\Python34.32\lib\test\test_httpservers.py, line 264, in check_status_and_reason self.assertEqual(response.status, status) AssertionError: 200 != 501 I know very little about http server stuff. -- nosy: +terry.reedy versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20155] Regression test test_httpservers fails, hangs on Windows
New submission from Jeff Allen: When I run: start python -m test.test_httpservers test_request_line_trimming reports ERROR, and the test hangs at test_version_none. If I run a copy of the test in which the latter test is skipped with @unittest.skipIf(sys.platform == win32, ...), the error report is: == ERROR: test_request_line_trimming (__main__.BaseHTTPServerTestCase) -- Traceback (most recent call last): File test_httpservers.py, line 122, in test_request_line_trimming res = self.con.getresponse() File C:\Python33\lib\http\client.py, line 1131, in getresponse response.begin() File C:\Python33\lib\http\client.py, line 354, in begin version, status, reason = self._read_status() File C:\Python33\lib\http\client.py, line 316, in _read_status line = str(self.fp.readline(_MAXLINE + 1), iso-8859-1) File C:\Python33\lib\socket.py, line 297, in readinto return self._sock.recv_into(b) ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host This is essentially the same for Python 2.7.6 and for the current development tip of Jython 2.7b1+, which is actually where the problem first manifested. My machine is running 64-bit Windows 7 SP1, recently re-installed to a new, empty disk. Careful testing, elaborating the failing tests, shows that what is sent in PUT and GET operations is not quite what is received. Something tampers with the connection between the client and the server. (Identical traffic where the verbs are not PUT and GET arrives as sent.) Something fiddles with the forward message, for example correcting the spurious \n in test_request_line_trimming to a full \r\n, and holding back the payload of a PUT even when it was in the first packet. On the reverse path, it appears to act on the error response itself by closing the connection, without passing it to the client. Disabling the firewall (Windows Firewall and a commercial one), with the network cable unplugged, makes no difference. Nor does stopping anti-virus, anti-phishing, parental controls, etc.. However, stopping the Windows Basic Filtering Engine (BFE), makes the regression test run without error. Stopping the BFE takes out several dependent services, including Windows Firewall, but it seems likely the BFE itself is the culprit. Although the cause lies in the platform, not in Python, it seems to me still an issue for Python that the tests fail on a common platform practically out of the box. I'll work on this in the context of the Jython test and report back here. -- components: Library (Lib), Windows messages: 207497 nosy: jeff.allen priority: normal severity: normal status: open title: Regression test test_httpservers fails, hangs on Windows versions: Python 2.7, Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20155 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
