[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Maxim Yegorushkin added the comment: I am still having this issue with Python-2.7.2 (which includes the patch) and openssl-devel-1.0.0e-1.fc14.x86_64: File "./download.py", line 195, in download_file ftp.retrbinary("retr " + filename, lambda data: file.write(data)) File "/usr/local/ots

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Ok, I've backported the shutdown fixes to 2.6 (r80419) and 3.1 (r80420). Thanks again everyone. -- status: pending -> closed versions: +Python 2.6, Python 3.1 ___ Python tracker _

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Patch committed to r80392 (trunk) and r80394 (py3k). Am pondering a partial backport of the SSL shutdown fixes to 2.6 and 3.1 (test_ftplib doesn't need any backport since it doesn't support TLS in these branches). -- assignee: -> pitrou resolution:

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > With regards to SSL_peek() blocking, you'd need to explain yourself > better on that one. The patch has been tested with the test cases > from Python SVN enough to be happy they run ok. IIRC it worked with OpenSSL 1.0.0 but failed with 0.9.8k. I could test i

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Darryl Miles added the comment: With regards to create test cases for certain situations, sure this would be possible but not with "pure python" since your APIs deny/inhibit the particular things required to force a situation for a test case. With regards to SSL_peek() blocking, you'd need to

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > If everyone agrees on "error: [Errno 0] Error" being a legitimate alias > for a "connection closed event" condition then I'd say the test server > looks good, even if I think that expecting a ssl.SSLError derived > exception would have made more sense, but I

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: If everyone agrees on "error: [Errno 0] Error" being a legitimate alias for a "connection closed event" condition then I'd say the test server looks good, even if I think that expecting a ssl.SSLError derived exception would have made more sense, but I und

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: After thinking a bit about it, we're in quasi-feature freeze now for 2.x and it doesn't sound reasonable to add options to unwrap(); we would need to write tests to exercise these options and their behaviour. Also, they aren't strictly needed to fix the issue

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Darryl Miles added the comment: To explain why you need 2 modes, a client/server would expect to do the following pseudo actions for maximum efficiency: set_socket_timeout(600_SECONDS) # or useful default send_data_over_ssl("QUIT\r\n") shutdown(SSL_SHUTDOWN_MODE_SENT) flush_data_down_to_socke

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: To just run the test_ftplib tests, you can use: $ ./python -m test.regrtest -v -uall test_ftplib (Other tests involving SSL sockets are test_ssl, test_smtpnet, test_imaplib and test_poplib) > 1) Some thing that concern me, the "unwrap()" philosophy looks to b

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Darryl Miles added the comment: I am unable to get "make test" to run from an unpatched version in SVN (details below of make output). Please find attached an updated patch for your consideration (and testing, as I can't test it due to 'make test' SIGSEGV on CentOS 5.4 i386). Patch Notes:

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Darryl Miles : Removed file: http://bugs.python.org/file16838/python_ssl.c.txt ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Darryl Miles added the comment: I've updated my attachment to the bug, if you read the old one please re-read the updated version (since some points in there were not accurate). With regards to the OpenSSL error return -1/ERROR_SYSCALL with errno==0 being observed, I shall respond into the Op

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Darryl Miles added the comment: In order to build Python with a specific version of OpenSSL followed the CYGWIN instructions and edited Modules/Setup to make it read (note - I added "-L$(SSL)" into the linker options too, since by default on CentOS 5.4 i386 OpenSSL build in static library mod

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: My analysis of SSL_shutdown() was missing something: timeout sockets are really non-blocking sockets in disguise. I guess it's never too late to notice. OpenSSL doesn't know about the timeout, so we have to do it ourselves. Ironically, the infrastructure is a

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Ok, I raised the "errno 0" issue on the openssl mailing-list, and I had a nice response by Darryl Miles here: http://www.mail-archive.com/openssl-us...@openssl.org/msg60710.html The bottom line is that any "socket error" return from unwrap(), that is any "SSL

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: New patch without the quit() method change. -- Added file: http://bugs.python.org/file16781/newssl3.patch ___ Python tracker ___ ___

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > - ftplib.FTP_TLS.quit: I wouldn't call unwrap() here as RFC-4217 does > that only when dealing with CCC command which is not implemented by > ftplib.py. Ok. > - I'd be for calling _do_ssl_shutdown() from handle_close() instead of > from close() (reasons expl

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: > I don't understand your reasoning. These are separate connections, > closing one shouldn't affect the other. I meant another thing. I was talking about the fact that the test server attempts to shutdown() both control and data connections while the clien

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > As for our test server we are currently shutting down the SSL layer for > both connections (control and data) so this might be the cause of EPIPE. I don't understand your reasoning. These are separate connections, closing one shouldn't affect the other. > A

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: > By the way, this "broken pipe" error could be due to the fact that the > FTP_TLS client never tries to call unwrap() on its SSL socket. Perhaps > the close() method should be overriden? ftplib.FTP_TLS class already calls unwrap() but only when closing a "s

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Here is a new patch. It doesn't change handle_error() but instead ignores abruptly closed sockets when doing the SSL shutdown. The underlying problem seems to be an asyncore limitation: close() can be called after an error such as EPIPE or ECONNABORTED, but a

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: By the way, this "broken pipe" error could be due to the fact that the FTP_TLS client never tries to call unwrap() on its SSL socket. Perhaps the close() method should be overriden? -- ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > What happens if you move the SSL shutdown() call in handle_close() > method and leave handle_error() unchanged? I get occasional crashes such as the following: Exception in thread Thread-60: Traceback (most recent call last): File "/home/antoine/cpython/ne

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: def handle_error(self): -raise +# Ignore errors while closing, because the remote end could have +# abruptly shut down the TCP connection while we are still +# waiting for SSL shutdown to finish. +if

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Antoine Pitrou : Added file: http://bugs.python.org/file16778/newssl.patch ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Ok, here is a patch fixing all test_ftplib failures (tested with openssl 0.9.8k and 1.0.0). There are a couple of fixes here: - a workaround for the strange SSL_shutdown() error return, also witnessed by LightHTTPd users (see URL in comments) - an auto-retry f

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: I've tried to build against OpenSSL 1.0.0, and I get further failures in test_ssl. Since I don't know whether they are related, I've created a separate issue for them: issue8322 -- ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: As suggested in this thread: http://mirt.net/pipermail/stunnel-users/2005-July/000661.html ...I made the following change to the Makefile: - LIBS= -lpthread -ldl -lutil + LIBS= -lpthread -ldl -lutil -lz That made the error change as follows:

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > No I haven't, but I tried just now and I get the same error. I think "inflate" is a function exported by the zlib. Perhaps you can add "-lz" to the linking flags. (Googling hints that OpenSSL can depend on the zlib if compression is enabled) --

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: No I haven't, but I tried just now and I get the same error. -- ___ Python tracker ___ ___ Python

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > Since it's not clear to me where exactly this comes from, whether it's > from the Python C binding or OpenSSL itself, I tried to put some > debugging printf() calls in Modules/_ssl.c, but it seems that after > installing OpenSSL 0.9.8m I'm no longer able to c

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Florent Xicluna : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: Yes, it's the _sslobj.shutdow() call: File "test_ftplib.py", line 332, in handle_close self.socket = self.socket.unwrap() File "/usr/local/lib/python2.7/ssl.py", line 258, in unwrap s = self._sslobj.shutdown() error: [Errno 0] Error Since it's n

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Benjamin Peterson : -- priority: release blocker -> critical ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Giampaolo, which call exactly triggers the "error: [Errno 0] Error"? It doesn't seem it can be _sslobj.shutdown itself. -- ___ Python tracker __

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: I don't think this should be a release blocker for the beta, although I agree it should be one for the final release (and RCs). By the way, it doesn't affect maintenance branches. -- versions: -Python 2.6, Python 3.1 _

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Benjamin Peterson : -- priority: high -> release blocker ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: Ok, I've finally been able to reproduce the issue by installing OpenSSL 0.9.8m on my Ubuntu box. By doing some debugging I'm starting to think that maybe there's something wrong with the shutdown() method implementation itself because no matter what, soone

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: Giampaolo, IMHO you need 0.9.8m to reproduce the issue. I installed it from the Debian testing repo (I'm on debian). ~ $ apt-cache policy openssl openssl: Installed : 0.9.8m-2 Candidate : 0.9.8m-2 Table de version : *** 0.9.8m-2 0 400 ftp://ftp.l

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: Patch ftplib/test_ftplib updated. It is as before (0.9.8k): SSL_ERROR_WANT_READ is ignored on connection unwrap(). -- Added file: http://bugs.python.org/file16648/issue8108_ftplib_ssl_098m_v2.diff ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: Florent can you give me a clue on how to make python use OpenSSL 0.9.8k on Ubuntu? Despite I compiled and installed 0.9.8k from sources the system version keeps being 0.9.8g and I guess that Python is using that one. Do I have to recompile Python by using a

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Florent Xicluna : Removed file: http://bugs.python.org/file16641/issue8108_ftplib_ssl_098m.diff ___ Python tracker ___ ___ Python-bu

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: You're right, there's no need to catch the socket error in "ftplib" module. Here is the result of the "debugging" patch in the console: $ ./python -m test.regrtest -uall test_ftplib test_ftplib success success success success success success success success 2

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Florent, you shouldn't need to silence socket.error at this level. I think what needs to happen is that, when unwrap() fails in SSLConnection.close(), handle_read() and handle_write() retry the unwrap() call. That's what I gather from the explanation of the n

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: I don't know if it is better to fix it in the ftplib or the ssl module. The previous patch fixes it at the SSL level (stdlib only). This patch implements the fix at the ftplib level (stdlib + test). -- Added file: http://bugs.python.org/file16641/issu

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: @Antoine Pitrou: ok, I think I'll have to be able to replicate the error in order to try to fix this. I'm gonna try on FreeBSD and another Linux box later today. -- ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: This patch solves the problem with 0.9.8m on my Debian workstation. Actually there's 4 kind of errors on shutdown(): - SSL_ERROR_WANT_READ - SSL_ERROR_WANT_WRITE - socket.error: errno.PIPE (32) - socket.error: 0 Since the command is "shutdown()", the patc

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Antoine Pitrou : -- stage: test needed -> needs patch versions: +Python 2.6, Python 3.1 ___ Python tracker ___ ___ Python-bu

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Giampaolo, I tried committing your patch but the buildbot still fails, with the following errors: test_auth_issued_twice (test.test_ftplib.TestTLS_FTPClass) ... ok test_auth_ssl (test.test_ftplib.TestTLS_FTPClass) ... ok test_control_connection (test.test_ftpl

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: > The patch isn't exactly correct: it should ideally retry the unwrap() > call later, rather than simply ignore the error. But since it's just > used for testing, it looks sufficient. Actually unwrap() should already be called. What the patch does is retur

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: Ok, I finally agree that test_ftplib should be patched. The patch isn't exactly correct: it should ideally retry the unwrap() call later, rather than simply ignore the error. But since it's just used for testing, it looks sufficient. (we probably lack a highe

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: >> Depends on what we want. It just suppresses information that's now >> available. What we'd really like is for the caller to recognize that >> close() can fail, and should be re-tried if it does. That requires >> that we signal the error back up and out of th

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: > Maybe because only the FTP test uses an SSL socket in non blocking mode. No, the SSL unit tests also do this. I think Giampaolo is right, what we're seeing is bad error handling in the FTP test code. -- ___ Python

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > Depends on what we want. It just suppresses information that's now > available. What we'd really like is for the caller to recognize that > close() can fail, and should be re-tried if it does. That requires > that we signal the error back up and out of the

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: > close() can fail Not close, unwrap() (shutdown()) ;-) > Again, why does this failure only show up in the FTP test? Maybe because only the FTP test uses an SSL socket in non blocking mode. -- ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: > AFAICT, his proposed "quick fix" snippet should be good enough for us. Depends on what we want. It just suppresses information that's now available. What we'd really like is for the caller to recognize that close() can fail, and should be re-tried if it doe

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscrib

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Antoine Pitrou added the comment: > I posted on openssl-users mailing list. The author of patch (change > SSL_shutdown() behaviour for non blocking socket) replied! > http://marc.info/?l=openssl-users&m=126838806919896&w=2 Interesting read, thank you. AFAICT, his proposed "quick fix" snippet sh

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: I posted on openssl-users mailing list. The author of patch (change SSL_shutdown() behaviour for non blocking socket) replied! http://marc.info/?l=openssl-users&m=126838806919896&w=2 -- ___ Python tracker

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: > I'd really love to understand what the state of the TCP connection > is here. I'm presuming that it's still open, because otherwise you'd > get a different error from OpenSSL. So what's the error that's > triggering this call? Is it simply that the sen

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: And it would be interesting to know why all the SSL module tests don't fail in the same way. -- ___ Python tracker ___ __

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: I think Giampaolo is right about this not being an ssl.py issue. Could the exception handling in ftp.py perhaps also be made more sophisticated? I'd really love to understand what the state of the TCP connection is here. I'm presuming that it's still open, b

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: Looking at this further, what we're seeing is the trace of an ineffective attempt to handle an exception presumably raised from the FTP code. Can we see that exception? What's the actual state of the TCP connection at this point? -- _

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Giampaolo Rodola' added the comment: I'm not sure this should be handled in ssl.py. ssl.py correctly raises ERROR_WANT_READ/WRITE if the shutdown operation didn't complete and that's ok. It should be up to the upper application (in our case asyncore) to deal with that. Moreover, the patch tri

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: What's happening is that the new state returns from SSL_shutdown() are saying, "the shutdown you asked for didn't happen this time, but call me again when you get a chance. And here is a hint about why it didn't happen, so that if you are sophisticated enough,

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Bill Janssen added the comment: Victor, I'll take a look. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: ht

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: There are different kind of bugs, but all of them are related to SSL_Shutdown(). It depends on the socket timeout: - blocking socket: nothing special (no change from previous OpenSSL version) - timeout or non blocking socket: if SSL_shutdown() returns -1 and

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Antoine Pitrou : -- nosy: +janssen ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: More informations: http://www.openssl.org/docs/ssl/SSL_shutdown.html http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=1891 Extract of the manual page: If the underlying BIO is non-blocking, SSL_shutdown() will also return when the underly

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: The patch changing SSL_Shutdown() behaviour on non blocking socket was written in 2006 by Darryl Miles and it will commited into 0.9.8m. The original patch: http://marc.info/?l=openssl-dev&m=115153998821797&w=2 In our testcase, SSL_Shutdown() fails with SSL_ER

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

STINNER Victor added the comment: The problem comes from SSL_Shutdown() on a non blocking connection. See: http://www.mail-archive.com/openssl-...@openssl.org/msg24097.html -- ___ Python tracker ___

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: OpenSSL changelog (scroll to 0.9.8m): http://www.openssl.org/news/changelog.html -- ___ Python tracker ___

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Florent Xicluna added the comment: It occurs only for trunk and 3.x, where the "ftp over ssl" is implemented. See #2054. And the Debian alpha buildbots have the same library 0.9.8m: http://www.python.org/dev/buildbot/all/builders/alpha%20Debian%20trunk http://www.python.org/dev/buildbot/all/bu

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

Changes by Florent Xicluna : -- nosy: +giampaolo.rodola, haypo, pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubsc

[issue8108] test_ftplib fails with OpenSSL 0.9.8m

New submission from Florent Xicluna : There's such failure on trunk and py3k when openssl 0.9.8m is installed. (Debian platform) No failure with 0.9.8k. test_ftplib Exception in thread Thread-40: Traceback (most recent call last): File "./Lib/threading.py", line 530, in __bootstrap_inner s