[issue1469] SSL tests leak memory
Bill Janssen added the comment: The server isn't handling the close event properly. I'll fix that. On Dec 13, 2007 9:06 PM, Guido van Rossum [EMAIL PROTECTED] wrote: Guido van Rossum added the comment: I spoke too soon. In a debug build, this hangs forever during the second iteration: ./python.exe Lib/test/regrtest.py -uall -R1:1 test_ssl Adding -v, it looks like two iterations are carried out perfectly (one must be a trial run, one the warm-up run), but the third run goes beserk; the output ends like this: testAsyncoreServer (test.test_ssl.ThreadedTests) ... server: read b'over\n' from client server: closed connection ssl.SSLSocket object, fd=6, family=2, type=1, proto=0 server: read b'' from client server: closed connection ssl.SSLSocket object, fd=10, family=2, type=1, proto=0 server: read b'' from client server: closed connection ssl.SSLSocket object, fd=10, family=2, type=1, proto=0 . . (the last two lines repeated forever) . __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ Added file: http://bugs.python.org/file8954/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __The server isn#39;t handling the close event properly.nbsp; I#39;ll fix that.brbrdiv class=gmail_quoteOn Dec 13, 2007 9:06 PM, Guido van Rossum lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt; wrote: brblockquote class=gmail_quote style=border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;div class=Ih2E3dbrGuido van Rossum added the comment:brbr/divI spoke too soon. In a debug build, this hangs forever during the brsecond iteration:brbr./python.exe Lib/test/regrtest.py -uall -R1:1 test_sslbrbrAdding -v, it looks like two iterations are carried out perfectly (onebrmust be a trial run, one the warm-up run), but the third run goes brbeserk; the output ends like this:brdiv class=Ih2E3dbrtestAsyncoreServer (test.test_ssl.ThreadedTests) ...br/divnbsp;server: nbsp;read b#39;over\n#39; from clientbrnbsp;server: nbsp;closed connection lt;ssl.SSLSocket object, fd=6, family=2,brtype=1, proto=0gt;brnbsp;server: nbsp;read b#39;#39; from clientbrnbsp;server: nbsp;closed connection lt;ssl.SSLSocket object, fd=10, family=2,brtype=1, proto=0gt;brnbsp;server: nbsp;read b#39;#39; from client brnbsp;server: nbsp;closed connection lt;ssl.SSLSocket object, fd=10, family=2,brtype=1, proto=0gt;br.br. (the last two lines repeated forever)brdivdiv/divdiv class=Wj3C7c.brbr__ brTracker lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt;brlt;a href=http://bugs.python.org/issue1469; target=_blankhttp://bugs.python.org/issue1469/agt;br__ br/div/div/blockquote/divbr ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Added file: http://bugs.python.org/file8955/patch-4 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ patch-4 Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Bill Janssen: Removed file: http://bugs.python.org/file8949/patch-3 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Bill Janssen: Removed file: http://bugs.python.org/file8954/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Perfect! Check it in. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Done. On Dec 14, 2007 9:44 AM, Guido van Rossum [EMAIL PROTECTED] wrote: Guido van Rossum added the comment: Here's an update version where the asyncore test server properly handles the EOF race condition. Perfect! Check it in. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ Added file: http://bugs.python.org/file8956/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __Done.brbrdiv class=gmail_quoteOn Dec 14, 2007 9:44 AM, Guido van Rossum lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt; wrote:brblockquote class=gmail_quote style=border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex; brGuido van Rossum added the comment:brdiv class=Ih2E3dbrgt; Here#39;s an update version where the asyncore test server properly handlesbrgt; the EOF race condition.brbr/divPerfect! Check it in.brdiv div/divdiv class=Wj3C7cbr__brTracker lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt;brlt;a href=http://bugs.python.org/issue1469; target=_blank http://bugs.python.org/issue1469/agt;br__br/div/div/blockquote/divbr ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Guido van Rossum: -- resolution: - fixed status: open - closed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: Hm, when I run the full test_ssl test suite with ./python Lib/test/regrtest.py -v -uall test_ssl it always hangs in testAsyncoreServer after printing this: testAsyncoreServer (test.test_ssl.ThreadedTests) ... server: new connection from 127.0.0.1:59781 client: sending 'FOO\n'... server: read 4 from client __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: It hangs on my Linux box, too. I've not yet tried on Windows. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Good catch. I flipped a bit cleaning up the C code. Here's a fixed patch. Added file: http://bugs.python.org/file8949/patch-3 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ patch-3 Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: Good catch. I flipped a bit cleaning up the C code. Here's a fixed patch. Added file: http://bugs.python.org/file8949/patch-3 Great! Go ahead and check it in. Sorry for deleting the _real_close() earlier BTW. Enjoy your time away! __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: I spoke too soon. In a debug build, this hangs forever during the second iteration: ./python.exe Lib/test/regrtest.py -uall -R1:1 test_ssl Adding -v, it looks like two iterations are carried out perfectly (one must be a trial run, one the warm-up run), but the third run goes beserk; the output ends like this: testAsyncoreServer (test.test_ssl.ThreadedTests) ... server: read b'over\n' from client server: closed connection ssl.SSLSocket object, fd=6, family=2, type=1, proto=0 server: read b'' from client server: closed connection ssl.SSLSocket object, fd=10, family=2, type=1, proto=0 server: read b'' from client server: closed connection ssl.SSLSocket object, fd=10, family=2, type=1, proto=0 . . (the last two lines repeated forever) . __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Giampaolo Rodola': -- nosy: +giampaolo.rodola __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: I think I've figured it out. My initial patch to socket.py and ssl.py had an extra method defined on socket.socket, _real_close(), which did' the cleanup work of deallocating the underlying socket, and in the SSL subclass, of releasing the SSL context. Guido must have removed that method, and folded what it does into the socket.close() call. But that means that subclasses can't hook onto that code, to get cleaned up when the socket is truly closed. So the SSL context gets leaked. Adding _real_close() back to the socket module fixes most of the leak, but not all of it. I'm looking for the other problem... __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: So, what's the final status of __del__ in py3K? The other bit of leak is due to _real_close() not being called when a socket is dropped on the floor (say, you try to connect, fail, and raise the exception back to the caller, without ever explicitly calling close() on the opened socket). It looks like the __del__ method in SSLSocket isn't being called. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Amaury Forgeot d'Arc added the comment: Is it possible that you have the same problem as in http://bugs.python.org/issue1540 ? To be sure, check the gc.garbage variable at the end of the test. It may contain the references you are looking for... -- nosy: +amaury.forgeotdarc __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: So, what's the final status of __del__ in py3K? The other bit of leak is due to _real_close() not being called when a socket is dropped on the floor (say, you try to connect, fail, and raise the exception back to the caller, without ever explicitly calling close() on the opened socket). It looks like the __del__ method in SSLSocket isn't being called. There are no current plans to change __del__ in Py3k. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: The other leak comes from this code: s = ssl.wrap_socket(socket.socket(), ...) s.connect((SOME BOGUS ADDRESS OR SERVER)) The connect() fails, and the SSLSocket sgets dropped on the floor, but never seems to be GC'd, (or the GC never seems to call the __del__ method). Any idea why? Could this be because the base class is a C class? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: gc.garbage is always empty. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: I wonder if Christian Heimes was correct that the ssl object needs GC support? This was part of his patch (which I checked in and then reverted because the other part of it didn't work as advertised). Alternatively, if 's' is involved in a cycle *and* any of the objects in the cycle has a __del__ method, the GC won't call __del__. This seems counter-intuitive but is actually correct since __del__ can only be called when there are no references to an object left. --Guido On Dec 10, 2007 3:30 PM, Bill Janssen [EMAIL PROTECTED] wrote: Bill Janssen added the comment: The other leak comes from this code: s = ssl.wrap_socket(socket.socket(), ...) s.connect((SOME BOGUS ADDRESS OR SERVER)) The connect() fails, and the SSLSocket sgets dropped on the floor, but never seems to be GC'd, (or the GC never seems to call the __del__ method). Any idea why? Could this be because the base class is a C class? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Ah, I see what's going on. The revision of the socket code (nice job, by the way) removed the distinction between the C socket object and the Python socket object. The C SSLContext keeps a pointer to the C socket object, which is now the Python socket object, or in the SSL case, the SSLSocket object. So there's a circular reference. The right fix is to make the ref in the SSL C code a weakref. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: Guido van Rossum wrote: Guido van Rossum added the comment: I wonder if Christian Heimes was correct that the ssl object needs GC support? This was part of his patch (which I checked in and then reverted because the other part of it didn't work as advertised). Alternatively, if 's' is involved in a cycle *and* any of the objects in the cycle has a __del__ method, the GC won't call __del__. This seems counter-intuitive but is actually correct since __del__ can only be called when there are no references to an object left. A combination of GC support and the removal of __del__ from SSLSocket has fixed the reference leak problem for me. I tried to move the _real_clean() call to PySSL_dealloc but it wasn't as easy as I thought. The code contains a reference cycle. ssl.SSLSocket() contains a reference to PySSLObject and PySSLObject-Socket is a reference to the same ssl.SSLSocket instance. Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: I think Christian analysis is right, in that it takes a bit of GC support, but not perhaps in the specifics of his approach. I've done two things to fix this: 1) Put _real_close() back in socket.py, and then override it in ssl.SSLSocket to release the SSLContext, and 2) Change the pointer to the SSLSocket in _ssl.c to a weakref. Now SSLSockets get cleaned up properly by the GC system. One question: _real_close() is what Java calls a protected method. Do we have any way to mark this with annotations or metaclass stuff? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Here's a patch -- take a look and let me know. I also added a real asyncore server test. Added file: http://bugs.python.org/file8919/a __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ a Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: Now, about the confused signature of SSLSocket.read(). I'm not sure how confused it is. It's sui generis; SSLSocket doesn't inherit from some other class with a different read() method with a different signature. But we could change the name, perhaps to readbytes, or something similar? I wrote it the way I did to keep some kind of compatibility with pre-3.0 uses of the SSL module. Anyway, the error is coming from a bug in the implementation of 'suppress_ragged_eofs'. The code in SSLSocket.read() that handles the EOF error needs to return the right kind of value. I've added it to the patch. Added file: http://bugs.python.org/file8922/patch-2 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ patch-2 Description: Binary data ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Bill Janssen: Removed file: http://bugs.python.org/file8919/a __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Guido van Rossum: Removed file: http://bugs.python.org/file8892/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: Don't worry, I did back it out before releasing 3.0a2. I believe I hacked your code a bit before checking it in (or after you checked it in, can't remember). Did you see my bug report with a TypeError? __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Guido van Rossum: Removed file: http://bugs.python.org/file8845/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: I'm still not sure what the problem is. This code was working fine when I committed it, no leaks, so something else must have changed under the covers. I don't believe that adding GC to the C code is the answer; it should be automatically GC'd. So I'd back that patch out. I intend to wolf-fence the tests in the test module until I find a single test that causes the leaks. Sorry I haven't had time to look at this earlier, but we're in the middle of a big presentation afternoon here at work. On Dec 6, 2007 11:45 AM, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: Guido van Rossum wrote: I just reverted it. I put a bit of debugging in the call to _real_close(), and even with Christian's corrections it fails miserably. I prefer leaking. I agree! I've not enough time to work on the patch. A working patch requires some redesign of the ssl interface. It's definitely too late now. Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ Added file: http://bugs.python.org/file8892/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __I#39;m still not sure what the problem is.nbsp; This code was working fine when I committed it, no leaks, so something else must have changed under the covers.nbsp; I don#39;t believe that adding GC to the C code is the answer; it should be automatically GC#39;d.nbsp; So I#39;d back that patch out. brbrI intend to wolf-fence the tests in the test module until I find a single test that causes the leaks.nbsp; Sorry I haven#39;t had time to look at this earlier, but we#39;re in the middle of a big presentation afternoon here at work. brbrdiv class=gmail_quoteOn Dec 6, 2007 11:45 AM, Christian Heimes lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt; wrote:brblockquote class=gmail_quote style=border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex; div class=Ih2E3dbrChristian Heimes added the comment:brbrGuido van Rossum wrote:br/divdiv class=Ih2E3dgt; I just reverted it. I put a bit of debugging in the call tobrgt; _real_close(), and even with Christian#39;s corrections it fails miserably. brgt; I prefer leaking.brbr/divI agree! I#39;ve not enough time to work on the patch. A working patchbrrequires some redesign of the ssl interface. It#39;s definitely too late now.brfont color=#88br Christianbr/fontdivdiv/divdiv class=Wj3C7cbr__brTracker lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt;brlt;a href=http://bugs.python.org/issue1469; target=_blank http://bugs.python.org/issue1469/agt;br__br/div/div/blockquote/divbr ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment:
I can reproduce the problem when I run the ssl tests with -unetwork.
I've used PYTHONDUMREFS and combinerefs.py to find the leaking objects.
I'm seeing lots of
0x9fd3e4c [1] SSLSocket
0x9fd2928 [1] ssl.SSLContext ssl.SSLContext object at 0x9fd2928
0x9fd5b74 [1] dict {'certfile': None, '_sslobj': ssl.SSLContext object
at 0x9fd2928, 'do_handshake_on_con
nect': True, 'ssl_version': 1, '_base': None, 'cert_reqs': 2,
'ca_certs': '/home/heimes/dev/python/py3k/Lib
/test/https_svn_python_org_root.pem', 'keyfile': None,
'suppress_ragged_eofs': True}
in the output.
I guess the problem is connected to::
_ssl.sslwrap(self, server_side,
keyfile, certfile,
cert_reqs, ssl_version, ca_certs)
in Lib/ssl.py. I'm digging in
--
assignee: janssen - tiran
nosy: +gvanrossum
priority: normal - high
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1469
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment:
I may have found the error. The PySSL_Type doesn't support GC but it
should in order to clean up self-Socket.
I've started to fix it but I don't have time to work on the problem 'til
tonight. The patch causes a seg fault. I may have overlooked or
misunderstood something.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1210324800 (LWP 31486)]
0x08127d03 in gc_list_remove (node=0x86d1cac) at Modules/gcmodule.c:158
158 node-gc.gc_prev-gc.gc_next = node-gc.gc_next;
Added file: http://bugs.python.org/file8884/ssl_gc.patch
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1469
__Index: Tools/scripts/combinerefs.py
===
--- Tools/scripts/combinerefs.py (Revision 59376)
+++ Tools/scripts/combinerefs.py (Arbeitskopie)
@@ -86,7 +86,8 @@
break
def combine(fname):
-f = file(fname)
+f = open(fname)
+
fi = iter(f)
for line in read(fi, re.compile(r'^Remaining objects:$'), False):
Index: Modules/_ssl.c
===
--- Modules/_ssl.c (Revision 59376)
+++ Modules/_ssl.c (Arbeitskopie)
@@ -1050,16 +1050,32 @@
return NULL;
}
-static void PySSL_dealloc(PySSLObject *self)
+/* GC support. */
+static int
+PySSL_traverse(PySSLObject *self, visitproc visit, void *arg)
{
+ Py_VISIT(self-Socket);
+ return 0;
+}
+
+static int
+PySSL_clear(PySSLObject *self)
+{
+ Py_CLEAR(self-Socket);
+ return 0;
+}
+
+static void
+PySSL_dealloc(PySSLObject *self)
+{
+ PySSL_clear(self);
if (self-peer_cert) /* Possible not to have one? */
X509_free (self-peer_cert);
if (self-ssl)
SSL_free(self-ssl);
if (self-ctx)
SSL_CTX_free(self-ctx);
- Py_XDECREF(self-Socket);
- PyObject_Del(self);
+ Py_Type(self)-tp_free((PyObject *)self);
}
/* If the socket has a timeout, do a select()/poll() on the socket.
@@ -1372,7 +1388,7 @@
/* methods */
(destructor)PySSL_dealloc, /*tp_dealloc*/
0,/*tp_print*/
- (getattrfunc)PySSL_getattr, /*tp_getattr*/
+ 0,/*tp_getattr*/
0,/*tp_setattr*/
0,/*tp_compare*/
0,/*tp_repr*/
@@ -1380,6 +1396,32 @@
0,/*tp_as_sequence*/
0,/*tp_as_mapping*/
0,/*tp_hash*/
+ 0,/* tp_call */
+ 0,/* tp_str */
+ PyObject_GenericGetAttr, /* tp_getattro */
+ 0,/* tp_setattro */
+ 0,/* tp_as_buffer */
+ Py_TPFLAGS_DEFAULT |
+ Py_TPFLAGS_HAVE_GC, /* tp_flags */
+ 0,/* tp_doc */
+ (traverseproc)PySSL_traverse, /* tp_traverse */
+ (inquiry)PySSL_clear, /* tp_clear */
+ 0,/* tp_richcompare */
+ 0,/* tp_weaklistoffset */
+ 0,/* tp_iter */
+ 0,/* tp_iternext */
+ PySSLMethods, /* tp_methods */
+ 0,/* tp_members */
+ 0,/* tp_getset */
+ 0,/* tp_base */
+ 0,/* tp_dict */
+ 0,/* tp_descr_get */
+ 0,/* tp_descr_set */
+ 0,/* tp_dictoffset */
+ 0,/* tp_init */
+ PyType_GenericAlloc, /* tp_alloc */
+ PyType_GenericNew, /* tp_new */
+ PyObject_GC_Del, /* tp_free */
};
#ifdef HAVE_OPENSSL_RAND
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: I found the problem! I've to use PyObject_GC_New instead of PyObject_New __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: I get a segfault when I run it like this: $ ./python.exe Lib/test/regrtest.py -uall test_ssl test_ssl Segmentation fault $ (Without -uall it passes.) __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Christian Heimes: Removed file: http://bugs.python.org/file8884/ssl_gc.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment:
New patch
The new patch doesn't cause a seg fault but it doesn't help. It
*increases* the number of reference leaks.
test_ssl leaked [1610, 1610] references, sum=3220
Added file: http://bugs.python.org/file8885/ssl_gc.patch
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1469
__Index: Tools/scripts/combinerefs.py
===
--- Tools/scripts/combinerefs.py (Revision 59376)
+++ Tools/scripts/combinerefs.py (Arbeitskopie)
@@ -86,7 +86,8 @@
break
def combine(fname):
-f = file(fname)
+f = open(fname)
+
fi = iter(f)
for line in read(fi, re.compile(r'^Remaining objects:$'), False):
Index: Modules/_ssl.c
===
--- Modules/_ssl.c (Revision 59376)
+++ Modules/_ssl.c (Arbeitskopie)
@@ -266,7 +266,7 @@
int ret;
int verification_mode;
- self = PyObject_New(PySSLObject, PySSL_Type); /* Create new object */
+ self = PyObject_GC_New(PySSLObject, PySSL_Type); /* Create new object */
if (self == NULL)
return NULL;
self-peer_cert = NULL;
@@ -385,6 +385,7 @@
self-Socket = Sock;
Py_INCREF(self-Socket);
+ _PyObject_GC_TRACK(self);
return self;
fail:
if (errstr)
@@ -1050,16 +1051,33 @@
return NULL;
}
-static void PySSL_dealloc(PySSLObject *self)
+/* GC support. */
+static int
+PySSL_traverse(PySSLObject *self, visitproc visit, void *arg)
{
+ Py_VISIT(self-Socket);
+ return 0;
+}
+
+static int
+PySSL_clear(PySSLObject *self)
+{
+ Py_CLEAR(self-Socket);
+ return 0;
+}
+
+static void
+PySSL_dealloc(PySSLObject *self)
+{
+ PyObject_GC_UnTrack(self);
if (self-peer_cert) /* Possible not to have one? */
- X509_free (self-peer_cert);
+ X509_free(self-peer_cert);
if (self-ssl)
SSL_free(self-ssl);
if (self-ctx)
SSL_CTX_free(self-ctx);
- Py_XDECREF(self-Socket);
- PyObject_Del(self);
+ PySSL_clear(self);
+ Py_Type(self)-tp_free((PyObject *)self);
}
/* If the socket has a timeout, do a select()/poll() on the socket.
@@ -1359,20 +1377,15 @@
{NULL, NULL}
};
-static PyObject *PySSL_getattr(PySSLObject *self, char *name)
-{
- return Py_FindMethod(PySSLMethods, (PyObject *)self, name);
-}
-
static PyTypeObject PySSL_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- ssl.SSLContext, /*tp_name*/
+ _ssl.SSLContext, /*tp_name*/
sizeof(PySSLObject), /*tp_basicsize*/
0,/*tp_itemsize*/
/* methods */
(destructor)PySSL_dealloc, /*tp_dealloc*/
0,/*tp_print*/
- (getattrfunc)PySSL_getattr, /*tp_getattr*/
+ 0,/*tp_getattr*/
0,/*tp_setattr*/
0,/*tp_compare*/
0,/*tp_repr*/
@@ -1380,6 +1393,32 @@
0,/*tp_as_sequence*/
0,/*tp_as_mapping*/
0,/*tp_hash*/
+ 0,/* tp_call */
+ 0,/* tp_str */
+ PyObject_GenericGetAttr, /* tp_getattro */
+ 0,/* tp_setattro */
+ 0,/* tp_as_buffer */
+ Py_TPFLAGS_DEFAULT |
+ Py_TPFLAGS_HAVE_GC, /* tp_flags */
+ 0,/* tp_doc */
+ (traverseproc)PySSL_traverse, /* tp_traverse */
+ (inquiry)PySSL_clear, /* tp_clear */
+ 0,/* tp_richcompare */
+ 0,/* tp_weaklistoffset */
+ 0,/* tp_iter */
+ 0,/* tp_iternext */
+ PySSLMethods, /* tp_methods */
+ 0,/* tp_members */
+ 0,/* tp_getset */
+ 0,/* tp_base */
+ 0,/* tp_dict */
+ 0,/* tp_descr_get */
+ 0,/* tp_descr_set */
+ 0,/* tp_dictoffset */
+ 0,/* tp_init */
+ PyType_GenericAlloc, /* tp_alloc */
+ PyType_GenericNew, /* tp_new */
+ PyObject_GC_Del, /* tp_free */
};
#ifdef HAVE_OPENSSL_RAND
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Changes by Christian Heimes: Removed file: http://bugs.python.org/file8885/ssl_gc.patch __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment:
The new patch works and fixes the ref leak. I had to move the call
self._real_close() from __del__ to PySSL_dealloc(). I don't know if you
are going to like it. :]
Added file: http://bugs.python.org/file8886/ssl_gc.patch
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue1469
__Index: Tools/scripts/combinerefs.py
===
--- Tools/scripts/combinerefs.py (Revision 59376)
+++ Tools/scripts/combinerefs.py (Arbeitskopie)
@@ -86,7 +86,8 @@
break
def combine(fname):
-f = file(fname)
+f = open(fname)
+
fi = iter(f)
for line in read(fi, re.compile(r'^Remaining objects:$'), False):
Index: Lib/ssl.py
===
--- Lib/ssl.py (Revision 59376)
+++ Lib/ssl.py (Arbeitskopie)
@@ -148,6 +148,10 @@
self.do_handshake_on_connect = do_handshake_on_connect
self.suppress_ragged_eofs = suppress_ragged_eofs
+# See Modules/_ssl.c:PySSL_dealloc()
+# def __del__(self):
+#self._real_close()
+
def dup(self):
raise NotImplemented(Can't dup() %s instances %
self.__class__.__name__)
@@ -300,6 +304,7 @@
socket.shutdown(self, how)
def _real_close(self):
+# real close is called by Modules/_ssl.c:PySSL_dealloc()
self._sslobj = None
# self._closed = True
if self._base:
@@ -348,10 +353,6 @@
self.do_handshake_on_connect),
addr)
-
-def __del__(self):
-self._real_close()
-
def wrap_socket(sock, keyfile=None, certfile=None,
server_side=False, cert_reqs=CERT_NONE,
ssl_version=PROTOCOL_SSLv23, ca_certs=None,
Index: Modules/_ssl.c
===
--- Modules/_ssl.c (Revision 59376)
+++ Modules/_ssl.c (Arbeitskopie)
@@ -266,7 +266,7 @@
int ret;
int verification_mode;
- self = PyObject_New(PySSLObject, PySSL_Type); /* Create new object */
+ self = PyObject_GC_New(PySSLObject, PySSL_Type); /* Create new object */
if (self == NULL)
return NULL;
self-peer_cert = NULL;
@@ -385,6 +385,7 @@
self-Socket = Sock;
Py_INCREF(self-Socket);
+ _PyObject_GC_TRACK(self);
return self;
fail:
if (errstr)
@@ -1050,16 +1051,41 @@
return NULL;
}
-static void PySSL_dealloc(PySSLObject *self)
+/* GC support. */
+static int
+PySSL_traverse(PySSLObject *self, visitproc visit, void *arg)
{
+ Py_VISIT(self-Socket);
+ return 0;
+}
+
+static int
+PySSL_clear(PySSLObject *self)
+{
+ Py_CLEAR(self-Socket);
+ return 0;
+}
+
+static void
+PySSL_dealloc(PySSLObject *self)
+{
+ PyObject *o;
+ PyObject *exc_type, *exc_value, *exc_tb;
+
+ PyErr_Fetch(exc_type, exc_value, exc_tb);
+ o = PyObject_CallMethod((PyObject*)self, _real_close, NULL);
+ Py_XDECREF(o);
+ PyErr_Restore(exc_type, exc_value, exc_tb);
+
+ PyObject_GC_UnTrack(self);
if (self-peer_cert) /* Possible not to have one? */
- X509_free (self-peer_cert);
+ X509_free(self-peer_cert);
if (self-ssl)
SSL_free(self-ssl);
if (self-ctx)
SSL_CTX_free(self-ctx);
- Py_XDECREF(self-Socket);
- PyObject_Del(self);
+ Py_CLEAR(self-Socket);
+ Py_Type(self)-tp_free((PyObject *)self);
}
/* If the socket has a timeout, do a select()/poll() on the socket.
@@ -1359,20 +1385,15 @@
{NULL, NULL}
};
-static PyObject *PySSL_getattr(PySSLObject *self, char *name)
-{
- return Py_FindMethod(PySSLMethods, (PyObject *)self, name);
-}
-
static PyTypeObject PySSL_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
- ssl.SSLContext, /*tp_name*/
+ _ssl.SSLContext, /*tp_name*/
sizeof(PySSLObject), /*tp_basicsize*/
0,/*tp_itemsize*/
/* methods */
(destructor)PySSL_dealloc, /*tp_dealloc*/
0,/*tp_print*/
- (getattrfunc)PySSL_getattr, /*tp_getattr*/
+ 0,/*tp_getattr*/
0,/*tp_setattr*/
0,/*tp_compare*/
0,/*tp_repr*/
@@ -1380,6 +1401,32 @@
0,/*tp_as_sequence*/
0,/*tp_as_mapping*/
0,/*tp_hash*/
+ 0,/* tp_call */
+ 0,/* tp_str */
+ PyObject_GenericGetAttr, /* tp_getattro */
+ 0,/* tp_setattro */
+ 0,/* tp_as_buffer */
+ Py_TPFLAGS_DEFAULT |
+ Py_TPFLAGS_HAVE_GC, /* tp_flags */
+ 0,/* tp_doc */
+ (traverseproc)PySSL_traverse, /* tp_traverse */
+ (inquiry)PySSL_clear, /* tp_clear */
+ 0,/* tp_richcompare */
+ 0,/* tp_weaklistoffset */
+ 0,/* tp_iter */
+ 0,/* tp_iternext */
+ PySSLMethods, /* tp_methods */
+ 0,/* tp_members */
+ 0,/* tp_getset */
+ 0,/* tp_base */
+ 0,/* tp_dict */
+ 0,/* tp_descr_get */
+ 0,/* tp_descr_set */
+ 0,/* tp_dictoffset */
+ 0,/* tp_init */
+ PyType_GenericAlloc, /* tp_alloc */
+ PyType_GenericNew, /* tp_new */
+ PyObject_GC_Del, /* tp_free */
};
#ifdef HAVE_OPENSSL_RAND
___
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: I will look at this in an hour or so, after I bring Orlijn to school and drive to work. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: Guido van Rossum wrote: I will look at this in an hour or so, after I bring Orlijn to school and drive to work. I found two problems with the _real_close() call in PySSL_dealloc. I'm digging into it now. o = PyObject_CallMethod((PyObject*)self-Socket, _real_close, ); Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: Oops, I just committed revision 59394. Please advise. -- priority: high - immediate __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Guido van Rossum added the comment: I'm giving up on this for now. There are other weird bugs in the code, e.g. this simple piece of code fails: x = urllib.urlopen(https://mail.google.com;).read() Traceback (most recent call last): File stdin, line 1, in module File /usr/local/google/home/guido/python/py3kd/Lib/io.py, line 463, in read return self.readall() File /usr/local/google/home/guido/python/py3kd/Lib/io.py, line 473, in readall data = self.read(DEFAULT_BUFFER_SIZE) File /usr/local/google/home/guido/python/py3kd/Lib/io.py, line 466, in read del b[n:] TypeError: slice indices must be integers or None or have an __index__ method [64813 refs] The debugger tells me that 'n' contains a bytes object instead of an int. This appears due to the confused signature of _sslobj.read() (which can behave as either read() or readinto()). Its docstring is wrong too. We'll deal with this after 3.0a2 is released. -- assignee: tiran - janssen priority: immediate - normal __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: Guido van Rossum wrote: I just reverted it. I put a bit of debugging in the call to _real_close(), and even with Christian's corrections it fails miserably. I prefer leaking. I agree! I've not enough time to work on the patch. A working patch requires some redesign of the ssl interface. It's definitely too late now. Christian __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: Are you sure that the SSL tests are still leaking memory? The tests aren't leaking references on Windows nor on Linux. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: I'm seeing the leaks on my Leopard machine. I haven't had a chance to look into it yet. On Dec 1, 2007 11:20 AM, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: Are you sure that the SSL tests are still leaking memory? The tests aren't leaking references on Windows nor on Linux. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ Added file: http://bugs.python.org/file8845/unnamed __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __spanI#39;m seeing the leaks on my Leopard machine.nbsp; I haven#39;t had a chance to look into it yet./spanbrbrdiv class=gmail_quoteOn Dec 1, 2007 11:20 AM, Christian Heimes lt;a href=mailto:[EMAIL PROTECTED] [EMAIL PROTECTED]/agt; wrote:brblockquote class=gmail_quote style=border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;brChristian Heimes added the comment:brbrAre you sure that the SSL tests are still leaking memory? The tests braren#39;t leaking references on Windows nor on Linux.brbr__brTracker lt;a href=mailto:[EMAIL PROTECTED][EMAIL PROTECTED]/agt;brlt;a href=http://bugs.python.org/issue1469; target=_blank http://bugs.python.org/issue1469/agt;br__br/blockquote/divbr ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Bill Janssen added the comment: What platform? Bill On Nov 20, 2007 11:21 PM, Christian Heimes [EMAIL PROTECTED] wrote: Christian Heimes added the comment: I don't see leaks when I run the tests with $ ./python Lib/test/regrtest.py -R:: test_ssl.py test_ssl beginning 9 repetitions 123456789 . 1 test OK. [80034 refs] -- nosy: +tiran priority: - normal __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: Ubuntu Linux x86 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
Christian Heimes added the comment: I don't see leaks when I run the tests with $ ./python Lib/test/regrtest.py -R:: test_ssl.py test_ssl beginning 9 repetitions 123456789 . 1 test OK. [80034 refs] -- nosy: +tiran priority: - normal __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1469] SSL tests leak memory
New submission from Bill Janssen: I'm seeing leaks in the test_ssl run, after various socket.py changes. I'm looking into it. -- assignee: janssen components: Library (Lib) keywords: py3k messages: 57667 nosy: janssen severity: normal status: open title: SSL tests leak memory versions: Python 3.0 __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue1469 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
