[issue3660] reference leaks in 3.0

[Neal Norwitz]

New submission from Neal Norwitz <[EMAIL PROTECTED]>:

Even after adding the current patch in http://bugs.python.org/issue3651
there are many reference leaks.  This bug can be a placeholder for all
the reference leaks returned from:  
  ./python ./Lib/test/regrtest.py -R 3:2 -uall,-bsddb 

The current list is:

test_unittest leaked [124, 124] references, sum=248
test_array leaked [110, 110] references, sum=220
test_audioop leaked [75, 75] references, sum=150
test_binascii leaked [4, 4] references, sum=8
test_binhex leaked [4, 4] references, sum=8
test_codecs leaked [3, 3] references, sum=6
test_ctypes leaked [9, 9] references, sum=18
test_dbm leaked [194, 194] references, sum=388
test_dbm_gnu leaked [2, 2] references, sum=4
test_fcntl leaked [2, 2] references, sum=4
test_file leaked [8, 8] references, sum=16
test_fileio leaked [1, 1] references, sum=2
test_memoryio leaked [3, 3] references, sum=6
test_minidom leaked [5, 5] references, sum=10
test_mmap leaked [307, 307] references, sum=614
test_ossaudiodev leaked [2, 2] references, sum=4
test_pickle leaked [130, 130] references, sum=260
test_pickletools leaked [503, 503] references, sum=1006
test_pyexpat leaked [1, 1] references, sum=2
test_re leaked [4, 4] references, sum=8
test_site leaked [88, 88] references, sum=176
test_socket leaked [13, 13] references, sum=26
test_sqlite leaked [17, 17] references, sum=34
test_ssl leaked [82, 82] references, sum=164
test_struct leaked [5, 5] references, sum=10
test_unicode leaked [2, 2] references, sum=4
test_urllib2_localnet leaked [3, 3] references, sum=6
test_xmlrpc leaked [18, 18] references, sum=36
test_xmlrpc_net leaked [1, 1] references, sum=2
test_zlib leaked [10, 10] references, sum=20

--
components: Interpreter Core
messages: 71851
nosy: nnorwitz
priority: release blocker
severity: normal
status: open
title: reference leaks in 3.0
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

As of r66047, I get the following results (without "-uall", though):

test_unittest leaked [124, 124] references, sum=248
test_binascii leaked [1, 1] references, sum=2
test_distutils leaked [141, 142] references, sum=283
test_logging leaked [219, 147] references, sum=366
test_multiprocessing leaked [0, 1] references, sum=1
test_pickle leaked [1, 1] references, sum=2
test_pickletools leaked [1, 1] references, sum=2
test_popen leaked [37, 0] references, sum=37
test_site leaked [88, 88] references, sum=176
test_sqlite leaked [17, 17] references, sum=34
test_unicode leaked [2, 2] references, sum=4
test_urllib2_localnet leaked [3, 3] references, sum=6
test_xmlrpc leaked [-84, 85] references, sum=1

24 tests skipped:
test_bsddb3 test_cProfile test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dbm_gnu test_kqueue test_nis test_normalization
test_ossaudiodev test_pep277 test_socketserver test_startfile
test_tcl test_timeout test_urllib2net test_urllibnet test_winreg
test_winsound test_xmlrpc_net test_zipfile64

--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

- the "test_site leaked [88, 88]" is the same as problem as issue3667.

- test_unicodes leaks in PyUnicode_AsEncodedString (attached patch), and
also with:
  str(memoryview(b'character buffers are decoded to unicode'), 'utf-8')
I tried another patch, but I'm not sure: I get lost between all these
buffers... a Py_DECREF(self->view.obj) in memory_releasebuf() seems to work.

--
keywords: +patch
nosy: +amaury.forgeotdarc
Added file: http://bugs.python.org/file11285/encode-leak.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

>   str(memoryview(b'character buffers are decoded to unicode'), 'utf-8')
> I tried another patch, but I'm not sure: I get lost between all these
> buffers... a Py_DECREF(self->view.obj) in memory_releasebuf() seems to work.

Could you open a separate issue for the latter? Adding a DECREF in
memory_releasebuf() isn't the right thing to do, because PyBuffer_Release()
already does such a DECREF. I think the problem is rather in memory_getbuf(), it
tries to take some strange shortcuts.

Oh, and I realize that memoryobject doesn't have tp_traverse and tp_clear, which
looks quite wrong...

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

issue3712 tracks the memoryview issues.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

the leaks in test_pickle and test_pickletools are corrected by the
attached patch.

Added file: http://bugs.python.org/file11288/pickle-leak.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Amaury, I believe the first part of encode-leak.patch is wrong, you
should Py_DECREF the bytearray after it has been converted to bytes, not
before. Here is an alternate patch.

Added file: http://bugs.python.org/file11309/encode-leak2.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

Oops, you are right of course.
I remove my patch so we don't get confused.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Changes by Amaury Forgeot d'Arc <[EMAIL PROTECTED]>:


Removed file: http://bugs.python.org/file11285/encode-leak.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

With the two patches applied, we are now at:

test_unittest leaked [124, 124] references, sum=248
test_distutils leaked [141, 142] references, sum=283
test_docxmlrpc leaked [85, -85] references, sum=0
test_logging leaked [366, -366] references, sum=0
test_os leaked [37, 0] references, sum=37
test_site leaked [88, 88] references, sum=176
test_smtplib leaked [0, 87] references, sum=87
test_sqlite leaked [17, 17] references, sum=34
test_telnetlib leaked [151, -151] references, sum=0
test_unicode leaked [1, 1] references, sum=2
test_urllib2_localnet leaked [3, 3] references, sum=6
test_xmlrpc leaked [-85, 0] references, sum=-85

24 tests skipped:
test_bsddb3 test_cProfile test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dbm_gnu test_kqueue test_nis test_normalization
test_ossaudiodev test_pep277 test_socketserver test_startfile
test_tcl test_timeout test_urllib2net test_urllibnet test_winreg
test_winsound test_xmlrpc_net test_zipfile64
2 skips unexpected on linux2:

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

Did you look at the patch for issue3667 ? it should at least correct
test_site.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Ok, after the two patches plus the patch in #3667, I get the following:

test_asyncore leaked [84, -84] references, sum=0
test_distutils leaked [141, 142] references, sum=283
test_docxmlrpc leaked [-85, 0] references, sum=-85
test_logging leaked [219, -219] references, sum=0
test_sqlite leaked [17, 17] references, sum=34
test_unicode leaked [1, 1] references, sum=2
test_urllib2_localnet leaked [3, 3] references, sum=6
test_xmlrpc leaked [-6, -79] references, sum=-85

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Changes by Antoine Pitrou <[EMAIL PROTECTED]>:


--
keywords: +needs review

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

The patch for _pickle has been committed in r66227.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Numbers for the current py3k branch (without encode-leak2.patch):

test_distutils leaked [141, 142] references, sum=283
test_docxmlrpc leaked [-7, -85] references, sum=-92
test_logging leaked [0, 219] references, sum=219
test_poplib leaked [0, 84] references, sum=84
test_sys leaked [0, 34] references, sum=34
test_unicode leaked [1, 1] references, sum=2
test_urllib2_localnet leaked [3, 3] references, sum=6
test_xmlrpc leaked [192, -190] references, sum=2

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Neal Norwitz]

Neal Norwitz <[EMAIL PROTECTED]> added the comment:

The only one that is probably an issue based on Antoine's info is:

test_unicode leaked [1, 1] references, sum=2

I've seen test_urllib2_localnet leak 3 before.  I don't know that it's
a real leak.  I'm pretty sure it is not a regression though.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

FWIW, applying encode-leak2.patch removes the leak in test_unicode.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

Antoine, it seem that with encode-leak2.patch, the error path after
PyErr_WarnEx() leaks the value of "v".

I rewrote the whole paragraph to make it more straightforward:
- the normal case is tested first
- all paths end with a "return", and no goto.
- no need to test that PyBytes_FromStringAndSize returns a PyBytes...

I find the code much easier to check in this form, but of course this is
a subjective POV.

Added file: http://bugs.python.org/file11391/encode-leak3.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Le vendredi 05 septembre 2008 à 13:08 +, Amaury Forgeot d'Arc a
écrit :
> Antoine, it seem that with encode-leak2.patch, the error path after
> PyErr_WarnEx() leaks the value of "v".

Hmm, you are right.

> I rewrote the whole paragraph to make it more straightforward:
> - the normal case is tested first
> - all paths end with a "return", and no goto.
> - no need to test that PyBytes_FromStringAndSize returns a PyBytes...

I'll take a look!

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Amaury, your patch is much clearer indeed and it fixes the leak.

--
keywords:  -needs review

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

encode-leak3.patch applied in r66234.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3660] reference leaks in 3.0

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Current status:

test_distutils leaked [141, 142] references, sum=283
test_logging leaked [0, -219] references, sum=-219
test_smtplib leaked [0, 87] references, sum=87

The distutils leak should be investigated, but the overall situation is
rather good now. The other, transient, leaks might be due to some thread
being cleaned up too late or something.

--
priority: release blocker -> high

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com