[issue24329] __qualname__ and __slots__

Changes by Xiang Zhang : -- keywords: +patch stage: -> patch review type: -> behavior Added file: http://bugs.python.org/file45420/slots_qualname.patch ___ Python tracker

[issue23839] Clear caches after every test

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file45419/regrtest_clear_caches_3.patch ___ Python tracker ___

[issue28615] Document clarification: Section 5.4 Complex Number

Mariatta Wijaya added the comment: Hi, here is the doc update, to be applied to 2.7 branch. Please review. Thanks :) -- keywords: +patch nosy: +Mariatta Added file: http://bugs.python.org/file45418/issue28615.patch ___ Python tracker

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: I think Yury is right here. The good plan would be to add the tearDown(), but also fix the problem with test_generic_forward_ref. I could work on it tomorrow (really don't have time today, sorry). -- ___ Python

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: It looks like the tearDown() works really well. It also somehow fixes the problem with test_generic_forward_ref -- ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > Segfault or traceback? Traceback. I should have said failed, not crashed :( > I'm not sure a crash the Python version of lru_cache with maxsize=10 is high on my list of worries. The problem is that the test fails depending on cache size, and caching

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: > * It looks like that some types in the cache are being *reused* by different > tests. I also noticed this some time ago, ideally we need to make all tests more isolated between each other. > Now this all of this is just my guess of what's going on here.

[issue28649] refleak in typing.py

Guido van Rossum added the comment: Segfault or traceback? Anyway, it implements a doubly-linked list where each node is implemented as a list of 4 items... Maybe there's something in the GC code that recurses down at the C level??? I'm not sure a crash the Python version of lru_cache with

[issue28649] refleak in typing.py

Yury Selivanov added the comment: >> But why is this so different from the C implementation of lru_cache? > I don't know. Maybe C version of lru cache creates a bit less pressure on GC. Actually test/refleak.py now cleans up typing's lru cache & calls "gc.collect()". It seems that the typing

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > But why is this so different from the C implementation of lru_cache? I don't know. Maybe C version of lru cache creates a bit less pressure on GC. BTW, if I set maxsize=10 for typing lru_cache, test_generic_forward_ref crashes in refleak-test mode.

[issue28649] refleak in typing.py

Guido van Rossum added the comment: But why is this so different from the C implementation of lru_cache? -- ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: So Ivan made an interesting observation: if we use Python version of functools.lru_cache, typing tests start to leak like crazy: beginning 6 repetitions 123456 .. test_typing leaked [24980, 24980, 24980] references, sum=74940 I

[issue28653] refleak in C functools.lru_cache

Yury Selivanov added the comment: Ivan, I think I figured it out, I'll close this one and reopen the typing one. -- resolution: -> fixed stage: needs patch -> resolved status: open -> closed title: refleak in functools.lru_cache -> refleak in C functools.lru_cache

[issue28653] refleak in functools.lru_cache

Ivan Levkivskyi added the comment: I am not sure what does this mean, but when I remove __slots__ = () from some classes, numbers of leaked references change. -- ___ Python tracker

[issue28653] refleak in functools.lru_cache

Ivan Levkivskyi added the comment: > Wow. I can reproduce refleaks in test_typing, but not in test_functools. This > is weird. Indeed, I copied the new test from test_functools to test_typing, but now I see that pure Python version of lru_cache fails even if I remove this test and test that

[issue28653] refleak in functools.lru_cache

Yury Selivanov added the comment: > Anyway, this is not urgent Not so sure, a refleak in pure-python version kind of scares me. It can be a bug in the core. -- ___ Python tracker

[issue28653] refleak in functools.lru_cache

Changes by Yury Selivanov : -- stage: commit review -> needs patch ___ Python tracker ___

[issue28653] refleak in functools.lru_cache

Yury Selivanov added the comment: > It seems to be unrelated to typing.py Wow. I can reproduce refleaks in test_typing, but not in test_functools. This is weird. -- ___ Python tracker

[issue28653] refleak in functools.lru_cache

Ivan Levkivskyi added the comment: Anyway, this is not urgent, typing.py uses the default version, which is the C version. -- ___ Python tracker ___

[issue28653] refleak in functools.lru_cache

Ivan Levkivskyi added the comment: It seems to be unrelated to typing.py With your test from test_functools: def test_lru_type_error(self): @functools.lru_cache(maxsize=None) def infinite_cache(o): pass with self.assertRaises(TypeError):

[issue28653] refleak in functools.lru_cache

Yury Selivanov added the comment: > FWIW, if I comment out this code Oh boy... This is something else. Please re-open #28649 explaining that it now leaks with pure-Python lru_cache. Could you please take a look at it? -- ___ Python tracker

[issue28653] refleak in functools.lru_cache

Roundup Robot added the comment: New changeset ba59f3328032 by Yury Selivanov in branch '3.5': Issue #28653: Fix a refleak in functools.lru_cache. https://hg.python.org/cpython/rev/ba59f3328032 New changeset 5b253d641826 by Yury Selivanov in branch '3.6': Merge 3.6 (issue #28653)

[issue28653] refleak in functools.lru_cache

Ivan Levkivskyi added the comment: FWIW, if I comment out this code try: from _functools import _lru_cache_wrapper except ImportError: pass in functools.py to use the pure Python version, then I get much larger numbers: $ ./python -m test -R : test_typing Run tests sequentially

[issue28653] refleak in functools.lru_cache

Serhiy Storchaka added the comment: LGTM. -- assignee: -> yselivanov stage: patch review -> commit review ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: See http://bugs.python.org/issue28653 for details. -- ___ Python tracker ___

[issue28653] refleak in functools.lru_cache

New submission from Yury Selivanov: The attached patch fixes a refleak in functools.lru_cache. -- components: Library (Lib) files: refleak.patch keywords: patch messages: 280468 nosy: gvanrossum, larry, ned.deily, serhiy.storchaka, yselivanov priority: release blocker severity: normal

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Found it, will open a separate issue. -- resolution: -> fixed stage: needs patch -> resolved status: open -> closed ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: reflesk with only functools: def test_lru_type_error(self): @functools.lru_cache(maxsize=None) def foo(o): raise TypeError with self.assertRaises(TypeError): foo([]) --

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: A very simple repro: with self.assertRaises(TypeError): Union[[]] It looks like the problem happens when a non-hashable argument is passed to cached function -- ___ Python tracker

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: My impression is like something wrong happens when TypeError is raised by a cached function. -- ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Oh. This is a bug in C implementation of functools.lru_cache. I'll take a look. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Ivan, this is the part of the test that leaks: def test_extended_generic_rules_eq(self): T = TypeVar('T') U = TypeVar('U') with self.assertRaises(TypeError): Callable[[T], U][[], int] --

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: Yury, yes I will take a look at this now. -- ___ Python tracker ___ ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: The specific test that leaks is test_extended_generic_rules_eq -- ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: With all commits pushed, test_typing still seems to leak. Ivan, do you have time to see what's going on? test_typing leaked [125, 125, 125] references, sum=375 test_typing leaked [49, 49, 49] memory blocks, sum=147 --

[issue28649] refleak in typing.py

Roundup Robot added the comment: New changeset d926b484d33a by Serhiy Storchaka in branch '3.5': Issue #28649: Clear the typing module caches when search for reference leaks. https://hg.python.org/cpython/rev/d926b484d33a New changeset caf3ceb93307 by Serhiy Storchaka in branch '3.6': Issue

[issue28649] refleak in typing.py

Guido van Rossum added the comment: Serhiy can you apply the non-typing.py part of typing-clear-caches.patch? The diff I applied only has the typing.py part (because I've done this upstream first). -- ___ Python tracker

[issue24379] Add operator.subscript as a convenience for creating slices

Josh Rosenberg added the comment: #28651 opened for the general problem with empty __slots__ and gc failures. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Roundup Robot added the comment: New changeset d920bfa5a71a by Guido van Rossum in branch '3.5': Issue #28649: typing-clear-caches.patch https://hg.python.org/cpython/rev/d920bfa5a71a New changeset bd2ec9965f47 by Guido van Rossum in branch '3.6': Issue #28649: typing-clear-caches.patch

[issue28649] refleak in typing.py

Roundup Robot added the comment: New changeset d790078797bd by Guido van Rossum in branch '3.5': Issue #28649: fix-typing-test-v2.diff https://hg.python.org/cpython/rev/d790078797bd New changeset 43be7891b1f5 by Guido van Rossum in branch '3.6': Issue #28649: fix-typing-test-v2.diff (3.5->3.6)

[issue28556] typing.py upgrades

Roundup Robot added the comment: New changeset 9c0df5f51baa by Guido van Rossum in branch '3.5': Issue #28556: More typing.py updates from upstream. https://hg.python.org/cpython/rev/9c0df5f51baa New changeset 9e65bc305a24 by Guido van Rossum in branch '3.6': Issue #28556: More typing.py

[issue19717] resolve() fails when the path doesn't exist

Guido van Rossum added the comment: Thanks! -- ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue19717] resolve() fails when the path doesn't exist

Roundup Robot added the comment: New changeset 03bbee2b0d28 by Steve Dower in branch '3.6': Issue #19717: Makes Path.resolve() succeed on paths that do not exist (patch by Vajrasky Kok) https://hg.python.org/cpython/rev/03bbee2b0d28 New changeset 445415e402be by Steve Dower in branch

[issue19717] resolve() fails when the path doesn't exist

Steve Dower added the comment: Applied. I changed the default for the parameter and updated the docs, but the rest is as in the patch. -- assignee: -> steve.dower resolution: -> fixed stage: -> commit review ___ Python tracker

[issue24379] Add operator.subscript as a convenience for creating slices

Changes by Yury Selivanov : -- Removed message: http://bugs.python.org/msg280422 ___ Python tracker ___

[issue28652] Make loop methods reject socket kinds they do not support.

Roundup Robot added the comment: New changeset 85022978d900 by Yury Selivanov in branch '3.5': Issue #28652: Make loop methods reject socket kinds they do not support. https://hg.python.org/cpython/rev/85022978d900 New changeset 1273f1a3ddf7 by Yury Selivanov in branch '3.6': Merge 3.5 (issue

[issue28652] Make loop methods reject socket kinds they do not support.

Changes by Yury Selivanov : -- resolution: -> fixed status: open -> closed ___ Python tracker ___

[issue28652] Make loop methods reject socket kinds they do not support.

New submission from Yury Selivanov: Proxy for https://github.com/python/asyncio/pull/453 -- assignee: yselivanov components: asyncio messages: 280448 nosy: gvanrossum, yselivanov priority: normal severity: normal stage: resolved status: open title: Make loop methods reject socket kinds

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: Here is the corrected patch to avoid crash. -- Added file: http://bugs.python.org/file45416/fix-typing-test-v2.diff ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Serhiy's patch typing-clear-caches.patch looks good, we should apply it (fixing the code style a little bit). -- ___ Python tracker

[issue28651] Make objects with empty __slots__ GC types

Yury Selivanov added the comment: So this patch has no impact on typing.py. Moreover, it's incomplete and fails test_gc tests, which can be fixed but not in 3.6. -- versions: -Python 3.6 ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > Yury, _tp_cache should be the only one (of course most generic classes have > _abc_cache since GenericMeta inherits from ABCMeta) Then there is at least one more bug not related to lru_cache (or any kind of cache in typing). And it's not

[issue28649] refleak in typing.py

Guido van Rossum added the comment: > It fixes 96% of refleaks. In typing.py? Or generally? -- ___ Python tracker ___

[issue28649] refleak in typing.py

Guido van Rossum added the comment: FYI I've opened https://github.com/python/typing/issues/323 to track this upstream. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: Yury, _tp_cache should be the only one (of course most generic classes have _abc_cache since GenericMeta inherits from ABCMeta) -- ___ Python tracker

[issue28649] refleak in typing.py

Serhiy Storchaka added the comment: > Does that patch fix the refleak completely or only partially? It fixes 96% of refleaks. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > Does that patch fix the refleak completely or only partially? I tried to empty the cache in test_typing.BaseTestCase.setUp and tearDown; this doesn't fix all refleaks. Do we have some other caches in typing.py? --

[issue28649] refleak in typing.py

Ivan Levkivskyi added the comment: Yury, Guido, I think I understand why test_typing crashes in refleak hunting mode. The test relies on caching, namely type variables are compared by id, so that if cache is cleared between those two lines, then test fails. I think we should just update those

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > Good sleuthing! So the bug is in _lru_cache? I don't think it's a bug of lru_cache. It stores strong references for arguments and return value of the decorated function (which btw isn't documented). I don't think it's possible to write efficient generic

[issue28649] refleak in typing.py

Guido van Rossum added the comment: I have no objections against typing-clear-caches.patch (though we'll have to make sure to apply it to the GitHub upstream as well -- I can take care of that once you merge it to CPython). Does that patch fix the refleak completely or only partially? I have

[issue28651] Make objects with empty __slots__ GC types

Serhiy Storchaka added the comment: Needed tests. If an example with typing causes a leak (msg280422) this should be fixed in all versions that have typing. -- ___ Python tracker

[issue28649] refleak in typing.py

Serhiy Storchaka added the comment: > Serhiy, why is `gc.collect()` returning 2 after `del A` proof of a leak? Sorry, it was bad example. I don't remember all details. > The problem is that functools.lru_cache is used in _tp_cache. If I remove > type caching, the original "refleak" is fixed.

[issue23839] Clear caches after every test

Serhiy Storchaka added the comment: Synchronized with current code and added clearing typing caches. -- Added file: http://bugs.python.org/file45413/regrtest_clear_caches_2.patch ___ Python tracker

[issue19717] resolve() fails when the path doesn't exist

Guido van Rossum added the comment: I'd be very happy if that landed in 3.6 with the default strict=False. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Guido van Rossum added the comment: Good sleuthing! So the bug is in _lru_cache? On Wed, Nov 9, 2016 at 10:37 AM, Guido van Rossum wrote: > > Guido van Rossum added the comment: > > I could use a hint on a complete program that establishes whether there is > or is not

[issue28649] refleak in typing.py

Guido van Rossum added the comment: I could use a hint on a complete program that establishes whether there is or is not a refleak. -- ___ Python tracker

[issue19717] resolve() fails when the path doesn't exist

Steve Dower added the comment: Anyone have any major concerns with add_non_strict_resolve_pathlib_v4.patch? I'd be quite happy without adding the extra parameter to Path.resolve(), but I'm not strongly offended. >From Guido's email we should default to strict=False (i.e. don't throw if the

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Josh, Serhiy, I've created an issue for tracking empty slots types: #28651. This issue is about typing.py using functools.lru_cache for internal cache That is what causing the refleaks, and inability of test_typing.py to be run in refleak hunting mode.

[issue28651] Make objects with empty __slots__ GC types

New submission from Yury Selivanov: The attached patch does the trick. I'm not sure if this should be fixed in 3.6. -- components: Interpreter Core files: slots_gc.patch keywords: patch messages: 280427 nosy: gvanrossum, serhiy.storchaka, yselivanov priority: normal severity: normal

[issue28649] refleak in typing.py

Josh Rosenberg added the comment: Serhiy: I think you forgot to make a global instance of the type for your demonstration. You mentioned in msg253899 that the loop is: global instance -> type -> method -> module globals -> global instance The example you gave doesn't instantiate class A, and

[issue24379] Add operator.subscript as a convenience for creating slices

Yury Selivanov added the comment: Even more reduced test case: def test_refleak(self): T = typing.TypeVar('T') typing.Generic[T] -- nosy: +yselivanov ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: The problem is that functools.lru_cache is used in _tp_cache. If I remove type caching, the original "refleak" is fixed. -- ___ Python tracker

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Wow. In default CPython branch, with no modifications, test_typing.py simply crashes in debug mode: yury@ysmbp ~/dev/py/cpython $ ./python.exe -m test -R3:3 test_typing Run tests sequentially 0:00:00 [1/1] test_typing beginning 6 repetitions 123456 .test

[issue27584] New addition of vSockets to the python socket module

Cathy Avery added the comment: Please forgive the long delay in providing this update. I got a little sidetracked. Attached is the patch for Python 3.7. It includes fixes suggested in rev 1 plus VSOCK tests in test_socket.py. Thanks, Cathy -- versions: +Python 3.7 -Python 3.6 Added

[issue28649] refleak in typing.py

Yury Selivanov added the comment: I removed all slots from typing.py, the refleak is still there. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Yury Selivanov added the comment: Attached patch (slots_gc.patch) makes objects with empty slots tracked by GC: class A: __slots__ = () gc.is_tracked(A()) == False # before gc.is_tracked(A()) == True # with the patch This doesn't fix the refleak though. -- keywords: +patch

[issue24329] __qualname__ and __slots__

Yury Selivanov added the comment: Because the following works without a problem: class F: __slots__ = ('__name__', ) f = F() f.__name__ = '' This bug with __qualname__ is why _GeneratorWrapper in types.py doesn't have __slots__. --

[issue24329] __qualname__ and __slots__

Xiang Zhang added the comment: Why should it work Yury? __qualname__ and __doc__(if exists) are inserted into the dict when creating a class. >>> class Foo: ... """bar""" ... __slots__ = ('__doc__',) ... Traceback (most recent call last): File "", line 1, in ValueError: '__doc__'

[issue28649] refleak in typing.py

Guido van Rossum added the comment: Serhiy, why is `gc.collect()` returning 2 after `del A` proof of a leak? Yes, there's a cycle, but it's being GC'ed -- isn't that fine? Without the slots the collect() call returns a larger number. -- ___ Python

[issue28643] Broken makefile depends for profile-opt target

Changes by Brett Cannon : -- stage: needs patch -> patch review ___ Python tracker ___ ___

[issue28643] Broken makefile depends for profile-opt target

Changes by Brett Cannon : -- nosy: +brett.cannon, gregory.p.smith ___ Python tracker ___

[issue24329] __qualname__ and __slots__

Yury Selivanov added the comment: Serhiy, maybe you know how to fix this? -- nosy: +serhiy.storchaka versions: +Python 3.7 -Python 3.5 ___ Python tracker

[issue28649] refleak in typing.py

Serhiy Storchaka added the comment: See also https://mail.python.org/pipermail/python-dev/2015-October/141993.html. -- ___ Python tracker ___

[issue28649] refleak in typing.py

Serhiy Storchaka added the comment: >>> import gc >>> gc.collect() 23 >>> gc.collect() 0 >>> class A: ... __slots__ = () ... >>> del A >>> gc.collect() 2 ^ leak -- ___ Python tracker

[issue28649] refleak in typing.py

Guido van Rossum added the comment: Ivan, this was an early typing.py update (merged into CPython on Sept. 27). But the refleak is still with us. Do you have any intuition on what could be going on here? My own intuition here is lacking, other than thinking there's a lot of metaclass magic

[issue28649] refleak in typing.py

Yury Selivanov added the comment: > Could it be related to leaks in objects with empty __slots__? See issue24379 > for details. Not sure... Do we have a test/patch for objects-with-empty-slots bug? -- ___ Python tracker

[issue28649] refleak in typing.py

Serhiy Storchaka added the comment: Could it be related to leaks in objects with empty __slots__? See issue24379 for details. -- ___ Python tracker ___

[issue28650] concurrent.futures.ThreadPoolExecutor: tasks in queue not marked as done

New submission from pyfm: Hi, I just realized that the ThreadPoolExecutor's workers do not call task_done() on the work_queue from which they took their task. Or is there a reason for not calling task_done that I am missing? Calling task_done decrements the queue's unfinished_tasks counter

[issue28637] Python startup performance regression

Guido van Rossum added the comment: Honestly now the basic startup time is under control I am fine with restoring the re enum. We can optimize that later. --Guido (mobile) -- ___ Python tracker

[issue27854] Installed 2.7: IDLE Help disabled because help.html is missing

Steve Dower added the comment: Tools/msi/msi.py has a whole lot of code to explicitly include certain files. Probably just needs a line (or a glob) added into the idlelib section -- ___ Python tracker

[issue28649] refleak in typing.py

New submission from Yury Selivanov: Looks like we have a refleak somewhere in the system that typing.py is exposing. The following code exposes the refleak: def test_refleak(self): T = typing.TypeVar('T') class C(typing.Generic[T], typing.Mapping[int, str]): ... The change

[issue28648] False assert in _Py_DecodeUTF8_surrogateescape

New submission from Xiang Zhang: The assert statement `assert(Py_UNICODE_IS_SURROGATE(ch));` in _Py_DecodeUTF8_surrogateescape is wrong. Code points > 0x could reach it and fail. -- files: false_assert.patch keywords: patch messages: 280406 nosy: serhiy.storchaka, xiang.zhang

[issue28647] python --help: -u is misdocumented as binary mode

STINNER Victor added the comment: Would it make sense to have two modes: line buferred and unbuffered, as the C function setvbuf() for stdout? -- ___ Python tracker

[issue27942] Default value identity regression

Serhiy Storchaka added the comment: > The test that tests it all :) test_stdlib_validates :) -- ___ Python tracker ___

[issue27942] Default value identity regression

Roundup Robot added the comment: New changeset 41613bb27f80 by Yury Selivanov in branch '2.7': Issue #27942: Fix memory leak in codeobject.c https://hg.python.org/cpython/rev/41613bb27f80 New changeset 2c6825c9ecfd by Yury Selivanov in branch '3.5': ssue #27942: Fix memory leak in codeobject.c

[issue27942] Default value identity regression

Yury Selivanov added the comment: Thanks for the review, Serhiy! -- priority: release blocker -> normal resolution: -> fixed status: open -> closed ___ Python tracker

[issue27942] Default value identity regression

Yury Selivanov added the comment: > Interesting, what tests in test_ast leak? I expected that this branch is > executed in rare circumstances. The test that tests it all :) test_stdlib_validates -- ___ Python tracker

[issue27942] Default value identity regression

Serhiy Storchaka added the comment: Good catch! The patch LGTM, thanks Yury. Interesting, what tests in test_ast leak? I expected that this branch is executed in rare circumstances. -- ___ Python tracker

[issue27942] Default value identity regression

Yury Selivanov added the comment: Looks like the attached patch also fixes test_trace leaked [12, 12, 12] references, sum=36 -- ___ Python tracker

[issue28635] Update What's New for 3.6

Yury Selivanov added the comment: Elvis, please take a look at http://bugs.python.org/issue28641 -- ___ Python tracker ___

  1   2   >