[issue28147] Unbounded memory growth resizing split-table dicts

Changes by Donald Stufft : -- pull_requests: +1001 ___ Python tracker ___ ___

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Roundup Robot added the comment: New changeset cc40470c10f8 by INADA Naoki in branch '3.5': Issue #28147: Fix a memory leak in split-table dictionaries https://hg.python.org/cpython/rev/cc40470c10f8 -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: The 3.5 patch LGTM. -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : Added file: http://bugs.python.org/file45968/fix-28147-py35-2.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: Before Python 3.6, instance.__dict__.pop(key) didn't trigger this. But there are many way to trigger this. This bug is live from 3.3. This script reproduce this issue on Python 3.5. class C: pass a = C() a.a = 1 while True: a.__dict__.popitem() # convert

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: So now that a fix has been released with 3.6.0rc2, what else needs to be done to close this issue? Why is 3.5 selected in the applicable versions? -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: > Oh, I'm sorry. It seems I had failed to push the commit yesterday. No problem. Thanks for your fix! I prefer to not see the bug in Python 3.6.0 final! -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: Oh, I'm sorry. It seems I had failed to push the commit yesterday. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: [cherrypicked for 3.6.0rc2] -- priority: release blocker -> deferred blocker ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Roundup Robot added the comment: New changeset b70b2d3f3167 by Victor Stinner in branch '3.6': Fix a memory leak in split-table dictionaries https://hg.python.org/cpython/rev/b70b2d3f3167 -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Serhiy Storchaka added the comment: I asked a question about the change in _PyObjectDict_SetItem. It is not clear to me. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Serhiy Storchaka added the comment: I think this can be tested without _testcapi. See an example in issue28894. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: I dislike pushing a change written by another core dev, I prefer that he/she directly push it, but we are out of time. Ned asked to first push to Python 3.6 to get a confirmation of buildbots, before being able to ask for a cherry-pick in 3.6.0 final.

[issue28147] Unbounded memory growth resizing split-table dicts

Roundup Robot added the comment: New changeset 85be9dcc16a8 by Victor Stinner in branch '3.6': Fix a memory leak in split-table dictionaries https://hg.python.org/cpython/rev/85be9dcc16a8 -- nosy: +python-dev ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: This patch updates the comment. -- Added file: http://bugs.python.org/file45911/fix28147-comment-update.patch ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: fix28147-py36-3.patch LGTM: Naoki, please push it right now. But please see also my comments on fix28147-py36-2.patch: I would still apprecicate that we enhance the comment in dictobject.c. The comment is not a blocker issue for 3.6.0, it can be enhanced

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: Ned: "If so, can someone please review it?" Done. Ned: "I am very reluctant to delay now for an issue that has been open now for 3 months throughout the beta phase of the release cycle. If enough people feel we should delay 3.6.0 to address this, we can."

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: I reviewed fix28147-py36-2.patch, I have some requests. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : Added file: http://bugs.python.org/file45910/fix28147-py36-3.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: fix28147-py36-2.patch: test_dict pass with DEBUG_PYDICT defined. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by Ned Deily : -- priority: high -> deferred blocker ___ Python tracker ___ ___

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: It appears this issue has stalled again. Is the most recent patch fix28147-py36-2.patch) ready for review? If so, can someone please review it? Then it needs to be pushed to the 3.6 branch for 3.6.1 and exposed to the buildbots. Only then could we consider

[issue28147] Unbounded memory growth resizing split-table dicts

Wenzel Jakob added the comment: Hi, pybind11 (https://github.com/pybind/pybind11) dev here. We're seeing massive memory increases due to this bug, which completely break our test suite (and likely any use of this library, which is commonly used to bind C++ code to Python). Please look at

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: I think these proposed patches need careful review but, even so, given the extent of the fix28147-py36-2.patch changes, I don't think they should go into 3.6.0 at the last minute. Unless somebody can convince me otherwise, I'm going to let this wait for 3.6.1.

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by Jason Madden : -- nosy: +jmadden ___ Python tracker ___ ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: http://bugs.python.org/issue28894 was duplicate of this issue. Since real world program suffered by this, I'm +1 to fix this by 3.6.0 -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : Added file: http://bugs.python.org/file45797/fix28147-py36-2.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: Here is patch for Python 3.6. -- Added file: http://bugs.python.org/file45796/fix28147-py36.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: Just follow the normal 3.6 branch which will ensure it gets into 3.6.1. If we end up deciding to also add it to 3.6.0 final, I will handle the cherrypicking. -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: Which revision should I make patch based on? 3.6 branch? or 3.6rc1 tag? After 3.6rc1 is tagged, I pushed optimization which contains same one line change. https://hg.python.org/cpython/rev/d03562dcbb82 -#define ESTIMATE_SIZE(n) (((n)*3) >> 1) +#define

[issue28147] Unbounded memory growth resizing split-table dicts

Min RK added the comment: This affects IPython (specifically the traitlets component), which is what prompted the report. We were able to push out a release of traitlets with a workaround for the bug (4.3.1), but earlier versions of IPython / traitlets will still be affected (all IPython >=

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: __dict__.pop seems not uncommon. Searching Github could give many practical codes using it. And many of them are acting as generic containers and could be used against massive cases. -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

STINNER Victor added the comment: In this case, I suggest to wait for 3.6.1 to fix it. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: > Ned Deily added the comment: > > This issue seems to have slipped through. Should it be a release blocker for > 3.6.0 final or can it wait for 3.6.1? On Python 3.5, instance.__dict__.popitem() cause this issue. On Python 3.6, instance.__dict__.popitem() and

[issue28147] Unbounded memory growth resizing split-table dicts

Serhiy Storchaka added the comment: Calling pop() for instance's __dict__ is not common operation. I expected that this bug does not affect any real code. But the case in issue28894 looks as a real case. This might be a release blocker. -- ___

[issue28147] Unbounded memory growth resizing split-table dicts

Ned Deily added the comment: This issue seems to have slipped through. Should it be a release blocker for 3.6.0 final or can it wait for 3.6.1? -- nosy: +ned.deily ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : -- priority: normal -> high stage: patch review -> commit review ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: haypo, Could you review fix-28147-py35.patch and fix-28147.patch ? Draft NEWS entry: - Issue #28147: Fixed split table dict may consume unlimited memory. -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : -- versions: +Python 3.5 ___ Python tracker ___ ___

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: Ahh, I see. > If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and > USABLE_FRACTION(n) == a - 1. There are, such as 11, 43... > a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a)) It can but needs another resize in

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: xiang: dictresize(d, n) may choose keysize==n (when n == 2**m) with this patch. If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and USABLE_FRACTION(n) == a - 1, a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a))

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: LGTM. But why this change? -#define ESTIMATE_SIZE(n) (((n)*3) >> 1) +#define ESTIMATE_SIZE(n) (((n)*3+1) >> 1) -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: This is patch for Python 3.5. The patch uses more conservative approach. -- Added file: http://bugs.python.org/file44696/fix-28147-py35.patch ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: xiang is right. Python 3.5 has same issue when using popitem(). I'll make patch for 3.5. But it will be bit differ from patch for 3.6 and they will conflict. -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : Added file: http://bugs.python.org/file44666/fix-28147.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: This issue is caused by dictresize() and _PyObjectDict_SetItem() 1. a.__dict__.pop('a') convert the dict to combined table which has double keysize. 2. a.a = 1 converts the dict to split table again if there are no instances sharing key with class. As I wrote

[issue28147] Unbounded memory growth resizing split-table dicts

Changes by INADA Naoki : Added file: http://bugs.python.org/file44665/fix-28147.patch ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: > popitem() before does the same thing and should never cause a problem. Ahh, this seems not true. Test it in py3.5 also crash. -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Xiang Zhang added the comment: > I confirmed and investigated it. Thanks! I'll post patch including more test in 24 hours. Please wait a second. The cause of this problem is that attribute setting causes a combined table to be splitted (the code path is [0]->[1]->[2]->[3]). So every time you

[issue28147] Unbounded memory growth resizing split-table dicts

INADA Naoki added the comment: I confirmed and investigated it. Thanks! I'll post patch including more test in 24 hours. -- ___ Python tracker ___

[issue28147] Unbounded memory growth resizing split-table dicts

Min RK added the comment: I pulled just now and saw changes in dictobject.c, and just wanted to confirm the memory growth bug is still in changeset 56294e03ad89 (I think I used the right hash, this time). -- ___ Python tracker

[issue28147] Unbounded memory growth resizing split-table dicts

Min RK added the comment: > Ah, is the leak happen in 3.6b1? The leak happens in 3.6b1 and master as of an hour ago (git: 3c06edfe9463f1cf81bc34b702f165ad71ff79b8, hg:r103797) -- title: Memory leak in new 3.6 dictionary resize -> Unbounded memory growth resizing split-table dicts