[issue28147] Unbounded memory growth resizing split-table dicts

2017-03-31 Thread Donald Stufft

Changes by Donald Stufft :


--
pull_requests: +1001

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-19 Thread INADA Naoki

Changes by INADA Naoki :


--
resolution:  -> fixed
stage: commit review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-19 Thread Roundup Robot

Roundup Robot added the comment:

New changeset cc40470c10f8 by INADA Naoki in branch '3.5':
Issue #28147: Fix a memory leak in split-table dictionaries
https://hg.python.org/cpython/rev/cc40470c10f8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-19 Thread STINNER Victor

STINNER Victor added the comment:

The 3.5 patch LGTM.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-19 Thread INADA Naoki

Changes by INADA Naoki :


Added file: http://bugs.python.org/file45968/fix-28147-py35-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-17 Thread INADA Naoki

INADA Naoki added the comment:

Before Python 3.6, instance.__dict__.pop(key) didn't trigger this.
But there are many way to trigger this. This bug is live from 3.3.
This script reproduce this issue on Python 3.5.

class C: pass
a = C()
a.a = 1
while True:
a.__dict__.popitem()  # convert split table into combined table.
a.a = 1   # convert combined table into split table again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-16 Thread Ned Deily

Ned Deily added the comment:

So now that a fix has been released with 3.6.0rc2, what else needs to be done 
to close this issue?  Why is 3.5 selected in the applicable versions?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-16 Thread STINNER Victor

STINNER Victor added the comment:

> Oh, I'm sorry.  It seems I had failed to push the commit yesterday.

No problem. Thanks for your fix! I prefer to not see the bug in Python
3.6.0 final!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-16 Thread INADA Naoki

INADA Naoki added the comment:

Oh, I'm sorry.  It seems I had failed to push the commit yesterday.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Ned Deily

Ned Deily added the comment:

[cherrypicked for 3.6.0rc2]

--
priority: release blocker -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Roundup Robot

Roundup Robot added the comment:

New changeset b70b2d3f3167 by Victor Stinner in branch '3.6':
Fix a memory leak in split-table dictionaries
https://hg.python.org/cpython/rev/b70b2d3f3167

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

I asked a question about the change in _PyObjectDict_SetItem. It is not clear 
to me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

I think this can be tested without _testcapi. See an example in issue28894.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread STINNER Victor

STINNER Victor added the comment:

I dislike pushing a change written by another core dev, I prefer that he/she 
directly push it, but we are out of time. Ned asked to first push to Python 3.6 
to get a confirmation of buildbots, before being able to ask for a cherry-pick 
in 3.6.0 final.

Anyway, thanks for the fix Naoki!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Roundup Robot

Roundup Robot added the comment:

New changeset 85be9dcc16a8 by Victor Stinner in branch '3.6':
Fix a memory leak in split-table dictionaries
https://hg.python.org/cpython/rev/85be9dcc16a8

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread INADA Naoki

INADA Naoki added the comment:

This patch updates the comment.

--
Added file: http://bugs.python.org/file45911/fix28147-comment-update.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread STINNER Victor

STINNER Victor added the comment:

fix28147-py36-3.patch LGTM: Naoki, please push it right now.

But please see also my comments on fix28147-py36-2.patch: I would still 
apprecicate that we enhance the comment in dictobject.c. The comment is not a 
blocker issue for 3.6.0, it can be enhanced later :-D

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread STINNER Victor

STINNER Victor added the comment:

Ned: "If so, can someone please review it?"

Done.


Ned: "I am very reluctant to delay now for an issue that has been open now for 
3 months throughout the beta phase of the release cycle.  If enough people feel 
we should delay 3.6.0 to address this, we can."

The bug was seen as minor and I didn't expect that anyone would notice it.

Wenzel Jakob wrote "RSS goes to 43MB to 4.3GB": for me, it's a major 
regression, and it's not possible to work around it. With such bug, Python 3.6 
is unusable on such application (pybind11). I easily imagine that such memory 
leak is a blocker issue for a server running for days.

I now consider that Python 3.6.0 must not be released with such blocker issue.

--
nosy: +larry
priority: deferred blocker -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread STINNER Victor

STINNER Victor added the comment:

I reviewed fix28147-py36-2.patch, I have some requests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread INADA Naoki

Changes by INADA Naoki :


Added file: http://bugs.python.org/file45910/fix28147-py36-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread STINNER Victor

STINNER Victor added the comment:

fix28147-py36-2.patch: test_dict pass with DEBUG_PYDICT defined.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Ned Deily

Changes by Ned Deily :


--
priority: high -> deferred blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-15 Thread Ned Deily

Ned Deily added the comment:

It appears this issue has stalled again.  Is the most recent patch 
fix28147-py36-2.patch) ready for review?  If so, can someone please review it?  
Then it needs to be pushed to the 3.6 branch for 3.6.1 and exposed to the 
buildbots.  Only then could we consider cherrypicking for 3.6.0 and a change of 
the magnitude in the patch would require at least another preview release and 
delay 3.6.0 final. 3.6.0 is scheduled to be released tomorrow. I am very 
reluctant to delay now for an issue that has been open now for 3 months 
throughout the beta phase of the release cycle.  If enough people feel we 
should delay 3.6.0 to address this, we can.  But without a fix reviewed and 
committed and without more feedback from other core developers, 3.6.0 is going 
out without it.  @inada.naoki ? @haypo ? @serhiy.storchaka ?  Others?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-14 Thread Wenzel Jakob

Wenzel Jakob added the comment:

Hi,

pybind11 (https://github.com/pybind/pybind11) dev here.

We're seeing massive memory increases due to this bug, which completely break 
our test suite (and likely any use of this library, which is commonly used to 
bind C++ code to Python).

Please look at the following issue ticket:

https://github.com/pybind/pybind11/issues/558

where RSS goes to 43MB to 4.3GB for the basic set of tests. The fancy fancy 
test suite which also covers NumPy/SciPy can't even be run anymore. All issues 
disappear when the dict patch listed here is applied.

We're really concerned that this is not slated to be fixed for 3.6.0. Pybind11 
is not doing anything particularly special with dicts -- if this is hitting us, 
others will likely have issues as well.

-Wenzel

--
nosy: +wenzel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-08 Thread Ned Deily

Ned Deily added the comment:

I think these proposed patches need careful review but, even so, given the 
extent of the fix28147-py36-2.patch changes, I don't think they should go into 
3.6.0 at the last minute.  Unless somebody can convince me otherwise, I'm going 
to let this wait for 3.6.1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-08 Thread Jason Madden

Changes by Jason Madden :


--
nosy: +jmadden

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-08 Thread INADA Naoki

INADA Naoki added the comment:

http://bugs.python.org/issue28894 was duplicate of this issue.

Since real world program suffered by this, I'm +1 to fix this by 3.6.0

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-08 Thread INADA Naoki

Changes by INADA Naoki :


Added file: http://bugs.python.org/file45797/fix28147-py36-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-08 Thread INADA Naoki

INADA Naoki added the comment:

Here is patch for Python 3.6.

--
Added file: http://bugs.python.org/file45796/fix28147-py36.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread Ned Deily

Ned Deily added the comment:

Just follow the normal 3.6 branch which will ensure it gets into 3.6.1.  If we 
end up deciding to also add it to 3.6.0 final, I will handle the cherrypicking.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread INADA Naoki

INADA Naoki added the comment:

Which revision should I make patch based on? 3.6 branch? or 3.6rc1 tag?

After 3.6rc1 is tagged, I pushed optimization which contains same one line 
change.

https://hg.python.org/cpython/rev/d03562dcbb82

-#define ESTIMATE_SIZE(n)  (((n)*3) >> 1)
+#define ESTIMATE_SIZE(n)  (((n)*3+1) >> 1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread Min RK

Min RK added the comment:

This affects IPython (specifically the traitlets component), which is what 
prompted the report. We were able to push out a release of traitlets with a 
workaround for the bug (4.3.1), but earlier versions of IPython / traitlets 
will still be affected (all IPython >= 4, traitlets 4.0 <= v < 4.3.1). So I 
hope 3.6.0 will be released with the fix attached here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread Xiang Zhang

Xiang Zhang added the comment:

__dict__.pop seems not uncommon. Searching Github could give many practical 
codes using it. And many of them are acting as generic containers and could be 
used against massive cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread STINNER Victor

STINNER Victor added the comment:

In this case, I suggest to wait for 3.6.1 to fix it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread INADA Naoki

INADA Naoki added the comment:

> Ned Deily added the comment:
>
> This issue seems to have slipped through. Should it be a release blocker for 
> 3.6.0 final or can it wait for 3.6.1?

On Python 3.5, instance.__dict__.popitem() cause this issue.
On Python 3.6, instance.__dict__.popitem() and instance.__dict__.pop()
cause this issue.

This is not new issue, but there is small regression.

I don't know this should be fixed in 3.6.0.
I'll make patch smaller anyway.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread Serhiy Storchaka

Serhiy Storchaka added the comment:

Calling pop() for instance's __dict__ is not common operation. I expected that 
this bug does not affect any real code. But the case in issue28894 looks as a 
real case. This might be a release blocker.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-12-07 Thread Ned Deily

Ned Deily added the comment:

This issue seems to have slipped through. Should it be a release blocker for 
3.6.0 final or can it wait for 3.6.1?

--
nosy: +ned.deily

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-11-06 Thread Serhiy Storchaka

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-10-25 Thread INADA Naoki

Changes by INADA Naoki :


--
priority: normal -> high
stage: patch review -> commit review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-28 Thread INADA Naoki

INADA Naoki added the comment:

haypo, Could you review fix-28147-py35.patch and fix-28147.patch ?

Draft NEWS entry:

- Issue #28147: Fixed split table dict may consume unlimited memory.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-26 Thread INADA Naoki

Changes by INADA Naoki :


--
versions: +Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-18 Thread Xiang Zhang

Xiang Zhang added the comment:

Ahh, I see.

> If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and 
> USABLE_FRACTION(n) == a - 1.

There are, such as 11, 43...

> a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a))

It can but needs another resize in insertdict which breaks the intention of 
ESTIMATE_SIZE.

Then everything looks fine to me. :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-18 Thread INADA Naoki

INADA Naoki added the comment:

xiang:
dictresize(d, n) may choose keysize==n (when n == 2**m) with this patch.

If there are any integer a such as ESTIMATE_SIZE(a) == n and n == 2**m and 
USABLE_FRACTION(n) == a - 1,
a items cannot be inserted into dict after dictresize(d, ESTIMATE_SIZE(a))

This is why ESTIMATE_SIZE should round up fraction.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-18 Thread Xiang Zhang

Xiang Zhang added the comment:

LGTM. But why this change?

-#define ESTIMATE_SIZE(n)  (((n)*3) >> 1)
+#define ESTIMATE_SIZE(n)  (((n)*3+1) >> 1)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-16 Thread INADA Naoki

INADA Naoki added the comment:

This is patch for Python 3.5.
The patch uses more conservative approach.

--
Added file: http://bugs.python.org/file44696/fix-28147-py35.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread INADA Naoki

INADA Naoki added the comment:

xiang is right. Python 3.5 has same issue when using popitem().
I'll make patch for 3.5.  But it will be bit differ from patch for 3.6 and they 
will conflict.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread INADA Naoki

Changes by INADA Naoki :


Added file: http://bugs.python.org/file44666/fix-28147.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread INADA Naoki

INADA Naoki added the comment:

This issue is caused by dictresize() and _PyObjectDict_SetItem()

1. a.__dict__.pop('a') convert the dict to combined table which has double 
keysize.
2. a.a = 1  converts the dict to split table again if there are no instances 
sharing key with class.

As I wrote before, pop, popitem, and del should not increase key size.

And _PyObjectDict_SetItem shouldn't convert the dict to split-table when the 
dict doesn't share keys
with the class before calling PyDict_SetItem.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread INADA Naoki

Changes by INADA Naoki :


Added file: http://bugs.python.org/file44665/fix-28147.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread Xiang Zhang

Xiang Zhang added the comment:

> popitem() before does the same thing and should never cause a problem.

Ahh, this seems not true. Test it in py3.5 also crash.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread Xiang Zhang

Xiang Zhang added the comment:

> I confirmed and investigated it.  Thanks!
I'll post patch including more test in 24 hours.

Please wait a second.

The cause of this problem is that attribute setting causes a combined table to 
be splitted (the code path is [0]->[1]->[2]->[3]). So every time you set an 
attribute and then pop, it will do dictresize(right now it will increase memory 
usage). So even if you make pop() not increase memory usage, the example Min 
gives will still do much work not wanted (dictresize and make_keys_shared).

Actually I think memory usage increasing is not a problem if other things are 
right. popitem() before does the same thing and should never cause a problem.
 
[0] https://hg.python.org/cpython/file/tip/Python/ceval.c#l2248
[1] https://hg.python.org/cpython/file/tip/Objects/object.c#l1119
[2] https://hg.python.org/cpython/file/tip/Objects/object.c#l1125
[3] https://hg.python.org/cpython/file/tip/Objects/object.c#l1172

--
nosy: +xiang.zhang

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread INADA Naoki

INADA Naoki added the comment:

I confirmed and investigated it.  Thanks!
I'll post patch including more test in 24 hours.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread Min RK

Min RK added the comment:

I pulled just now and saw changes in dictobject.c, and just wanted to confirm 
the memory growth bug is still in changeset 56294e03ad89 (I think I used the 
right hash, this time).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com



[issue28147] Unbounded memory growth resizing split-table dicts

2016-09-14 Thread Min RK

Min RK added the comment:

> Ah, is the leak happen in 3.6b1?

The leak happens in 3.6b1 and master as of an hour ago (git: 
3c06edfe9463f1cf81bc34b702f165ad71ff79b8, hg:r103797)

--
title: Memory leak in new 3.6 dictionary resize -> Unbounded memory growth 
resizing split-table dicts

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com