[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

> What is left to do with this issue?

Nothing that I can see.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What is left to do with this issue?

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
priority: release blocker -> normal
resolution: fixed -> 
status: closed -> open
versions:  -Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Anselm Kruis]

Anselm Kruis added the comment:

>> why the different fix for 3.3 
>
> I reverted the 2.7.4 addition of __dict__ rather than introduce more
> differences between point releases with possible unintended effects.

__dict__ was a 2.7.3 addition (changeset 26d5f022eb1a). Now unpickling of named 
tuples created by 2.7.3 and 2.7.4 fails.

--
nosy: +anselm.kruis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 31eaf8a137ea by Georg Brandl in branch '3.2':
Issue #15535: Fix pickling of named tuples.
http://hg.python.org/cpython/rev/31eaf8a137ea

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Barry A. Warsaw]

Changes by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

> why the different fix for 3.3 

I reverted the 2.7.4 addition of __dict__ rather than introduce more 
differences between point releases with possible unintended effects.

In 3.3, the __dict__ attribute was there from the outset and was advertised in 
the docs, so it made more sense to leave it in and just suppress its inclusion 
in pickling.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Ben Hoyt]

Ben Hoyt added the comment:

2.7 fix works for me, thanks! Just curious -- why the different fix for 3.3 
(addition of __getstate__ instead of removal of __dict__)?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee: amaury.forgeotdarc -> rhettinger
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg188064

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 65cd71abebc8 by Raymond Hettinger in branch '3.3':
Issue #15535:  Fix pickling of named tuples.
http://hg.python.org/cpython/rev/65cd71abebc8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Roundup Robot]

Roundup Robot added the comment:

New changeset 18303391b981 by Raymond Hettinger in branch '2.7':
Issue #15535:  Fix regression in pickling of named tuples.
http://hg.python.org/cpython/rev/18303391b981

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Florent Xicluna]

Changes by Florent Xicluna :


--
nosy: +flox

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
Removed message: http://bugs.python.org/msg188063

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

Amaury, please go ahead and apply your patch.

--
assignee: rhettinger -> amaury.forgeotdarc

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

[Antoine]
> I would like to call this a critical regression.

I concur.  
I will post the suggested fix with tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Georg Brandl]

Changes by Georg Brandl :


--
versions: +Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +georg.brandl, larry
stage:  -> needs patch
versions: +Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Antoine Pitrou]

Antoine Pitrou added the comment:

I would like to call this a critical regression.
Under 2.7 and 3.2, all pickle protocols are affected.
Under 3.3 and 3.4, pickle protocols 0 and 1 are affected.

(unfortunately, 3.2 doesn't receive bugfixes anymore)

--
nosy: +benjamin.peterson, pitrou
priority: high -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
priority: normal -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Ben Hoyt]

Ben Hoyt added the comment:

I just hit this issue in a big way -- would have been nice for this fix to go 
into Python 2.7.4. :-)

It was quite hard to track down (as in, a day or two of debugging :-) because 
the symptoms didn't point directly to namedtuple. In our setup we 
pickle/unpickle some big files, and the symptoms we noticed were extremely high 
memory usage after *un*pickling -- as in, 3x what we were getting before 
upgrading from Python 2.6. We first tracked it down to unpickling, and then 
from there narrowed it down to namedtuple.

The first "fix" I discovered was that I could use pickletools.optimize() to 
reduce the memory-usage-after-unpickling back down to sensible levels. I don't 
know enough about pickle to know exactly why this is -- perhaps fragmentation 
due to extra unpickling data structures allocated on the heap, that optimize() 
removes?

Here's the memory usage of our Python process after unpickling a ~9MB pickle 
file (protocol 2) which includes a lot of namedtuples. This is on Python 2.7.4 
64-bit. With the original collections.py -- "normal" means un-optimized pickle, 
"optimized" means run through pickletools.optimize():

Memory usage after loading normal: 106664 KB
Memory usage after loading optimized: 31424 KB

With collections.py modified so namedtuple's templates include "def 
__getstate__(self): return None":

Memory usage after loading normal: 33676 KB
Memory usage after loading optimized: 26392 KB

So you can see the Python 2.7 version of namedtuple makes the process use 
basically 3x the RAM when unpickled (without pickletools.optimize). Note that 
Python 2.6 does *not* do this (it doesn't define __dict__ or use OrderedDict so 
doesn't have this issue). And for some Python 3.3(.1) doesn't have the issue 
either, even though that does define __dict__ and use OrderedDict. I guess 
Python 3.3 does pickling (or garbage collection?) somewhat differently.

You can verify this yourself using the attached unpickletest.py script. Note 
that I'm running on Windows 7, but I presume this would happen on Linux/OS X 
too, as this issue has nothing to do with the OS. The script should work on 
non-Windows OSes, but you have to type in the RAM usage figures manually (using 
"top" or similar).

Note that I'm doing a gc.collect() just before fetching the memory usage figure 
just in case there's uncollected cyclical garbage floating around, and I didn't 
want that to affect the measurement.

I'm not sure I fully understand the cause (of where all this memory is going), 
or the fix for that matter. The OrderedDict is being pickled along with the 
namedtuple instance, because an OrderedDict is returned by __dict__, and pickle 
uses that. But is that staying in memory on unpickling? Why does optimizing the 
pickle fix the RAM usage issue to a large extent?

In any case, I've made the __getstate__ fix in our code, and that definitely 
fixes the RAM usage for us. (We're also going to be optimizing our pickles from 
now on.)

--
nosy: +benhoyt
Added file: http://bugs.python.org/file30021/unpickletest.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Jesús Cea Avión]

Changes by Jesús Cea Avión :


--
nosy: +jcea

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc added the comment:

Adding "def __getstate__(self): return None" to the namedtuple template fixes 
the issue.  Here is a patch with test.

--
nosy: +amaury.forgeotdarc
Added file: http://bugs.python.org/file27131/namedtuple-pickle.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15535] Fix pickling efficiency of named tuples in 2.7.3

[Raymond Hettinger]

Raymond Hettinger added the comment:

FWIW, all pickle protocol levels are affected:

Point = colletions.namedtuple('Point', ['x', 'y', 'z'])
for proto in range(3):
pickletools.dis(dumps(Point(10, 20, 30), proto))

I'll look at the proposed fix in more detail when I get a chance -- we want to 
make sure that subclasses aren't adversely affected and that there aren't any 
other unintended side-effects.

--
title: Fix pickling of named tuples in 2.7.3 (BUG) -> Fix pickling  efficiency 
of named tuples in 2.7.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com