Alexandre Vassalotti alexan...@peadrop.com added the comment:
Too late to make this change in 2.x. And the patch in issue 9410 includes the
optimization for 3.x.
--
resolution: - duplicate
stage: patch review - committed/rejected
status: open - closed
superseder: - Add Unladen
Changes by Skip Montanaro s...@pobox.com:
--
nosy: -skip.montanaro
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
___
Python-bugs-list
Skip Montanaro s...@pobox.com added the comment:
You can fix it if you are dumping to a file, however if you are calling dumps()
you are kind of screwed if dumping large objects. There's no place to flush
the buffer.
I have a fix to Unladen Swallow's cPickle module. I'm run it by them
Antoine Pitrou pit...@free.fr added the comment:
Do we need an intermediate buffer at all when called from dumps()? How about
allocating the buffer as a PyStringObject, so that it can be used directly for
the result in that case?
(IIRC there's a handy _PyString_Resize function)
--
Skip Montanaro s...@pobox.com added the comment:
Perhaps. Let's take it one step at a time though. If I change your
large pickle example to use dumps() instead of dump() in an unsullied
Python2.5 I get a MemoryError. In general, I think you have to be
careful using dumps(). Any attempt to
Skip Montanaro s...@pobox.com added the comment:
Oh, BTW, the proposed fix is in Rietveld: http://codereview.appspot.com/189051
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
Antoine Pitrou pit...@free.fr added the comment:
There were a couple of comments on the Rietveld code review above.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
Skip Montanaro s...@pobox.com added the comment:
Antoine There were a couple of comments on the Rietveld code review
Antoine above.
Indeed there are. Given that the Unladen Swallow folks were focusing on the
2.6 branch and their goal was to improve performance I don't see any reason
to not
Antoine Pitrou pit...@free.fr added the comment:
Indeed there are. Given that the Unladen Swallow folks were focusing on the
2.6 branch and their goal was to improve performance I don't see any reason
to not accept what they've done, then tweak it for 2.7/3.1 assuming the
changes you and
Skip Montanaro s...@pobox.com added the comment:
Antoine The main thing I'm worried about is the potentially unbounded
Antoine buffering, since it could reduce performance (or even thrash
Antoine the machine) instead of improving it.
Got a test case in mind? If so, I'll code it up and
Antoine Pitrou pit...@free.fr added the comment:
Got a test case in mind? If so, I'll code it up and compare 2.6 and Unladen
Swallow as well as offer it up for inclusion in the U-S test suite.
Trying to pickle a structure that's larger than half the RAM should do
the trick. Something like a
Antoine Pitrou pit...@free.fr added the comment:
Quick test on a 3GB machine:
Without patch (top shows the process reaches 1.2GB RAM max):
$ time ./python -c import cPickle;l=['a'*1024 for i in
xrange(100)];cPickle.dump(l, open('/dev/null', 'wb'))
10.67user 1.47system 0:12.92elapsed
Skip Montanaro s...@pobox.com added the comment:
Antoine With the patch, the same command quickly swaps hopelessly and
Antoine after 5 minutes of elapsed time I finally manage to kill the
Antoine process.
Verified with an Unladen Swallow test case. I'll see if I can fix it.
S
Skip Montanaro s...@pobox.com added the comment:
Updated the patch against the latest version of cPickle.c (r77393). All tests
pass on my Mac.
--
nosy: +skip.montanaro
Added file: http://bugs.python.org/file15808/cPickle.-r77393.patch
___
Python
Alexandre Vassalotti alexan...@peadrop.com added the comment:
Last august, I worked on integrating Collin's optimization work into
py3k in a local Mercurial branch. So, I can champion these changes into
py3k, if Collin is unavailable.
And if Collin allows me, I would like to merge the other
Antoine Pitrou pit...@free.fr added the comment:
Are you still willing to work on this?
--
versions: +Python 3.2 -Python 3.1
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
Jerry Chen je...@3rdengine.com added the comment:
Applied collinwinter's patch and svn up'd to r73872; replaced additional
instances of self-write_func with _Pickler_Write.
Passed test_xpickle.py as referenced by issue 5665.
--
nosy: +jcsalterego
Added file:
Antoine Pitrou pit...@free.fr added the comment:
Any updates?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
___
Python-bugs-list
New submission from Collin Winter coll...@gmail.com:
This patch simplifies cPickle's complicated internal buffering system.
The new version uses a single buffer closer to the Pickler object,
flushing to a file object only when necessary. This avoids the overhead
of several indirection layers for
Changes by Antoine Pitrou pit...@free.fr:
--
nosy: +pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5683
___
___
Python-bugs-list mailing
20 matches
Mail list logo
