Roundup Robot added the comment:
New changeset f3f23ecdb1c6 by Serhiy Storchaka in branch '2.7':
Issue #13555: Fix an integer overflow check.
http://hg.python.org/cpython/rev/f3f23ecdb1c6
--
___
Python tracker rep...@bugs.python.org
Serhiy Storchaka added the comment:
Thank you for the report. Standard tests do not cover pickling/unpickling to
real files.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Serhiy Storchaka added the comment:
I have opened issue17299 for testing issue.
--
resolution: - fixed
stage: - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Changes by Jesús Cea Avión j...@jcea.es:
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Python-bugs-list mailing list
Arfrever Frehtes Taifersar Arahesis added the comment:
680959a3ae2e has caused that g-ir-scanner tool from gobject-introspection
(which seems to create some pickles and next load them?) fails with
MemoryError, e.g. during building of GTK+.
MemoryError occurs only for a subset of pickles.
I
Changes by Serhiy Storchaka storch...@gmail.com:
--
resolution: - fixed
stage: patch review - committed/rejected
status: open - closed
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Roundup Robot added the comment:
New changeset 680959a3ae2e by Serhiy Storchaka in branch '2.7':
Issue #13555: cPickle now supports files larger than 2 GiB.
http://hg.python.org/cpython/rev/680959a3ae2e
--
nosy: +python-dev
___
Python tracker
Changes by Serhiy Storchaka storch...@gmail.com:
Removed file: http://bugs.python.org/file29020/pickle_overflow-3.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Serhiy Storchaka added the comment:
Test updated too. Now it doesn't try to write a string larger than 2 GiB (it's
impossible), instead writes a lot of shorter strings with total size larger
than 2 GiB.
--
Added file: http://bugs.python.org/file29042/pickle_overflow-4.diff
Serhiy Storchaka added the comment:
Here is an updated patch. Fixed two bugs found by Antoine (an inappropriate
format and a memory error in bigmemtest), fixed resizing of marks array and one
possible integer overflow in write_other(). Used workaround to bypass
limitations of cStringIO API.
Changes by Serhiy Storchaka storch...@gmail.com:
--
dependencies: +cStringIO not 64-bit safe -cStringIO.StringIO aborted when more
then INT_MAX bytes written
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Serhiy Storchaka added the comment:
Charles-François, are you going to finish this before 2.7.4 RC released? The
patch should be updated to address Antoine's comments.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Charles-François Natali added the comment:
Charles-François, are you going to finish this before 2.7.4 RC released? The
patch should be updated to address Antoine's comments.
No.
I don't have access to a 64-bit box, which makes it difficult to write
(I don't get compiler warnings) and test.
Serhiy Storchaka added the comment:
I have picked up it.
Actually BINSTRING format supports only strings less 2GiB, so test should be
changed. But there are other bugs which prohibit pickling a lot of data. For
one I open issue17054.
--
dependencies: +cStringIO.StringIO aborted when
Ramchandra Apte added the comment:
Bump.
@neologix
I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though
one could use swap)
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Antoine Pitrou added the comment:
I may tackle this but rare 2.7-only bugs are pretty low on my priorities list.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Charles-François Natali added the comment:
I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though
one could use swap)
AFAICT, a binary string a little longer than 1GB should be enough to
reproduce the bug.
Just make sure Python isn't built with '-fwrapv'.
--
Charles-François Natali added the comment:
Could someone with a 64-bit box take over this one?
I won't go anywhere with my 32-bit box...
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Changes by Serhiy Storchaka storch...@gmail.com:
--
nosy: +serhiy.storchaka
stage: - needs patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com:
--
nosy: +Arfrever
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Charles-François Natali neolo...@free.fr added the comment:
Antoine, could you test the last version (test_pickle and if possible
with the OP testcase)?
I can't test it myself (32-bit machine with 1 GB).
--
___
Python tracker rep...@bugs.python.org
Antoine Pitrou pit...@free.fr added the comment:
Le vendredi 06 janvier 2012 à 19:17 +, Charles-François Natali a
écrit :
Charles-François Natali neolo...@free.fr added the comment:
Antoine, could you test the last version (test_pickle and if possible
with the OP testcase)?
I can't
Charles-François Natali neolo...@free.fr added the comment:
Here's a new version with a test (untested).
Note that I'm absolutely not sure that the 'memsize' argument to
bigmemtest is correct.
--
Added file: http://bugs.python.org/file24079/pickle_overflow-2.diff
Changes by Florent Xicluna florent.xicl...@gmail.com:
--
nosy: +flox
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Python-bugs-list
Antoine Pitrou pit...@free.fr added the comment:
New version.
Looks good to me.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Charles-François Natali neolo...@free.fr added the comment:
So it seems unlikely to be the explanation.
Victor reproduced in on IRC, and it's indeed an overflow.
The problematic code is in readline_file:
bigger = self-buf_size 1;
if (bigger = 0) { /* overflow */
STINNER Victor victor.stin...@haypocalc.com added the comment:
Should we fix this (Py_ssize_t, overflow check before computation), as in
#11564?
Yes. Use Py_ssize_t type for the buf_size attribute, and replace bigger = 0
(test if an overflow occurred) by self-buf_size (PY_SSIZE_T_MAX 1).
Antoine Pitrou pit...@free.fr added the comment:
Ah, I see. It's a bit of a pity not to be able to load files 1GB, especially
on a 64-bit build (!). Perhaps cPickle could be made partly 64-bit compatible?
Or at least, indeed, do a proper anti-overflow check.
--
Charles-François Natali neolo...@free.fr added the comment:
Here's a patch which should fix this. However, I'm unable to test it.
--
keywords: +patch
Added file: http://bugs.python.org/file24006/pickle_overflow.diff
___
Python tracker
Antoine Pitrou pit...@free.fr added the comment:
I think there's a problem here:
+self-data = realloc(self-data, self-size * sizeof(PyObject *));
+if (self-data == NULL)
goto nomemory;
If realloc() fails, the old data pointer is lost and therefore will never get
free()ed.
Charles-François Natali neolo...@free.fr added the comment:
New version.
--
Added file: http://bugs.python.org/file24008/pickle_overflow-1.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Philipp Lies p...@bethgelab.org added the comment:
a) it's 122GB free RAM (out of 128GB total RAM)
b) when I convert the numpy array to a list it works. So seems to be a problem
with cPickle and numpy at/from a certain array size
c) $ /usr/bin/time -v python test_np.py
Traceback (most recent
Charles-François Natali neolo...@free.fr added the comment:
@Antoine
Couldn't this be linked to #11564 (pickle not 64-bit ready)? AFAICT this wasn't
fixed in 2.7. Basically, an integer overflow, and malloc() would bail out when
asked a ridiculous size.
@Philipp
I'd be curious to see the last
Charles-François Natali neolo...@free.fr added the comment:
@Antoine
Couldn't this be linked to #11564 (pickle not 64-bit ready)? Basically, an
integer overflow, and malloc() would bail out when asked a ridiculous size.
AFAICT this wasn't fixed in 2.7.
@Philipp
I'd be curious to see the last
Changes by Charles-François Natali neolo...@free.fr:
--
Removed message: http://bugs.python.org/msg149321
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
Antoine Pitrou pit...@free.fr added the comment:
Couldn't this be linked to #11564 (pickle not 64-bit ready)?
Well, I don't know anything about numpy, but:
196 * 24
4704
196 * 24 * 8 # assuming 8 bytes per float
37632
2**31
2147483648
So it seems unlikely to be the
Antoine Pitrou pit...@free.fr added the comment:
Oh, what's more interesting is that it works here (Python 2.7.1 and numpy 1.6.1
under Mageia with 8GB RAM). Looking at a pickle disassembly, the only
remarkable thing is the presence of a long binary string (the raw serialization
of all IEEE
New submission from Phillies p...@bethgelab.org:
When I try to load a large file (1GB) cPickle crashes with a MemoryError:
$python test.py
Traceback (most recent call last):
File /tmp/test.py, line 8, in module
A2 = cPickle.load(f2)
MemoryError
test.py contains following code:
import
maniram maniram maniandra...@gmail.com added the comment:
Maybe Ubuntu doesn't think it is safe to allocate the memory.
--
nosy: +maniram.maniram
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Philipp Lies p...@bethgelab.org added the comment:
Well, replace cPickle by pickle and it works. So if there is a memory
allocation problem cPickle should be able to handle it, especially since it
should be completely compatible to pickle.
--
___
maniram maniram maniandra...@gmail.com added the comment:
Have you checked the system monitor after all cPickle can use more memory than
1GB.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Antoine Pitrou pit...@free.fr added the comment:
Is there a way to reproduce that doesn't involve numpy?
--
components: +Extension Modules -None
nosy: +pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
Changes by Antoine Pitrou pit...@free.fr:
--
nosy: +mark.dickinson
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13555
___
___
Python-bugs-list
Charles-François Natali neolo...@free.fr added the comment:
my computer has 122GB free RAM
122, or 1.22?
Well, replace cPickle by pickle and it works.
cPickle makes some direct call to malloc()/realloc()/free(), contrarily to
pickle which uses pymalloc. This could lead to heap
44 matches
Mail list logo
