[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-26 Thread Roundup Robot
Roundup Robot added the comment: New changeset f3f23ecdb1c6 by Serhiy Storchaka in branch '2.7': Issue #13555: Fix an integer overflow check. http://hg.python.org/cpython/rev/f3f23ecdb1c6 -- ___ Python tracker rep...@bugs.python.org

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-26 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Thank you for the report. Standard tests do not cover pickling/unpickling to real files. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-26 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: I have opened issue17299 for testing issue. -- resolution: - fixed stage: - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-26 Thread Jesús Cea Avión
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list mailing list

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-25 Thread Arfrever Frehtes Taifersar Arahesis
Arfrever Frehtes Taifersar Arahesis added the comment: 680959a3ae2e has caused that g-ir-scanner tool from gobject-introspection (which seems to create some pickles and next load them?) fails with MemoryError, e.g. during building of GTK+. MemoryError occurs only for a subset of pickles. I

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-15 Thread Serhiy Storchaka
Changes by Serhiy Storchaka storch...@gmail.com: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-12 Thread Roundup Robot
Roundup Robot added the comment: New changeset 680959a3ae2e by Serhiy Storchaka in branch '2.7': Issue #13555: cPickle now supports files larger than 2 GiB. http://hg.python.org/cpython/rev/680959a3ae2e -- nosy: +python-dev ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-11 Thread Serhiy Storchaka
Changes by Serhiy Storchaka storch...@gmail.com: Removed file: http://bugs.python.org/file29020/pickle_overflow-3.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-11 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Test updated too. Now it doesn't try to write a string larger than 2 GiB (it's impossible), instead writes a lot of shorter strings with total size larger than 2 GiB. -- Added file: http://bugs.python.org/file29042/pickle_overflow-4.diff

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-02-09 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Here is an updated patch. Fixed two bugs found by Antoine (an inappropriate format and a memory error in bigmemtest), fixed resizing of marks array and one possible integer overflow in write_other(). Used workaround to bypass limitations of cStringIO API.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-01-28 Thread Serhiy Storchaka
Changes by Serhiy Storchaka storch...@gmail.com: -- dependencies: +cStringIO not 64-bit safe -cStringIO.StringIO aborted when more then INT_MAX bytes written ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-01-27 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-01-27 Thread Charles-François Natali
Charles-François Natali added the comment: Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments. No. I don't have access to a 64-bit box, which makes it difficult to write (I don't get compiler warnings) and test.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2013-01-27 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: I have picked up it. Actually BINSTRING format supports only strings less 2GiB, so test should be changed. But there are other bugs which prohibit pickling a lot of data. For one I open issue17054. -- dependencies: +cStringIO.StringIO aborted when

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-12-30 Thread Ramchandra Apte
Ramchandra Apte added the comment: Bump. @neologix I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-12-30 Thread Antoine Pitrou
Antoine Pitrou added the comment: I may tackle this but rare 2.7-only bugs are pretty low on my priorities list. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-12-30 Thread Charles-François Natali
Charles-François Natali added the comment: I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap) AFAICT, a binary string a little longer than 1GB should be enough to reproduce the bug. Just make sure Python isn't built with '-fwrapv'. --

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-12-24 Thread Charles-François Natali
Charles-François Natali added the comment: Could someone with a 64-bit box take over this one? I won't go anywhere with my 32-bit box... -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-12-02 Thread Serhiy Storchaka
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka stage: - needs patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-07-30 Thread Arfrever Frehtes Taifersar Arahesis
Changes by Arfrever Frehtes Taifersar Arahesis arfrever@gmail.com: -- nosy: +Arfrever ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-01-06 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: Antoine, could you test the last version (test_pickle and if possible with the OP testcase)? I can't test it myself (32-bit machine with 1 GB). -- ___ Python tracker rep...@bugs.python.org

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2012-01-06 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Le vendredi 06 janvier 2012 à 19:17 +, Charles-François Natali a écrit : Charles-François Natali neolo...@free.fr added the comment: Antoine, could you test the last version (test_pickle and if possible with the OP testcase)? I can't

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-22 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: Here's a new version with a test (untested). Note that I'm absolutely not sure that the 'memsize' argument to bigmemtest is correct. -- Added file: http://bugs.python.org/file24079/pickle_overflow-2.diff

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-18 Thread Florent Xicluna
Changes by Florent Xicluna florent.xicl...@gmail.com: -- nosy: +flox ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-18 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: New version. Looks good to me. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: So it seems unlikely to be the explanation. Victor reproduced in on IRC, and it's indeed an overflow. The problematic code is in readline_file: bigger = self-buf_size 1; if (bigger = 0) { /* overflow */

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread STINNER Victor
STINNER Victor victor.stin...@haypocalc.com added the comment: Should we fix this (Py_ssize_t, overflow check before computation), as in #11564? Yes. Use Py_ssize_t type for the buf_size attribute, and replace bigger = 0 (test if an overflow occurred) by self-buf_size (PY_SSIZE_T_MAX 1).

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Ah, I see. It's a bit of a pity not to be able to load files 1GB, especially on a 64-bit build (!). Perhaps cPickle could be made partly 64-bit compatible? Or at least, indeed, do a proper anti-overflow check. --

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: Here's a patch which should fix this. However, I'm unable to test it. -- keywords: +patch Added file: http://bugs.python.org/file24006/pickle_overflow.diff ___ Python tracker

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: I think there's a problem here: +self-data = realloc(self-data, self-size * sizeof(PyObject *)); +if (self-data == NULL) goto nomemory; If realloc() fails, the old data pointer is lost and therefore will never get free()ed.

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-17 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: New version. -- Added file: http://bugs.python.org/file24008/pickle_overflow-1.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Philipp Lies
Philipp Lies p...@bethgelab.org added the comment: a) it's 122GB free RAM (out of 128GB total RAM) b) when I convert the numpy array to a list it works. So seems to be a problem with cPickle and numpy at/from a certain array size c) $ /usr/bin/time -v python test_np.py Traceback (most recent

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? AFAICT this wasn't fixed in 2.7. Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. @Philipp I'd be curious to see the last

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: @Antoine Couldn't this be linked to #11564 (pickle not 64-bit ready)? Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size. AFAICT this wasn't fixed in 2.7. @Philipp I'd be curious to see the last

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Charles-François Natali
Changes by Charles-François Natali neolo...@free.fr: -- Removed message: http://bugs.python.org/msg149321 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Couldn't this be linked to #11564 (pickle not 64-bit ready)? Well, I don't know anything about numpy, but: 196 * 24 4704 196 * 24 * 8 # assuming 8 bytes per float 37632 2**31 2147483648 So it seems unlikely to be the

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-12 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Oh, what's more interesting is that it works here (Python 2.7.1 and numpy 1.6.1 under Mageia with 8GB RAM). Looking at a pickle disassembly, the only remarkable thing is the presence of a long binary string (the raw serialization of all IEEE

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread Phillies
New submission from Phillies p...@bethgelab.org: When I try to load a large file (1GB) cPickle crashes with a MemoryError: $python test.py Traceback (most recent call last): File /tmp/test.py, line 8, in module A2 = cPickle.load(f2) MemoryError test.py contains following code: import

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread maniram maniram
maniram maniram maniandra...@gmail.com added the comment: Maybe Ubuntu doesn't think it is safe to allocate the memory. -- nosy: +maniram.maniram ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread Philipp Lies
Philipp Lies p...@bethgelab.org added the comment: Well, replace cPickle by pickle and it works. So if there is a memory allocation problem cPickle should be able to handle it, especially since it should be completely compatible to pickle. -- ___

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread maniram maniram
maniram maniram maniandra...@gmail.com added the comment: Have you checked the system monitor after all cPickle can use more memory than 1GB. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread Antoine Pitrou
Antoine Pitrou pit...@free.fr added the comment: Is there a way to reproduce that doesn't involve numpy? -- components: +Extension Modules -None nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread Antoine Pitrou
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +mark.dickinson ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13555 ___ ___ Python-bugs-list

[issue13555] cPickle MemoryError when loading large file (while pickle works)

2011-12-08 Thread Charles-François Natali
Charles-François Natali neolo...@free.fr added the comment: my computer has 122GB free RAM 122, or 1.22? Well, replace cPickle by pickle and it works. cPickle makes some direct call to malloc()/realloc()/free(), contrarily to pickle which uses pymalloc. This could lead to heap