[issue11564] pickle not 64-bit ready

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: Should be fixed as far as possible (OverflowErrors will be raised instead of crashing). Making people actually 64-bit compliant is part of PEP 3154 (http://www.python.org/dev/peps/pep-3154/). -- resolution: -> fixed stage: patch review -> committed/r

[issue11564] pickle not 64-bit ready

Roundup Robot added the comment: New changeset babc90f3cbf4 by Antoine Pitrou in branch '3.2': Issue #11564: Avoid crashes when trying to pickle huge objects or containers http://hg.python.org/cpython/rev/babc90f3cbf4 New changeset 56242682a931 by Antoine Pitrou in branch 'default': Issue #1156

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: Tested the latest patch with -M11G. All tests pass. -- ___ Python tracker ___ ___ Python-bugs-list ma

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: Here is a new patch against 3.2. I can't say it works for sure, but it should be much better. It also adds a couple more tests. There seems to be a separate issue where pure-Python pickle.py considers 32-bit lengths signed where the C impl considers them unsig

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: > With data as (b"a" * _4G) the result is as expected: > > Traceback (most recent call last): > File "pickle-bigmem-test.py", line 5, in > out = dumps(data) > OverflowError: cannot serialize a string larger than 4GB > > But with (b"a" * _2G)

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: Some more info: the first few bytes of the output for the _2G case are this: b'\x80\x03X\x00\x00\x00\x80aa' -- ___ Python tracker ___ __

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: I was playing around with pickling large Unicode strings in an interactive interpreter, and it seems that you have to have at least 4G chars (not bytes) to trigger the OverflowError. Consider the following snippet of code: out = dumps(data) del data r

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: > Uh, does it? With 4G it should raise OverflowError, and not try to do > anything else. > Could I ask you to try to take a look? :S Sure; I'll see what I can figure out tomorrow. > Running bigmem tests in parallel doesn't make much sense IMO. You want > to run

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > > Can you replace "_2G" with "_4G" in the decorator for that test? > > When I do that, it pushes the memory usage for the test up to 16GB, which is > beyond what the machine can handle. When I tried with 2.5G (_2G * 5 // 4), > that was enough to make it swap

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: > Can you replace "_2G" with "_4G" in the decorator for that test? When I do that, it pushes the memory usage for the test up to 16GB, which is beyond what the machine can handle. When I tried with 2.5G (_2G * 5 // 4), that was enough to make it swap heavily (and

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: > Can you replace "_2G" with "_4G" in the decorator for that test? I'm not at work any more, but I'll try that out on Monday. -- ___ Python tracker

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > D'oh. I just realized why the -M option wasn't being recognized - I had > passed it > after the actual test name, so it was being treated as another test instead > of an > option. Sorry for the confusion :/ > > As for the actual test results, test_huge_byte

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: D'oh. I just realized why the -M option wasn't being recognized - I had passed it after the actual test name, so it was being treated as another test instead of an option. Sorry for the confusion :/ As for the actual test results, test_huge_bytes_(32|64)b both

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > I'd be glad to do so, but I'm not sure what the aim of the "dryrun" flag is. > Do you want to make it the default that precisionbigmem tests are skipped, > unless the decorator invocation explicitly specifies dryrun=False? No, the point is to avoid running th

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: > How much does it say is required? > Did you remove the skips in BigmemPickleTests? Yes, I did remove the skips. It says 2GB for some, and 4GB for others. > Well, perhaps I got the logic wrong. Debugging welcome :) I'd be glad to do so, but I'm not sure what t

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > I have tried running the tests on a machine with 12GB of RAM, but when I do > so, > the new tests get skipped saying "not enough memory", even when I specify "-M > 11G" > on the command-line. How much does it say is required? Did you remove the skips in Big

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: I have tried running the tests on a machine with 12GB of RAM, but when I do so, the new tests get skipped saying "not enough memory", even when I specify "-M 11G" on the command-line. The problem seems to be the change to the precisionbigmemtest decorator in tes

[issue11564] pickle not 64-bit ready

Nadeem Vawda added the comment: pickle64.patch applies cleanly to 3.2, but not 3.3. I've attached an adapted version that applies cleanly to 3.3. -- Added file: http://bugs.python.org/file22914/pickle64-3.3.patch ___ Python tracker

[issue11564] pickle not 64-bit ready

Changes by Nadeem Vawda : -- nosy: +nadeem.vawda ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.py

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: This patch contains assorted improvements for 64-bit compatibility of the pickle module. The protocol still doesn't support >4GB bytes or str objects, but at least its behaviour shouldn't be misleading anymore. -- keywords: +patch stage: -> patch rev

[issue11564] pickle not 64-bit ready

Jorgen Skancke added the comment: I recently ran into this problem when I tried to multiprocess jobs with large objects (3-4 GB). I have plenty of memory for this, but multiprocessing hangs without error, presumably because pickle hangs without error when multiprocessing tries to pickle the o

[issue11564] pickle not 64-bit ready

Changes by Santoso Wijaya : -- nosy: +santa4nt ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue11564] pickle not 64-bit ready

Nik Galitsky added the comment: Thank you all for your responses. While getting the meaningful error message in this case is very important, the main thing for us is to somehow fix this problem to allow larger objects serialization which is not at all uncommon on a 64-bit machines with large

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > I don't think so. OverflowError is a subclass of ArithmeticError and > is raised when result of an arithmetic operation cannot be represented > by the python type. For example, If you grep for OverflowError in the C code base, you'll see that it is in prac

[issue11564] pickle not 64-bit ready

Alexander Belopolsky added the comment: On Wed, Mar 16, 2011 at 6:59 PM, Antoine Pitrou wrote: .. >> >> What would be the "proper exception" here? > > OverflowError. This is the exception that gets raised when some > user-supplied value exceeds some internal limit. I don't think so. OverflowE

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: > On Tue, Mar 15, 2011 at 7:05 PM, Antoine Pitrou > wrote: > .. > > - (bugfix) raise a proper exception when an object too large for handling > > by pickle is given > > What would be the "proper exception" here? OverflowError. This is the exception that get

[issue11564] pickle not 64-bit ready

Alexander Belopolsky added the comment: On Tue, Mar 15, 2011 at 7:05 PM, Antoine Pitrou wrote: .. > - (bugfix) raise a proper exception when an object too large for handling by > pickle is given What would be the "proper exception" here? With _pickle acceleration disabled, I get a struct.err

[issue11564] pickle not 64-bit ready

Alexandre Vassalotti added the comment: We could resort to the text-based protocol which doesn't have these limitations with respect to object lengths (IIRC). Performance won't be amazing, but we won't have to modify the current pickle protocol. -- ___

[issue11564] pickle not 64-bit ready

Antoine Pitrou added the comment: Indeed: >>> s = b'a' * (2**31) >>> d = pickle.dumps(s) Traceback (most recent call last): File "", line 1, in SystemError: error return without exception set There are two aspects to this: - (bugfix) raise a proper exception when an object too large for han