[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Thanks Martin too. Nobody than me knows how much work you have done to this. :) I could have made it better at first. :( Sorry for the noise to everyone. -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Thanks Martin too. Nobody than me knows how much work you have done to this. :) I could have made it better at first. :( Sorry for the noise to everyone. -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Klamann added the comment: Thanks Xiang and Martin for solving this, you guys are awesome :) -- ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Thanks Xiang for your work on this, and Klamann for the report. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Roundup Robot added the comment: New changeset 2192edcfea02 by Martin Panter in branch '2.7': Issue #27130: Fix handling of buffers exceeding (U)INT_MAX in “zlib” module https://hg.python.org/cpython/rev/2192edcfea02 -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Roundup Robot added the comment: New changeset bd61bcd9bee8 by Martin Panter in branch '3.5': Issue #27130: Fix handling of buffers exceeding UINT_MAX in “zlib” module https://hg.python.org/cpython/rev/bd61bcd9bee8 New changeset bd556f748cf8 by Martin Panter in branch 'default': Issue #27130:

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: All tests passed now. :) I think it's OK. Also upload the v10 version restore the statement to avoid the crash you mentioned in comments. [1/1] test_zlib test_abcdefghijklmnop (test.test_zlib.ChecksumTestCase) test issue1202 compliance: signed crc32, adler32 in

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Here is a new patch for 2.7 that should avoid the problem with ChecksumBigBufferTestCase. I tested a hacked version of the test, so I am more confident in it now :) -- Added file: http://bugs.python.org/file43823/64bit_support_for_zlib_v11-2.7.patch

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Hi, Martin. I replied your last comment and your patch looks good to me except one test fails: [1/1] test_zlib test_abcdefghijklmnop (test.test_zlib.ChecksumTestCase) test issue1202 compliance: signed crc32, adler32 in 2.x ... ok test_adler32empty

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Here is a possible patch for 2.7. To fix everything on 2.7 I changed the module to accept input buffers over 2 GiB by enabling PY_SSIZE_T_CLEAN. As a consequence, the patch also includes ports of Nadeem Vawda’s adler32(), crc32() changes from Issue 10276, and

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: I may be hard to test this change without enough memory. I upload the test results with the latest change. -- Added file: http://bugs.python.org/file43775/zlib_test_result ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Martin Panter : -- Removed message: http://bugs.python.org/msg270561 ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: I added one comment, but I think this might almost be ready -- ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Upload the v9 version. It applies your last comment and catch up with the hg tip. -- Added file: http://bugs.python.org/file43750/64bit_support_for_zlib_v9.patch ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: This is on my list of things to look at, just that I have been away and am a bit backlogged atm. -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: ping -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Removed file: http://bugs.python.org/file43564/64bit_support_for_zlib_v8.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Added file: http://bugs.python.org/file43565/64bit_support_for_zlib_v8.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Make v8 patch consistent with the latest changeset. -- Added file: http://bugs.python.org/file43564/64bit_support_for_zlib_v8.patch ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Removed file: http://bugs.python.org/file43563/64bit_support_for_zlib_v8.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Removed file: http://bugs.python.org/file43561/64bit_support_for_zlib_v8.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Added file: http://bugs.python.org/file43563/64bit_support_for_zlib_v8.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: This is the v8 patch. It does two things: [1] Apply Martin's comment about decompressobj.decompress so when user passes in PY_SSIZE_T_MAX and there is enough memory, no error will be raised. [2] Now decompressobj.flush can still work even if

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Add the newest version applying Martin's comments. -- Added file: http://bugs.python.org/file43538/64bit_support_for_zlib_v7.patch ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Upload the near-final patch. This one adds large buffer tests on 64bit platforms. I have tested them on a server with enough memory. I don't add the @support.requires_resource('cpu') since large memory tests are born to be slow. And actually when I experiment,

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Removed file: http://bugs.python.org/file43460/64bit_support_for_zlib_v5.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Added file: http://bugs.python.org/file43461/64bit_support_for_zlib_v5.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Attach patch to restore argument parsing support for __int__. -- Added file: http://bugs.python.org/file43460/64bit_support_for_zlib_v5.patch ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Removed file: http://bugs.python.org/file43450/64bit_support_for_zlib_v4.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : Added file: http://bugs.python.org/file43459/64bit_support_for_zlib_v4.patch ___ Python tracker ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Upload the new patch to fix bugs and make improvements. I'll add tests later. -- Added file: http://bugs.python.org/file43450/64bit_support_for_zlib_v4.patch ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Sorry, I suddenly find that I've misunderstood one of your comments. I changed according then. Please use the new version. -- Added file: http://bugs.python.org/file43408/64bit_support_for_zlib_v3.patch ___ Python

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: I'm willing to and thanks for your work too :) I have replied to your comments and adjusted my patch accordingly. But there are still several I am confused or want to negotiate more. I now upload the adjusted patch. -- Added file:

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Thanks for working on this. I did a pass over your patch and left a bunch of comments. -- stage: needs patch -> patch review ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Hello Martin. I've finished a patch to add 64bit support to zlib. I think it solves the 9 problems you mentioned and let the input and output both be larger than UINT_MAX. Hope you are willing to review and we can move this forward. :) -- keywords:

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Jack McCracken added the comment: Don't know how useful this will be, but here's a crash report from Mac OS X 10.11 with Klamann's example (Python 3.5). -- nosy: +Jack.McCracken Added file: http://bugs.python.org/file43251/coredump_macosx10.11.5.crash

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: I'd like to help but it'll need some time. And I'd like to start after issue27164 is solved. zdict now also checks for 4GB limit. -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Klamann, thanks for crash report. I think your decompress crash is explained by the bug expanding past UINT_MAX I identified above. The key is that length = 0 in zlib_Decompress_decompress_impl(), as if wrapped around, and the return value will have been

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Klamann added the comment: > You should be able to use a compression (or decompression) object as a > workaround. OK, let's see >>> import zlib >>> zc = zlib.compressobj() >>> c1 = zc.compress(b'a' * 2**31) >>> c2 = zc.compress(b'a' * 2**31) >>> c3 = zc.flush() >>> c = c1 + c2 + c3 >>> zd =

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Yes. It's compression object not compress. I find more. The overflow checking is introduced to solve problem in issue8650. It seems the explicit overflow checking is introduced to keep compatibility with py2 (py2 raises the overflowerror in pyargparse). I

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: You should be able to use a compression (or decompression) object as a workaround. But calling zlib.compress() multiple times would generate multiple separate deflated streams, which is different. I think it is reasonable for Python to handle larger data sizes

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: Quick and careless scanning at night lead me to a wrong result, Sorry. > You would need to compress just under 4 GiB of data that requires 5 MB or > more when compressed (i.e. not all the same bytes, or maybe try level=0). With enough memory, compressing with

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: Sorry Issue 10276 regarding crc32() and adler32() was only fixed for Python 3. Issue 23306 is open about crc32() in Python 2. -- ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Martin Panter added the comment: This is similar, but different to the other bug. The other bug was only about output limits for incrementally decompressed data. Klamann’s bug is about the actual size of input (and possibly also output) buffers. The gzip.compress() implementation uses

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Klamann added the comment: > But you can only get that feature with Python3.5+. Well, I have Python 3.5.1 installed and the problem still persists. I'm not sure that 25626 ist the same problem - in the comments they say this was not an issue in Python 3.4 or 2.x, but this is clearly the case

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Changes by Xiang Zhang : -- nosy: +martin.panter ___ Python tracker ___ ___

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

Xiang Zhang added the comment: This behaviour seems to have been fixed in issue25626. But you can only get that feature with Python3.5+. -- nosy: +xiang.zhang ___ Python tracker

[issue27130] zlib: OverflowError while trying to compress 2^32 bytes or more

New submission from Klamann: zlib fails to compress files larger than 4gb due to some 32bit issues. I've tested this in Python 3.4.3 and 3.5.1: > python3 -c "import zlib; zlib.compress(b'a' * (2**32 - 1))" > python3 -c "import zlib; zlib.compress(b'a' * (2**32))" Traceback (most recent call