[issue25626] Gzip fails for file over 2**32 bytes

Roundup Robot added the comment: New changeset 80f6f77a7cc3 by Martin Panter in branch '3.5': Issue #25626: Change zlib to accept Py_ssize_t and cap to UINT_MAX https://hg.python.org/cpython/rev/80f6f77a7cc3 New changeset afa1b6cd77a5 by Martin Panter in branch 'default': Issue #25626: Merge

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Thank you Serhiy for you help, and Ben for reporting this. -- resolution: -> fixed stage: commit review -> resolved status: open -> closed ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Added new comments. -- ___ Python tracker ___ ___ Python-bugs-list

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Thanks for your testing Serhiy. I can reproduce this on 32-bit Linux (also by building with CC="gcc -m32"). It is easy to produce the bug with flush(), but with Decompress.decompress() it would need over 2 GiB of memory and data to expand the buffer enough to

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Ah, I found yet one bug. >>> zlib.decompress(zlib.compress(b'abcd'), 0, sys.maxsize+1) Traceback (most recent call last): File "", line 1, in SystemError: Negative size passed to PyBytes_FromStringAndSize There are similar bugs in decompressor's methods

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: New patch addressing comments. I used the undocumented API _PyLong_FromNbInt() to call __int__() rather than __index__(). -- Added file: http://bugs.python.org/file41107/zlib-Py_ssize_t.2.patch ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Actually it did make an existing bug a bit worse. The bug is triggered with valid sizes greater than LONG_MAX, due to an exception not being cleared, and my change made this bug more dramatic, turning the OverflowError into a crash. This new patch should fix

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Thank you Martin, the patch LGTM. The difference between __int__() and __index__() is that float has the former, but not the latter. For better compatibility we have to use __int__() in maintained releases. But obviously __index__() is more appropriate

[issue25626] Gzip fails for file over 2**32 bytes

Changes by Serhiy Storchaka : -- assignee: -> martin.panter stage: patch review -> commit review ___ Python tracker ___

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Okay. For the gzip module, I cannot easily test this myself. Quickly looking at other cases, I guess it would look something like this, but I need to spend some time understanging the bigmemtest decorator properly: @unittest.skipIf(sys.maxsize < _4G, "Requires

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Could you please add tests for other two functions? And tests for the gzip module? -- ___ Python tracker ___

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: You can commit your patch right now (it shouldn't make things worse) and add new tests later. -- ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: I am inclined to commit my patch to get this fixed in the upcoming 3.5.1 release, unless anyone thinks that could be a problem. -- components: +Extension Modules ___ Python tracker

[issue25626] Gzip fails for file over 2**32 bytes

Serhiy Storchaka added the comment: Is 3.4 affected? A uint_converter was added in issue18294. -- ___ Python tracker ___

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: 3.4 shouldn’t be affected by the gzip regression. In 3.4 the gzip module does not set max_length to limit the decompression size. In 3.4 the underlying zlib module will also raise OverflowError if max_lenth=2**32, and my patch could be applied, but I don’t

[issue25626] Gzip fails for file over 2**32 bytes

Changes by Serhiy Storchaka : -- nosy: +serhiy.storchaka ___ Python tracker ___ ___

[issue25626] Gzip fails for file over 2**32 bytes

Martin Panter added the comment: Ben: By adding the cap, it means when the lower-level zlib module is asked to decode say 5 GiB, it will decode at most 4 GiB (rather than raising an exception). The 4 GiB limit (UINT_MAX to be precise) is due to the underlying zlib library; it’s not Python’s

[issue25626] Gzip fails for file over 2**32 bytes

Changes by SilentGhost : -- components: +Library (Lib) nosy: +nadeem.vawda, twouters versions: +Python 3.6 ___ Python tracker ___