[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Changes by Serhiy Storchaka storch...@gmail.com: -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: Thanks. I guess I know who to ask now. It was just painful seeing so much import-time computation, pretty much guaranteed to happen every time Python starts up, being wasted on a feature that is rarely used. On the Raspberry Pi the majority of the import time is spent building this table and the speed difference is noticeable. On Mon, Jan 6, 2014, at 10:22 AM, Larry Hastings wrote: Larry Hastings added the comment: Since this isn't a bugfix, it was inappropriate to check this in after feature-freeze for 3.4. However it looks harmless enough, so I'm not asking you to revert it at this time. I guess it's easier to get forgiveness than permission, huh. -- nosy: +larry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Larry Hastings added the comment: Since this isn't a bugfix, it was inappropriate to check this in after feature-freeze for 3.4. However it looks harmless enough, so I'm not asking you to revert it at this time. I guess it's easier to get forgiveness than permission, huh. -- nosy: +larry ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: Fixed in http://hg.python.org/cpython/rev/536a2cf5f1d2 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: I am withdrawing zipfile-no-crc32.patch. It did not work correctly. zdlazy.patch should go in. It avoids creating the rarely-needed crc32 table until the first time it is needed, saving some memory and the majority of time needed to import the module. zdlazy.patch should not affect its C replacement at all, except that the rarely-needed CRC32 table that we are not generating becomes a never-needed table. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Serhiy Storchaka added the comment: I meant my pure Python patch in issue10030. Binding crctable to local variable is one of microoptimizations. Not the largest one however. So in general I not objects. Your patch LGTM. Only one nitpick -- instead not _ZipDecrypter.crctable use _ZipDecrypter.crctable is None. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Serhiy Storchaka added the comment: The objection to zipfile-no-crc32.patch is that binascii.crc32() and _crc32() have different signatures. binascii.crc32() accepts a byte object while _crc32() accepts a single integer. With packing this value into a bytes object _crc32() will be much slower. As for zdlazy.patch, there is a proposed patch in issue10030 which speedups pure Python decryption. I'm afraid that with zdlazy.patch this path will lose a part of it's speedup. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
New submission from Daniel Holth: http://hg.python.org/cpython/file/e7305517260b/Lib/zipfile.py#l460 I noticed this table taking up time on import. I'd guess that it pre-dates zlib.crc32 which is imported at the top of the file. I also suspect that most of the time this table isn't even used at all. -- files: zipfile-no-crc32.patch keywords: patch messages: 193408 nosy: dholth priority: normal severity: normal status: open title: zipfile._ZipDecryptor generates wasteful crc32 table on import versions: Python 2.7, Python 3.3, Python 3.4 Added file: http://bugs.python.org/file30990/zipfile-no-crc32.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: Someone who has a better understanding of zipfile may be able to get zlib.crc32(bytes[ch], running_crc) to work correctly. This patch that passes the zipfile tests generates the crctable only when _ZipDecrypter() is instantiated. -- Added file: http://bugs.python.org/file30991/zdlazy.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Changes by Serhiy Storchaka storch...@gmail.com: -- components: +Library (Lib) nosy: +alanmcintyre, loewis, serhiy.storchaka stage: - patch review type: - performance ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Serhiy Storchaka added the comment: How much time take it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: It takes 706 microseconds. On my machine %timeit import sys; del sys.modules['zipfile']; import zipfile import zipfile takes 2.51 ms without the patch and 1.7 ms with the patch. On Sat, Jul 20, 2013, at 12:13 PM, Serhiy Storchaka wrote: Serhiy Storchaka added the comment: How much time take it? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18515] zipfile._ZipDecryptor generates wasteful crc32 table on import
Daniel Holth added the comment: I tried it on a raspberry pi. zipfile takes 36 ms to import and 10 ms if it does not generate the crctable. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue18515 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
