[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment: Any chance to commit the patch before final feature freeze? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by Antoine Pitrou pit...@free.fr: -- assignee: docs@python - nosy: +nadeem.vawda stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Nadeem Vawda nadeem.va...@gmail.com added the comment: Patch looks fine to me. Antoine, can you commit this? I'm currently away from the computer that has my SSH key on it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Roundup Robot devn...@psf.upfronthosting.co.za added the comment: New changeset 0e8285321659 by Antoine Pitrou in branch 'default': On behalf of Nadeem Vawda: issue #10376: micro-optimize reading from a Zipfile. http://hg.python.org/cpython/rev/0e8285321659 -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Antoine Pitrou pit...@free.fr added the comment: Antoine, can you commit this? Ok, done. -- resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment: Martin, now the patch is good? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment: The patch updated to reflect Martin's stylistic comments. Sorry for the delay, Martin. I have not received an email with your review from 2012-05-13, and only today accidentally discovered your comments in Rietveld. It seems to have been some bug in Rietveld. -- Added file: http://bugs.python.org/file25769/zipfile_optimize_read_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by Martin v. Löwis mar...@v.loewis.de: Added file: http://bugs.python.org/file25565/zipfile_optimize_read.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment: Thank you, Martin, now I understood why not work Rietveld review. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment:
This is not because zipfile module is unbuffered. This is the difference
between expensive function call and cheap bytes slicing. Replace
`zf.open(namelist [0])` to `io.BufferedReader(zf.open(namelist [0]))` to see
the effect of a good buffering. In 3.2 zipfile read() implemented not optimal,
so it slower (twice), but in 3.3 it will be almost as fast as using
io.BufferedReader. It is still several times more slowly than bytes slicing,
but there's nothing you can do with it.
Here is a patch, which is speeds up (+20%) the reading from a zip file by small
chunks. Microbenchmark:
./python -m zipfile -c test.zip python
./python -m timeit -n 1 -s import zipfile;zf=zipfile.ZipFile('test.zip')
with zf.open('python') as f:while f.read(1):pass
Python 3.3 (vanilla): 1 loops, best of 3: 36.4 sec per loop
Python 3.3 (patched): 1 loops, best of 3: 30.1 sec per loop
Python 3.3 (with io.BufferedReader): 1 loops, best of 3: 30.2 sec per loop
And, for comparison, Python 3.2: 1 loops, best of 3: 74.5 sec per loop
--
components: -Documentation
keywords: +patch
versions: -Python 2.7, Python 3.2
Added file: http://bugs.python.org/file25530/zipfile_optimize_read.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10376
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by STINNER Victor victor.stin...@gmail.com: -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
James Hutchison jamesghutchi...@gmail.com added the comment: See attached, which will open a zipfile that contains one file and reads it a bunch of times using unbuffered and buffered idioms. This was tested on windows using python 3.2 You're in charge of coming up with a file to test it on. Sorry. Example output: Enter filename: test.zip Timing unbuffered read, 5 bytes at a time. 10 loops took 6.67131335449 Timing buffered read, 5 bytes at a time (4000 byte buffer). 10 loops took 0.7350001335144043 -- Added file: http://bugs.python.org/file25432/zipfiletest.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Serhiy Storchaka storch...@gmail.com added the comment: Actually reading from the zip file is buffered (at least 4 KiB of uncompressed data at a time). Can you give tests, scripts and data, which show the problem? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by Xuanji Li xua...@gmail.com: -- nosy: +xuanji ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
Changes by Terry J. Reedy tjre...@udel.edu: -- versions: +Python 3.2, Python 3.3 -Python 2.5, Python 2.6, Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
New submission from James Hutchison jamesghutchi...@gmail.com: The Unzip module is always unbuffered (tested v.3.1.2 Windows XP, 32-bit). This means that if one has to do many small reads it is a lot slower than reading a chunk of data to a buffer and then reading from that buffer. It seems logical that the unzip module should default to buffered reading and/or have a buffered argument. Likewise, the documentation should clarify that there is no buffering involved when doing a read, which runs contrary to the default behavior of a normal read. start Zipfile read done 27432 reads done took 0.859 seconds start buffered Zipfile read done 27432 reads done took 0.072 seconds start normal read (default buffer) done 27432 reads done took 0.139 seconds start buffered normal read done 27432 took 0.137 seconds -- assignee: d...@python components: Documentation, IO, Library (Lib) messages: 120871 nosy: Jimbofbx, d...@python priority: normal severity: normal status: open title: ZipFile unzip is unbuffered type: performance versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10376] ZipFile unzip is unbuffered
James Hutchison jamesghutchi...@gmail.com added the comment: I should clarify that this is the zipfile constructor I am using: zipfile.ZipFile(filename, mode='r', allowZip64=True); -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue10376 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
