[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Following our discussion and Guido's answer on python-3000 (*), I committed a modified fix in r65264. (*) http://mail.python.org/pipermail/python-3000/2008-July/014466.html -- resolution: - fixed status: open - closed

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: When I revised the patch I had a weak understanding of nonblocking I/O. I thought the exponential reads were for nonblocking I/O, but I see now that is non-sense. Fine, so it will make the patch simpler. As for non-blocking IO, I think we

[issue2523] binary buffered reading is quadratic

Alexandre Vassalotti [EMAIL PROTECTED] added the comment: Antoine wrote: Le lundi 21 juillet 2008 à 21:18 +, Martin v. Löwis a écrit : IIUC, a read of the full requested size would achieve exactly that: on a non-blocking stream (IIUC), a read will always return min(bytes_available,

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Selon Martin v. Löwis [EMAIL PROTECTED]: Martin v. Löwis [EMAIL PROTECTED] added the comment: I don't understand the second loop (where n is given). If n is given, there should be only a single read operation, using max(buffer_size,

[issue2523] binary buffered reading is quadratic

Martin v. Löwis [EMAIL PROTECTED] added the comment: max(buffer_size, n-avail) I mimicked the original logic rather than rethink the algorithm. I'm not totally sure what motivates the original logic but the purpose seems to be that non-blocking streams can return at least a few bytes

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Le lundi 21 juillet 2008 à 21:18 +, Martin v. Löwis a écrit : IIUC, a read of the full requested size would achieve exactly that: on a non-blocking stream (IIUC), a read will always return min(bytes_available, bytes_requested). Hmm, it

[issue2523] binary buffered reading is quadratic

Martin v. Löwis [EMAIL PROTECTED] added the comment: I don't understand the second loop (where n is given). If n is given, there should be only a single read operation, using max(buffer_size, n-avail) (i.e. the way it is in patch 2). In particular, if the stream is unbuffered, it shouldn't

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: If nobody objects I'll commit Alexandre's patch in a few days (after beta 2 though). ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2523 ___

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Yup. However, if you try it, you'll probably notice that it decreases performance of normal (blocking) reads as well :-) Anyway, non-blocking file objects are pretty much second-class citizens in Py3k right now, so my remark was theoretical.

[issue2523] binary buffered reading is quadratic

Alexandre Vassalotti [EMAIL PROTECTED] added the comment: Oh, that is simple to fix. You can round the value 2*avail to the nearest block by doing something like (2*avail) ~(bksize-1) where bksize is a power of 2, or the less magic (2*avail//bksize) * bksize.

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: I recommend not letting this issue rot too much :) Eating 20+ seconds to read the contents of a 10MB binary file in one pass is not very good marketing-wise, and the betas are coming soon... ___ Python

[issue2523] binary buffered reading is quadratic

Changes by Gregory P. Smith [EMAIL PROTECTED]: -- nosy: +gregory.p.smith priority: - high __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2523 __ ___ Python-bugs-list

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Some code relies on -1 being usable as the default value for read() (instead of None), this new patch conforms to this expectation. It fixes some failures in test_mailbox. Added file: http://bugs.python.org/file10222/binaryio2.patch

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Here is a pure Python patch removing the quadratic behaviour and trying to make read operations generally faster. Here are some numbers: ./python -m timeit -s f = open('50KB', 'rb') f.seek(0) while f.read(11): pass - py3k without patch: 23.6

[issue2523] binary buffered reading is quadratic

Alexandre Vassalotti [EMAIL PROTECTED] added the comment: I see that the code is still using the immutable bytes object for its buffer (which forces Python to create a new buffer every time its modified). Also, I think it worthwhile to check if using a pre-allocated bytearray object (i.e.,

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: Hi Alexandre, I first tried to use a (non-preallocated) bytearray object and, after trying several optimization schemes, I found out that the best one worked as well with an immutable bytes object :) I also found out that the bytes - bytearray

[issue2523] binary buffered reading is quadratic

Changes by Alexandre Vassalotti [EMAIL PROTECTED]: -- nosy: +alexandre.vassalotti __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2523 __ ___ Python-bugs-list mailing

[issue2523] binary buffered reading is quadratic

Antoine Pitrou [EMAIL PROTECTED] added the comment: By the way, a simple way to fix it would be to use a native BytesIO object (as provided by Alexandre's patch in #1751) rather than a str object for the underlying buffer. __ Tracker [EMAIL PROTECTED]

[issue2523] binary buffered reading is quadratic

New submission from Antoine Pitrou [EMAIL PROTECTED]: In py3k, buffered binary IO can be quadratic when e.g. reading a whole file. This is a small test on 50KB, 100KB and 200KB files: - py3k with buffering: ./python -m timeit -s f = open('50KB', 'rb') f.seek(0); f.read() 1000 loops, best of 3: