[issue12053] Add prefetch() for Buffered IO (experiment)
Antoine Pitrou added the comment: The proposed API seems strange to me. Is there an illustration of how it might be used? I suspect it wouldn’t be all that useful, and could more or less be implemented with the existing methods: True, but having it a Buffered method would allow it to optimize buffer usage and avoid some memory copies (the first read() call and the peek() call, for example). In any case, Guido was rather against this proposal, so I'm not sure there's much sense in keeping discussing it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Martin Panter added the comment: Sounds like this might be more appropriate for the BufferedReader and related classes, and less so for the writer and abstract base class. The proposed API seems strange to me. Is there an illustration of how it might be used? I suspect it wouldn’t be all that useful, and could more or less be implemented with the existing methods: def prefetch(buffered_reader, buffer, skip, minread): buffered_reader.read(skip) consumed = buffered_reader.readinto(buffer[:minread]) if consumed minread: return consumed spare = len(buffer) - consumed extra = buffered_reader.peek(spare)[:spare] total = consumed + len(extra) buffer[consumed:total] = extra return total Maybe it would be better to focus on clarifying or redefining the existing peek() method (Issue 5811), rather than making a brand new do-everything method which only seems to do what the other methods already do. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
John O'Connor tehj...@gmail.com added the comment: Here is an update with the C implementation. I think a working prototype will be helpful before another round on python-dev. I'm not sure how to handle unseekable, non-blocking streams where the read returns before `skip` bytes are exhausted. If prefetch() returns 0, then the caller would then have to use tell() to ensure subsequent reads are sane. In other words it seems prefetch() will leave the stream in an unpredictable state. Antoine, what are your thoughts? -- Added file: http://bugs.python.org/file23308/prefetch.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Antoine Pitrou pit...@free.fr added the comment: I started a draft in python. I am attaching the _pyio version along with tests. I will continue work on the C implementation and eventually documentation if this is well received. It seems straightforward, I am interested to see what you guys think. Thank you. I think performance measurements are prematurate until we have an optimized C implementation anyway. I think ultimately we also want a default implementation of read(), peek() and read1() which uses prefetch(), so that BufferedReader implementations only have to implement prefetch(). (care must be taken to avoid infinite loops) That said, I think the python-dev mailing-list needs to be convinced of the usefulness of prefetch() (if it was only me, there wouldn't be any problem :-)). Perhaps you want to run another discussion there. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
John O'Connor tehj...@gmail.com added the comment:
I started a draft in python. I am attaching the _pyio version along with tests.
I will continue work on the C implementation and eventually documentation if
this is well received. It seems straightforward, I am interested to see what
you guys think.
Also, there are now 2 places which use hasattr(self, peek). I was wondering
if it would make sense to add peek() to BufferedIOBase and raise
UnsupportedOperation or return b.
Some benchmarks..
$ ./python -m timeit -s from _pyio import open;f = open('LICENSE', 'rb');
b=bytearray(128) 'while f.prefetch(b, 4, 4): pass'
_pyio.BufferedIOBase.prefetch:
10 loops, best of 3: 10.6 usec per loop
_pyio.BufferedReader.prefetch:
10 loops, best of 3: 6 usec per loop
$ ./python -m timeit -s from _pyio import open;f = open('LICENSE',
'rb');b=bytearray(4); 'while f.read(4): f.readinto(b)'
10 loops, best of 3: 5.07 usec per loop
--
keywords: +patch
Added file: http://bugs.python.org/file22168/issue12053-pyio.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12053
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Changes by John O'Connor tehj...@gmail.com: Added file: http://bugs.python.org/file22169/issue12053-tests.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
New submission from John O'Connor tehj...@gmail.com: A prefetch() method for Buffered IO may greatly assist 3rd party buffering among other gains. If nothing else, it is worth experimenting with. Discussion on the topic is here: http://mail.python.org/pipermail/python-ideas/2010-September/008180.html A summary of the method proposed (by Antoine Pitrou): prefetch(self, buffer, skip, minread) Skip `skip` bytes from the stream. Then, try to read at least `minread` bytes and write them into `buffer`. The file pointer is advanced by at most `skip + minread`, or less if the end of file was reached. The total number of bytes written in `buffer` is returned, which can be more than `minread` if additional bytes could be prefetched (but, of course, cannot be more than `len(buffer)`). Arguments: - `buffer`: a writable buffer (e.g. bytearray) - `skip`: number of bytes to skip (must be = 0) - `minread`: number of bytes to read (must be = 0 and = len(buffer)) -- components: IO messages: 135731 nosy: benjamin.peterson, haypo, jcon, pitrou, stutzbach priority: normal severity: normal status: open title: Add prefetch() for Buffered IO (experiment) type: feature request versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12053] Add prefetch() for Buffered IO (experiment)
Changes by Nadeem Vawda nadeem.va...@gmail.com: -- nosy: +nadeem.vawda ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue12053 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
