On 25 February 2016 at 06:50, Steven D'Aprano
<[email protected]> wrote:
>
> I have a need to read to an arbitrary delimiter, which might be any of a
> (small) set of characters. For the sake of the exercise, lets say it is
> either ! or ? (for example).
>
> I want to read from files reasonably efficiently. I don't mind if there is a
> little overhead, but my first attempt is 100 times slower than the built-in
> "read to the end of the line" method.
You can get something much faster using mmap and searching for a
single delimiter:
def readuntil(m, delim):
start = m.tell()
index = m.find(delim, start)
if index == -1:
return m.read()
else:
return m.read(index - start)
def readmmap(f):
m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
f.seek(0)
while True:
chunk = readuntil(m, b'!') # Note byte-string
if not chunk:
return
# Do stuff with chunk
pass
My timing makes that ~7x slower than iterating over the lines of the
file but still around 100x faster than reading individual characters.
I'm not sure how to generalise it to looking for multiple delimiters
without dropping back to reading individual characters though.
--
Oscar
--
https://mail.python.org/mailman/listinfo/python-list
