[Trimming the CCs. All CCed persons are subscribed to the list.]
Hi Chris,
> I think we can do this more efficiently. First, we can use freadptr() to
> peek at the stdio internal read buffer without copying, which resolves
> to a direct struct member access on glibc (_IO_read_ptr/_IO_read_end),
> BSD (_p/_r), and most other platforms.
>
> Second, we can use memchr() to scan the available buffered data for the
> delimiter. On modern glibc memchr() is SIMD optimised (even without
> -march=native since it's called as IFUNC), so we do a significantly
> fewer number of vectorised scans.
>
> Third, once the delimiter position is known, we can use fread() (which
> is remapped to fread_unlocked() on glibc) to consume exactly those bytes
> from the stream into our linebuffer. Since the data is already in the
> stdio buffer, this is effectively a memcpy.
I like the idea, and the 2x speed increase is what one can expect.
(Vaguely remembering the speed increase that I observed when doing a
similar optimization to GNU m4, 1.6.x branch.)
Around 2007-2010, we still had hesitations to use the modules that
rely on lib/stdio-impl.h, for portability reasons. These hesitations are
gone, because libc vendors nowadays tend to implement the necessary
primitives (freading, fwriting, freadptr, etc.) by themselves, so GNU
programs can use them.
Before applying your patch, we would need two things, though:
* A copyright assignment from you to the FSF for this change. This is
necessary because Gnulib's copyright is with the FSF, and your patch
is sizeable (see [1]). Would you be willing to start the process for
doing so? I can guide you through it via private email.
* A unit test for the 'linebuffer' module. This is necessary because
your patch is changing the major part of the implementation.
It does not need to be you who contributes this unit test; any
Gnulib contributor (like Collin, Paul, or me) can step in for this.
Thanks for presenting this nice idea and the patch!
Bruno
[1] https://www.gnu.org/prep/maintain/html_node/Legally-Significant.html
