On Wed, 16 May 2012 16:30:43 -0400, H. S. Teoh <hst...@quickfur.ath.cx>
wrote:
On Wed, May 16, 2012 at 04:15:22PM -0400, Steven Schveighoffer wrote:
On Wed, 16 May 2012 15:38:02 -0400, H. S. Teoh
<hst...@quickfur.ath.cx> wrote:
[...]
>One direction that _could_ be helpful, perhaps, is to extend the
concept
>of range to include, let's tentatively call it, a ChunkedRange.
>Basically a ChunkedRange implements the usual InputRange operations
>(empty, front, popfront) but adds the following new primitives:
>
>- bool hasAtLeast(R)(R range, int n) - true if underlying range has at
> least n elements left;
>
>- E[] frontN(R)(R range, int n) - returns a slice containing the front
n
> elements from the range: this will buffer the next n elements from
the
> range if they aren't already; repeated calls will just return the
> buffer;
>
>- void popN(R)(R range, int n) - discards the first n elements from the
> buffer, thus causing the next call to frontN() to fetch more data if
> necessary.
>
On such ranges, what would popFront and front do? I'm assuming since
frontN and popN are referring to how many elements, and since the most
logical definition for elements is bytes, that front gets the next
byte, and popFront discards the next byte. This seems useless to me.
How so? It's still useful for implementing readByte, for example.
readByte is covered by frontN(1). Why the need for front()?
Let me answer that question for you -- so it can be treated as a normal
range. But nobody will want to do that.
i.e. copy to appender will read one byte at a time into the array.
I still don't get the need to "add" this to ranges. The streaming API
works fine on its own.
But there is an omission with your proposed API regardless --
reading data is a mutating event. It destructively mutates the
underlying data stream so that you cannot get the data again. This
means you must double-buffer data in order to support frontN and
popN that are not necessary with a simple read API.
For example:
auto buf = new ubyte[1000000];
stream.read(buf);
does not need to first buffer the data inside the stream and then
copy it to buf, it can read it from the OS *directly* into buf.
[...]
The idea is that by asking for N elements at a time instead of calling
front/popFront N times, the underlying implementation can optimize the
request by creating a buffer of size N and have the OS read exactly N
bytes directly into that buffer.
// Reads 1,000,000 bytes into newly allocated buffer and returns
// buffer.
auto buf = stream.frontN(1_000_000);
OK, so stream is providing data via return value and allocation.
// Since 1,000,000 bytes is already read into the buffer, this
// simply returns a slice of the same buffer:
auto buf2 = stream.frontN(1_000_000);
Is buf2 mutable? If so, this is no good, buf could have mutated this
data. But this can be fixed by making the return value of frontN be
const(ubyte)[].
assert(buf is buf2);
// This consumes the buffer:
stream.popN(1_000_000);
What does "consume" mean, discard? Obviously not "reuse", due to line
below...
// This will read another 1,000,000 bytes into a new buffer
auto buf3 = stream.frontN(1_000_000);
OK, you definitely lost me here, this will not fly. The whole point of
buffering is to avoid having to reallocate on every read. If you have to
allocate every read, "buffering" is going to have a negative impact on
performance!
-Steve
