On Thu, 14 Oct 2010 14:43:56 -0400, Andrei Alexandrescu
<[email protected]> wrote:
On 10/14/10 13:14 CDT, Steven Schveighoffer wrote:
On Thu, 14 Oct 2010 13:39:03 -0400, Andrei Alexandrescu
<[email protected]> wrote:
On 10/14/10 12:27 CDT, Steven Schveighoffer wrote:
On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu
<[email protected]> wrote:
Please, use the term "seek", and allow an anchor. Every OS allows
this,
it makes no sense not to provide it.
I've always thought that's a crappy appendix. Every OS that ever
allows seek/tell with anchors allows ALL anchors, and always allows
either both or none of seek and tell. So I decided to cut through the
crap and simplify. You want to seek 100 bytes from here, you write
stream.position = stream.position + 100.
Um.. yuck. We need to use two system calls to seek 100 bytes?
seek and tell don't always issue system calls.
seek *is* a system call (lseek64). tell is simply seeking with (0,
SEEK_CUR).
Are we talking about the same thing?
Oh, that reminds me I need to provide length as a property as well.
This would save us crap like seek(0, SEEK_END); ftell() to figure out
the length of a file.
So now you need to do stream.position = stream.length to seek to the end
of the file instead of stream.seek(0, Anchor.END)?
Yes.
Plus, how will you
implement length, probably like this:
auto curpos = seek(0, SEEK_CUR);
auto len = seek(0, SEEK_END);
seek(curpos, SEEK_BEG);
return len;
Depends. For files, you can just use stat.
True, but still, 2 system calls (stat and then seek).
So that looks like 3 system calls instead of one, plus you just wasted
time seeking back to the current position.
Well again they don't always issue system calls, but point taken. I do
see a need for fast positioning at end of stream. Perhaps we could
accommodate an enum equal to ulong.max such that this goes to the end of
stream:
stream.position = StreamBase.atEnd;
OK, that is acceptable. What about seeking to N bytes before the end?
What about seeking N bytes ahead of the current position (as previously
stated)?
I don't like appendDelim. We don't need to define that until we have
buffering.
Why?
Because appendDelim deals with buffering. If I defined a buffered
stream, I'd include a function like this:
size_t read(bool delegate(T[] data) sink);
which buffers data until sink returned false (passing each read chunk
into sink), extending the buffer as necessary.
Then it's trivial to implement readDelim on top of this.
Interesting. But that would still force readDelim to store leftover
bytes.
What does readDelim do with them in your implementation? It must read
data via a block read, that's the only thing the OS provides (via read
system call), so what do you do, just return the extra data or throw it
away? You need to buffer it somewhere.
The simple function of an input stream is to read data.
It does read data.
I mean, that's *all* it should do. It should not be appending to
buffers.
This comes from a practical need. I've often had a buffer and wanted to
read one more line into it, keeping the existing content. It was
impossible without extra allocation and copying.
This can be accomodated via a buffer type. Buffering provides everything
you need to implement readDelim.
I think the appendDelim method allows fast and simple implementations
of a variety of patterns. As I (thought I) have shown elsethread,
without appendDelim there's no way to efficiently implement a
line-oriented stream on top of a block-oriented one.
Um... the read system call is the same interface as the proposed
block-oriented interface. How are you avoiding using system calls?
I think we don't have the same definition for "system call". For example
by my definition fread is NOT a system call.
OK, now I think I see the issue :) You are assuming we are implementing
all of this on top of FILE *. FILE * provides buffering already, so you
are not avoiding buffering at all and you are unnecessarily using an
extremely outdated interface.
BTW, fread eventually calls read, there's no way around it.
I think we can provide a version of the BufferedStream interface which
uses C's FILE * for stdout/stdin/stderr (to play nice with C), but we
should avoid FILE * for everything else.
Basically, appendDelim can be defined outside this class, because the
primitive read is enough.
You can only define it if you accept extra copying. I'd say one extra
interface function is acceptable for fast I/O.
No, you can define it without extra copying.
How? Denis' implementation has two copies in the mix. (I'm not counting
.dup etc.) Anyhow, let's do this - write down your interfaces so I can
comment on them. We talk "oh that's a buffering interface" and "that
requires buffering" and "that's an extra copy" and so on but we have
little concrete contenders. I put my cards on the table, you put yours.
I'll see if I can put something together.
If you don't allow direct
access to the buffer, then you have extra copying. But we don't have to
mimic C here. We should not be encouraging constant reinventing of the
buffer wheel here. Buffering is a well-defined task that can be
implemented once.
Just as a note, Tango does this, and it's very fast. There is certainly
no extra copying there.
Shouldn't the text transport be defined on top of the binary
transport?
No, because there are transports that genuinely do not accept binary
data.
I mean, a text transport uses a binary transport underneath. What text
transport doesn't use a binary transport to do its dirty work? And what
exactly does a text transport do so differently that it needs to be a
separate interface?
A text transport does not accept raw binary data and requires e.g.
Base64 encoding (e.g. mail attachments do that). The console is a text
device - makes no sense to dump binary data on it. A JSON encoder is
also a text transport.
-- that writes to/reads from a binary transport. A file is a binary
transport.
My opinion is that a text reader/writer should be a class/struct that uses
a binary transport. If you are going to implement the binary transport
stuff also inside the text transport, then it seems like an unnecessary
duplication.
And in any case, I'd expect buffering to go between the two.
How do you define buffering? Would a buffered transport implement a
different interface?
Yes, but if we expect to reuse code, I'd expect a buffered transport to
use a primitive transport underneath for actually reading/writing data.
If you have multiple versions of the class that actually reads/writes
data (such as binary vs. text), then the buffer which uses it must
support all of them.
Text based processing to me seems to be a buffered activity (reading
lines, ensuring you don't have sliced utf-8 data, etc.).
Yes. What may be not so obvious is that binary processing with
user-imposed data lenghts is ALSO a buffer activity. This is because the
low-level buffers do NOT come at arbitrary positions (alignment
restrictions) and to NOT come at arbitrary lengths.
You can still avoid copying. I'll see if I can show you.
-Steve
