On Wed, 13 Oct 2010 20:55:04 +0400, Andrei Alexandrescu
<seewebsiteforem...@erdani.org> wrote:
On 10/13/10 11:16 CDT, Denis Koroskin wrote:
On Wed, 13 Oct 2010 18:32:15 +0400, Andrei Alexandrescu
So far so good. I will point out, however, that the classic read/write
routines are not all that good. For example if you want to implement a
line-buffered stream on top of a block-buffered stream you'll be
forced to write inefficient code.
Never heard of filesystems that allow reading files in lines - they
always read in blocks, and that's what streams should do.
http://www.gnu.org/s/libc/manual/html_node/Buffering-Concepts.html
I don't think streams must mimic the low-level OS I/O interface.
I in contrast think that Streams should be a lowest-level possible
platform-independent abstraction.
No buffering besides what an OS provides, no additional functionality. If
you need to be able to read something up to some character (besides, what
should be considered a new-line separator: \r, \n, \r\n?), this should be
done manually in "byLine".
That's because
most of the steams are binary streams, and there is no such thing as a
"line" in them (e.g. how often do you need to read a line from a
SocketStream?).
http://www.opengroup.org/onlinepubs/009695399/functions/isatty.html
These are special cases I don't like. There is no such thing in Windows
anyway.
You need a line when e.g. you parse a HTML header or a email header or
an FTP response. Again, if at a low level the transfer occurs in blocks,
that doesn't mean the API must do the same at all levels.
BSD sockets transmits in blocks. If you need to find a special sequence in
a socket stream, you are forced to fetch a chunk, and manually search for
a needed sequence. My position is that you should do it with an external
predicate (e.g. read until whitespace).
I don't think streams should buffer anything either (what an underlying
OS I/O API caches should suffice), buffered streams adapters can do that
in a stream-independent way (why duplicate code when you can do that as
efficiently with external methods?).
Most OS primitives don't give access to their own internal buffers.
Instead, they ask user code to provide a buffer and transfer data into
it.
Right. This is why Stream may not cache.
So clearly buffering on the client side is a must.
I don't see how is it implied from above.
Besides, as you noted, the buffering is redundant for byChunk/byLine
adapter ranges. It means that byChunk/byLine should operate on
unbuffered streams.
Chunks keep their own buffer so indeed they could operate on streams
that don't do additional buffering. The story with lines is a fair
amount more complicated if it needs to be done efficiently.
Yes. But line-reading is a case that I don't see a need to be handled
specially.
I'll explain my I/O streams implementation below in case you didn't read
my message (I've changed some stuff a little since then).
Honest, I opened it to remember to read it but somehow your fonts are
small and make my eyes hurt.
My Stream
interface is very simple:
// A generic stream
interface Stream
{
@property InputStream input();
@property OutputStream output();
@property SeekableStream seekable();
@property bool endOfStream();
void close();
}
You may ask, why separate Input and Output streams?
I think my first question is: why doesn't Stream inherit InputStream and
OutputStream? My hypothesis: you want to sometimes return null. Nice.
Right.
Well, that's because
you either read from them, write from them, or both.
Some streams are read-only (think Stdin), some write-only (Stdout), some
support both, like FileStream. Right?
Sounds good. But then where's flush()? Must be in OutputStream.
That's probably because unbuffered streams don't need them.
Not exactly. Does FileStream support writing when you open file for
reading? Does it support reading when you open for writing?
So, you may or may not read from a generic stream, and you also may or
may not write to a generic stream. With a design like that you can make
a mistake: if a stream isn't readable, you have no reference to invoke
read() method on.
That is indeed pretty nifty. I hope you would allow us to copy that
feature in Phobos (unless you are considering submitting your library
wholesale). Let me know.
Would love to contribute with design and implementation.
Similarly, a stream is either seekable, or not. SeekableStreams allow
stream cursor manipulation:
interface SeekableStream : Stream
{
long getPosition(Anchor whence = Anchor.begin);
void setPosition(long position, Anchor whence = Anchor.begin);
}
Makes sense. Why is getPosition signed? Why do you need an anchor for
getPosition?
long is chosen to be consistent with setPosition. Also getPosition may
return a negative value:
long pos = getPosition(Anchor.end); // how far is it till file end?
Also this is how you can get file size (need to invert though). This is
consistent with setPosition:
setPosition(getPosition(anchor), anchor); // a no-op for any kind of achor
I just thought why not? I'm okay with dropping it, but I find it nice.
InputStream doesn't really has many methods:
interface InputStream
{
// reads up to buffer.length bytes from a stream
// returns number of bytes read
// throws on error
size_t read(ubyte[] buffer);
That makes implementation of line buffering inefficient :o).
There is no way you can do it more efficient on Windows. Fetch a chunk;
search for a line end; found ? return : continue.
// reads from current position
AsyncReadRequest readAsync(ubyte[] buffer, Mailbox* mailbox = null);
}
Why doesn't Sean's concurrency API scale for your needs? Can that be
fixed? Would you consider submitting some informed bug reports?
It's rather a design issue than a bug on its own. I'll write a separate
letter on that.
So is OutputStream:
interface OutputStream
{
// returns number of bytes written
// throws on error
size_t write(const(ubyte)[] buffer);
// writes from current position
AsyncWriteRequest writeAsync(const(ubyte)[] buffer, Mailbox* mailbox =
null);
}
They basically support only reading and writing in blocks, nothing else.
I'm surprised there's no flush().
No buffering - no flush.
However, they support asynchronous reads/writes, too (think of mailbox
as a std.concurrency's Tid).
Unlike Daniel's proposal, my design reads up to buffer size bytes for
two reasons:
- it avoids potential buffering and multiple sys calls
But there's a problem. It's very rare that the user knows what a good
buffer size is. And often there are size and alignment restrictions at
the low level.
I agree, but he can guess. Or a library can give him a hint. E.g.
BUFFER_SIZE is a good buffer size to start with :)
So somewhere there is still buffering going on, and also there are
potential inefficiencies (if a user reads small buffers).
- it is the only way to go with SocketStreams. I mean, you often don't
know how many bytes an incoming socket message contains. You either have
to read it byte-by-byte, or your application might stall for potentially
infinite time (if message was shorter than your buffer, and no more
messages are being sent)
But if you don't know how many bytes are in an incoming socket message,
a better design is to do this:
void read(ref ubyte[] buffer);
That could work, too.
and resize the buffer to accommodate the incoming packet. Your design
_imposes_ that the socket does additional buffering.
The socket API does it anyway. I just don't complicate it even further but
providing an additional layer of buffering.
Why do my streams provide async methods? Because it's the modern
approach to I/O - blocking I/O (aka one thread per client) doesn't
scale. E.g. Java adds a second revision of Async I/O API in JDK7 (called
NIO2, first appeared in February, 2002), C# has asynchronous operations
as part of their Stream interface since .NET 1.1 (April, 2003).
Async I/O is nice, no two ways about that. I have on my list to define
byChunkAsync that works exactly like byChunk from the client's
perspective, except it does I/O concurrently with client code.
[snip]
I strongly believe we shouldn't ignore this type of API.
P.S. For threads this deep it's better fork a new one, especially when
changing the subject.
I thought I did by changing the title...
Andrei
No, changing title isn't enough.
