Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
Starting a new thread from Denis' question: Can we outline basic Stream interface now so that we could move on? Here's the input transport layer. Transport has no concern for formatting - it just moves stuff around. interface TransportBase { /// True if stream has no more data. @pro

Re: Streaming transport interfaces: input

2010-10-14 Thread Pelle
On 10/14/2010 05:34 PM, Andrei Alexandrescu wrote: Starting a new thread from Denis' question: Can we outline basic Stream interface now so that we could move on? Here's the input transport layer. Transport has no concern for formatting - it just moves stuff around. interface TransportBase {

Re: Streaming transport interfaces: input

2010-10-14 Thread anon
Andrei Alexandrescu Wrote: > Starting a new thread from Denis' question: > > > Can we outline basic Stream interface now so that we could move on? > > Here's the input transport layer. Transport has no concern for > formatting - it just moves stuff around. > > interface TransportBase > { >

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/2010 11:43 AM, Pelle wrote: Why is positionable in TransportBase? Should it not be a separate interface? It could. Also, shouldn't the reads accept any output range? appendDelim as well. This would do away with the two extra versions in the text input. Interfaces can't have overrid

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/2010 12:15 PM, anon wrote: I can't understand why we need binary vs. text modes. Seems to me a silly design decision from the old days when we used 7-bit terminals or something. This always complicates the APIs for no reason. A variety of Internet protocols are text-based. Also, for ex

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Starting a new thread from Denis' question: Can we outline basic Stream interface now so that we could move on? Here's the input transport layer. Transport has no concern for formatting - it just moves stuff around. interf

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Please, use the term "seek", and allow an anchor. Every OS allows this, it makes no sense not to provide it. I've always thought that's a crappy appendix. Every OS that ever allows

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
With generic element type: interface InputBinaryTransport(T) : TransportBase if (!hasPointers!T && !is(T == class)) { /// Reads up to target.length items into target, /// returns number of items read. If target.length < 1 throws. size_t read(T[] target); /// Appends to target up u

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Thu, 14 Oct 2010 21:39:03 +0400, Andrei Alexandrescu wrote: On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Please, use the term "seek", and allow an anchor. Every OS allows this, it makes no sense not to provide it. I'v

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Thu, 14 Oct 2010 21:42:04 +0400, Andrei Alexandrescu wrote: With generic element type: interface InputBinaryTransport(T) : TransportBase if (!hasPointers!T && !is(T == class)) { /// Reads up to target.length items into target, /// returns number of items read. If target.length <

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 13:39:03 -0400, Andrei Alexandrescu wrote: On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Please, use the term "seek", and allow an anchor. Every OS allows this, it makes no sense not to provide it. I'v

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 12:56 CDT, Denis Koroskin wrote: appendDelim *requires* buffering for to be implemented. No OS provides an API to read from a file (be it pipe, socket, whatever) to read up to some abstract delimiter. It *always* reads in blocks. Clear. What may be not so clear is that read(ubyte[]

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 13:56:19 -0400, Denis Koroskin <2kor...@gmail.com> wrote: Buffering requires and additional level of data copying, and this is bad for fast I/O. If you need fast I/O or must pull that out of the stream interface. Otherwise chunked read will be less efficient due to add

Re: Streaming transport interfaces: input

2010-10-14 Thread anon
Andrei Alexandrescu Wrote: > On 10/14/2010 12:15 PM, anon wrote: > > I can't understand why we need binary vs. text modes. Seems to me a > > silly design decision from the old days when we used 7-bit terminals > > or something. This always complicates the APIs for no reason. > > A variety of Inte

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 13:14 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 13:39:03 -0400, Andrei Alexandrescu wrote: On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Please, use the term "seek", and allow an anchor. Every OS allow

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Thu, 14 Oct 2010 22:22:07 +0400, Andrei Alexandrescu wrote: On 10/14/10 12:56 CDT, Denis Koroskin wrote: appendDelim *requires* buffering for to be implemented. No OS provides an API to read from a file (be it pipe, socket, whatever) to read up to some abstract delimiter. It *always* read

Re: Streaming transport interfaces: input

2010-10-14 Thread Dmitry Olshansky
On 14.10.2010 22:14, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 13:39:03 -0400, Andrei Alexandrescu wrote: On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wrote: Please, use the term "seek", and allow an anchor. Every OS allow

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 14:43:56 -0400, Andrei Alexandrescu wrote: On 10/14/10 13:14 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 13:39:03 -0400, Andrei Alexandrescu wrote: On 10/14/10 12:27 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 11:34:12 -0400, Andrei Alexandrescu wro

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 14:00 CDT, Denis Koroskin wrote: On Thu, 14 Oct 2010 22:22:07 +0400, Andrei Alexandrescu wrote: On 10/14/10 12:56 CDT, Denis Koroskin wrote: appendDelim *requires* buffering for to be implemented. No OS provides an API to read from a file (be it pipe, socket, whatever) to read up

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrej Mitrovic
Now say that 10 times fast! On 10/14/10, Andrei Alexandrescu wrote: > ..lower level layer. > > Andrei >

Re: Streaming transport interfaces: input

2010-10-14 Thread Michel Fortin
On 2010-10-14 16:47:13 -0400, "Steven Schveighoffer" said: stream.position = StreamBase.atEnd; OK, that is acceptable. What about seeking to N bytes before the end? What about seeking N bytes ahead of the current position (as previously stated)? Would be nice if we could do all this:

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Fri, 15 Oct 2010 02:01:55 +0400, Andrei Alexandrescu wrote: On 10/14/10 14:00 CDT, Denis Koroskin wrote: On Thu, 14 Oct 2010 22:22:07 +0400, Andrei Alexandrescu wrote: On 10/14/10 12:56 CDT, Denis Koroskin wrote: appendDelim *requires* buffering for to be implemented. No OS provides a

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/2010 07:57 PM, Denis Koroskin wrote: In my original version there was a setbuf(f, null) call. I removed it because it had 0 impact on performance. I also tried using unistd open/read functions, that had zero impact, too Yah, I looked more into the behavior of fread. At least on a coupl

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Fri, 15 Oct 2010 05:49:05 +0400, Andrei Alexandrescu wrote: On 10/14/2010 07:57 PM, Denis Koroskin wrote: In my original version there was a setbuf(f, null) call. I removed it because it had 0 impact on performance. I also tried using unistd open/read functions, that had zero impact, too

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 16:47:13 -0400, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 14:43:56 -0400, Andrei Alexandrescu How? Denis' implementation has two copies in the mix. (I'm not counting .dup etc.) Anyhow, let's do this - write down your interfaces so I can comment on them. We talk

Re: Streaming transport interfaces: input

2010-10-14 Thread Steven Schveighoffer
On Thu, 14 Oct 2010 19:30:21 -0400, Michel Fortin wrote: On 2010-10-14 16:47:13 -0400, "Steven Schveighoffer" said: stream.position = StreamBase.atEnd; OK, that is acceptable. What about seeking to N bytes before the end? What about seeking N bytes ahead of the current position (as

Re: Streaming transport interfaces: input

2010-10-14 Thread Brad Roberts
On 10/14/2010 8:34 AM, Andrei Alexandrescu wrote: > Starting a new thread from Denis' question: > >> Can we outline basic Stream interface now so that we could move on? > > Here's the input transport layer. Transport has no concern for formatting - it > just moves stuff around. > I suggest havi

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 21:58 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 16:47:13 -0400, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 14:43:56 -0400, Andrei Alexandrescu How? Denis' implementation has two copies in the mix. (I'm not counting .dup etc.) Anyhow, let's do this - write down yo

Re: Streaming transport interfaces: input

2010-10-14 Thread Andrei Alexandrescu
On 10/14/10 23:37 CDT, Brad Roberts wrote: On 10/14/2010 8:34 AM, Andrei Alexandrescu wrote: Starting a new thread from Denis' question: Can we outline basic Stream interface now so that we could move on? Here's the input transport layer. Transport has no concern for formatting - it just mov

Re: Streaming transport interfaces: input

2010-10-14 Thread Denis Koroskin
On Fri, 15 Oct 2010 08:37:24 +0400, Brad Roberts wrote: On 10/14/2010 8:34 AM, Andrei Alexandrescu wrote: Starting a new thread from Denis' question: Can we outline basic Stream interface now so that we could move on? Here's the input transport layer. Transport has no concern for forma

Re: Streaming transport interfaces: input

2010-10-15 Thread Andrei Alexandrescu
On 10/14/2010 09:01 PM, Denis Koroskin wrote: On Fri, 15 Oct 2010 05:49:05 +0400, Andrei Alexandrescu wrote: Btw, I've re-run my tests, O_DIRECT works but also has no effect. Best of 3 results for parsing 33Mb log file: using getline (C++ -O2): real 0m0.081s user 0m0.068s sys 0m0.016s using by

Re: Streaming transport interfaces: input

2010-10-15 Thread Denis Koroskin
On Fri, 15 Oct 2010 20:06:47 +0400, Andrei Alexandrescu wrote: On 10/14/2010 09:01 PM, Denis Koroskin wrote: On Fri, 15 Oct 2010 05:49:05 +0400, Andrei Alexandrescu wrote: Btw, I've re-run my tests, O_DIRECT works but also has no effect. Best of 3 results for parsing 33Mb log file: using g

Re: Streaming transport interfaces: input

2010-10-15 Thread Andrei Alexandrescu
On 10/15/10 11:11 CDT, Denis Koroskin wrote: On Fri, 15 Oct 2010 20:06:47 +0400, Andrei Alexandrescu wrote: On 10/14/2010 09:01 PM, Denis Koroskin wrote: On Fri, 15 Oct 2010 05:49:05 +0400, Andrei Alexandrescu wrote: Btw, I've re-run my tests, O_DIRECT works but also has no effect. Best of 3

Re: Streaming transport interfaces: input

2010-10-15 Thread Steven Schveighoffer
On Fri, 15 Oct 2010 00:37:55 -0400, Andrei Alexandrescu wrote: On 10/14/10 21:58 CDT, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 16:47:13 -0400, Steven Schveighoffer wrote: On Thu, 14 Oct 2010 14:43:56 -0400, Andrei Alexandrescu How? Denis' implementation has two copies in the mix