On 2/18/16 6:52 PM, Chad Joan wrote:
Steve: My apologies in advance if I a misunderstood any of the functionality of your IO library. I haven't read any of the documentation, just this thread, and I my time is over-committed as usual.
Understandable.
Anyhow... I believe that when I am dealing with streams, >90% of the time I am dealing with data that is *structured* and *heterogeneous*. Here are some use-cases: 1. Parsing/writing configuration files (ex: XML, TOML, etc) 2. Parsing/writing messages from some protocol, possibly over a network socket (or sockets). Example: I am writing a PostgreSQL client and need to deserialize messages: http://www.postgresql.org/docs/9.2/static/protocol-message-formats.html 3. Serializing/deserializing some data structures to/from disk. Example: I am writing a game and I need to implement save/load functionality. 4. Serializing/deserializing tabular data to/from disk (ex: .CSV files). 5. Reading/writing binary data, such as images or video, from/to disk. This will probably involve doing a bunch of (3), which is kind of like (2), but followed by large homogenous arrays of some data (ex: pixels). 6. Receiving unstructured user input. This is my <10%. Note that (6) is likely to happen eventually but also likely to be minuscule: why are we receiving user input? Maybe it's just to store it for retrieval later. BUT, maybe we actually want it to DO something. If we want it to do something, then we need to structure it before code will be able to operate on it. (5) is a mix of structured heterogeneous data and structured homogenous data. In aggregate, this is structured heterogeneous data, because you need to do parsing to figure out where the arrays of homogeneous data start and end (and what they *mean*). This is why I think it will be much more important to have at least these two interfaces take front-and-center: A. The presence of a .popAs!(...) operation (mentioned by Wyatt in this thread, IIRC) for simple deserialization, and maybe for other miscellaneous things like structured user interaction.
To me, this is a higher-level function. popAs cannot assume to know how to read what it is reading. If you mean something like reading an entire struct in binary form, that's not difficult to do.
B. The ability to attach parsers to streams easily. This might be as easy as coercing the input stream into the basic encoding that the parser expects (ex: char/wchar/dchar Ranges for compilers, or maybe ubyte Ranges for our PostgreSQL client's network layer), though it might need (A) to help a bit first if the encoding isn't known in advance (text files can be represented in sooo many ways! isn't it fabulous!).
This is the fundamental goal for my library -- enabling parsers to read data from a "stream" efficiently no matter how that data is sourced. I know your time is limited, but I would invite you to take a look at the convert program example that I created in my library. In it, I handle converting any UTF format to any other UTF format.
https://github.com/schveiguy/iopipe/blob/master/examples/convert/convert.d
I understand that most unsuspecting programmers will arrive at a stream library expecting to immediately see an InputRange interface. This /probably/ is not what they really want at the end of the day. So, I think it will be very important for any such library to concisely and convincingly explain the design methodology and rationale early and aggressively. Neglect to do this, and the library and it's documentation will become a frustration and a violation of expectations (an "astonishment"). Do it right, and the library's documentation will become a teaching tool that leaves visitors feeling enlightened and empowered.
Good points! I will definitely spend some time explaining this.
Of course, I have to wonder if someone else has contrasting experiences with stream use-cases. Maybe they really would be frustrated with a range-agnostic design. I don't want to alienate this hypothetical individual either, so if this is you, then please share your experiences. I hope this helps and is worth making a bunch of you read a wall of text ;)
Thanks for taking the time. -Steve
