It's no secret that I've been looking to create an updated io library
for phobos. In fact, I've been working on one on and off since 2011 (ouch).
After about 5 iterations of API and design, and testing out ideas, I
think I have come up with something pretty interesting. It started out
as a plan to replace std.stdio (and that did not go over well:
https://forum.dlang.org/post/j3u0l4$1atr$1...@digitalmars.com), in addition
to trying to find a better way to deal with i/o. However, I've scaled
back my plan of world domination to just try for the latter, and save
tackling the replacement of Phobos's i/o guts for a later battle, if at
all. It's much easier to reason about something new than to muddle the
discussion with how it will break code. It's also much easier to build
something that doesn't have to be a drop-in replacement of something so
insanely complex.
I also have been inspired over the last few years by various great
presentations and libraries, two being Dmitry's proof-of-concept library
to have buffers that automatically move/fill when more data is needed,
and Andrei's std.allocator library. They have changed drastically the
way I have approached this challenge.
Therefore, I now have a new dub-based repository available for playing
with: https://github.com/schveiguy/iopipe. First, the candy:
- This is a piping library. It allows one to hook buffered i/o through
various processors/transformers much like unix pipes or range
functions/algorithms. However, unlike unix pipes, this library attempts
to make as few copies as possible of the data.
example:
foreach(line; (new IODevice(0)).bufferedInput
.asText!(UTFType.UTF8)
.byLine
.asInputRange)
// handle line
- It can handle 5 forms of UTF encoding - UTF8, UTF16, UTF16LE, UTF32,
UTF32LE (phobos only partially handles UTF8). Sorry, no grapheme support
or other utf-related things, but this of course can be added later.
- Arrays are first-class ioPipe types. This works:
foreach(line; "one\ntwo\nthree\nfour\n".byLine.asInputRange)
- Everything is compile-time for the most part, and uses lots of
introspection. The intent is to give the compiler full gamut of
optimization capabilities.
- I added rudimentary compression/decompression support using
etc.c.zlib. Using compression is done like so:
foreach(line; (new IODevice(0)).bufferedInput
.unzip
.asText!(UTFType.UTF8)
.byLine
.asInputRange)
- The plan is for this to be a basis to make super-fast and modular
parsing libraries. I plan to write a JSON one as a proof of concept. So
all you have to do is add a parseJSON function to the end of any chain,
as long as the the input is some pipe of text data (including a string
literal).
=================
I will stress some very very important things:
1. This library is FAR from finished. Even the concepts probably need
some tweaking. But I'm very happy with the current API/usage.
2. Docs are very thin. Unit tests are sparse (but do pass).
3. The focus of this library is NOT replacement of std.stream, or even
low-level i/o in general. In fact, I have copied over my stream class
from previous attempts at this i/o rewrite ONLY as a mechanism to have
something that can read/write from file descriptors with the right API
(located in iopipe/stream.d). I admit to never having looked at
std.stream really, so I have no idea how it would compare.
4. As the stream framework is only for playing with the other useful
parts of the library, I only wrote it for my OS (OSX), so you won't be
able to play out of the box on Windows (probably can be added without
much effort, or use another stream library such as this one that was
recently announced:
https://forum.dlang.org/post/xtxiuxcmewxnhseub...@forum.dlang.org), but
it will likely work on other Unixen.
5. This is NOT thread-aware out of the box.
6. There is a concept in here I called "valves". It's very weird, but it
allows unifying input and output into one seamless chain. In fact, I
can't think of how I could have done output in this regime without them.
See the convert example application for details on how it is used.
7. I expect to be changing the buffer API, as I think perhaps I have the
wrong abstraction for buffers. However, I did attempt to have a
std.allocator version of the buffer.
8. It's not on code.dlang.org yet. I'll work on this.
Destroy!
-Steve
