On 12-03-10 05:16 AM, Paolo Capriotti wrote:
On Sat, Mar 10, 2012 at 4:21 AM, Mario Blažević<[email protected]> wrote:
I like your design, it seems to strike a good balance between elegance
and practicality. The only thing missing for the latter is a deeper support
for chunking. Of course, that would probably destroy some of the elegance
[1]. I don't think that problem has been solved in any of the
enumerator/iteratee/pipe/wire/conduit libraries so far.
Chunking is supported but not by primitive constructs. The way you
implement chunked streams is to simply use some form of "container"
representing a chunk as your input/output type.
Of course, that means that the abstraction is now operating at the
level of chunks instead of elements, which may be inconvenient, but I
doubt that there exists a way to "lift" element operations to chunks
in an efficient and general way.
Another issue is how to deal with unconsumed input. For that, there is
a ChunkPipe type (in pipes-extra) with a specialized monad instance
that threads unconsumed input along. You can see an example of
ChunkPipe in action in this prototype http server by Jeremy Shaw:
http://src.seereason.com/pipes-http-2/pipes-http-2/. Note that this is
based on a old version of pipes-core, however.
The only sane way I've found to deal with chunks is to move the
responsibility into the glue logic, which would be your (>+>) and (>>)
combinators. The upstream argument of (>+>) could then produce chunks of
any size it finds suitable, while the downstream argument would specify
exactly how much input it needs without having to worry about the
upstream chunk boundaries. How these requests are phrased is an open
question. I've developed the incremental-parser package specifically for
this purpose. Other approaches are possible, but I'm convinced that
chunking should not be left to individual components. The chunk type
probably shouldn't be reflected even in their types.
Did you consider adding some stream-splitting and merging pipes, like
those in the SCC package [2] or those described in the last Monad.Reader
issue [3]? Your arrow-like combinators seem well thought out, but they
don't go very far.
I'm not sure why you say that they don't go very far. I looked at
Splitter and Join in Monad.Reader 19, and they seem equivalent to
'splitP' and 'joinP' in pipes-core.
The main purpose of the Splitter type is to act as a conditional,
sending each input item *either* into the left or into the right output,
marking is as either true or false. Your splitterP sends each item to
*both* left and right output. It's a tee, not a split.
If your point is that the Splitter a m r type is isomorphic to Pipe
a (Either a a) m r, that is true. There is a benefit to the abstraction,
though. Once you introduce chunking into the picture, however, the
Splitter type can be changed under the hood to send an entire chunk to
its left or right output. The corresponding efficient chunked Pipe type
would be Pipe [a] (Either [a] [a]) m r, which is not at all the same as
Pipe [a] [Either a a] m r -- if you have to pack every single item of
the input chunk into an Either value, you've lost all performance
benefits of chunking. The former type is efficient but I'm not sure if
it would allow you to abstract the chunking logic out of the individual
components.
There shouldn't be any problem implementing all the other combinators
there in terms of monoidal primitives (e.g. 'not' is just 'swap').
I agree, with the chunking reservation above. Anyway, consider
adding the Boolean combinators to the library. I find them quite intuitive.
_______________________________________________
Haskell-Cafe mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/haskell-cafe
