Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Thu, 2009-03-05 at 23:52 +0100, Henning Thielemann wrote: > On Wed, 4 Mar 2009, John Lato wrote: > > >> John A. De Goes schrieb: > >> > >>> Elsewhere, laziness can be a real boon, so I don't understand your > >>> question, "Why have laziness in Haskell at all?" > >> > >> As I have written, many libaries process their data lazily (or could be > >> changed to do so without altering their interface) but their interface > >> can forbid application to data that is fetched from the outside world. > >> Say you are used to 'map', 'filter', 'foldr' - you cannot use them on > >> data fetched by the iteratee/enumerator approach. > >> > > > > Thank you for replying to this; it's good to see what features people > > would like to make iteratees more useful. > > > > Where did you get the idea that you can't use 'map'? > > What I meant was, that you cannot just use the functions you are used from > Data.List. You need functions adapted to Iteratee. This also implies that > all libraries written in terms of Data.List functions cannot be used as > they are. Maybe it's a good time to review those libraries, whether they > need lists at all, or whether they would also work with functionality that > can be provided by Iteratees. The question, whether libraries should be > bound to lists and Strings did already arise with ByteStrings. So > Iteratees may be just one more reason to generalize the libraries. In most cases we can get the list version from a continuation based version. A continuation based version can also be used to implement an iterator version. Leaving aside the amount of work, I've been trying to think of how to do a iterator version at all for some libs. Specifically, my zlib binding. Currently the zlib package provides functions on lazy bytestrings: compress, decompress :: ByteString -> ByteString This hides all the complications of blocks and filling and draining buffers. However since it uses lazy bytestrings it is most naturally used with lazy io. The question is can I make an interface that also keeps the devotees of iterator based IO happy and also lets me implement the existing api above. I cannot see how to do it. Note that compress and decompress are pure. This is something that I want to keep. I don't want to go back to using IO for a pure operation like compression just for the sake of iterator IO. The problem of course is that while zlib compression is pure overall, there is an internal state thread. That must be threaded linearly. We cannot hand out a continuation that can be run multiple times. Lining up the zlib state thread with the memoising lazy list works perfectly. I cannot see how we do the same with an iterator interface. I'm not saying it cannot be done. On the contrary, I would like to see it done. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Wed, 4 Mar 2009, John Lato wrote: John A. De Goes schrieb: Elsewhere, laziness can be a real boon, so I don't understand your question, "Why have laziness in Haskell at all?" As I have written, many libaries process their data lazily (or could be changed to do so without altering their interface) but their interface can forbid application to data that is fetched from the outside world. Say you are used to 'map', 'filter', 'foldr' - you cannot use them on data fetched by the iteratee/enumerator approach. Thank you for replying to this; it's good to see what features people would like to make iteratees more useful. Where did you get the idea that you can't use 'map'? What I meant was, that you cannot just use the functions you are used from Data.List. You need functions adapted to Iteratee. This also implies that all libraries written in terms of Data.List functions cannot be used as they are. Maybe it's a good time to review those libraries, whether they need lists at all, or whether they would also work with functionality that can be provided by Iteratees. The question, whether libraries should be bound to lists and Strings did already arise with ByteStrings. So Iteratees may be just one more reason to generalize the libraries. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
jwlato: > On Tue, Mar 3, 2009 at 1:03 AM, Henning Thielemann > wrote: > > > > On Mon, 2 Mar 2009, John Lato wrote: > > > > While I think that the Iteratee pattern has benefits, I suspect that it > > can't be combined with regular lazy functions, e.g. of type [a] -> [a]. Say > > I have a chain of functions: read a file, parse it into a tag soup, parse > > that into an XML tree, transform that tree, format that into a string, write > > that to a file, and all of these functions are written in a lazy way, which > > is currently considered good style, I can't use them in conjunction with > > iteratees. This means, almost all Haskell libraries have to be rewritten or > > extended from lazy style to iteratee style. The question for me is then: Why > > having laziness in Haskell at all? Or at least, why having laziness by > > default, why not having laziness annotation instead of strictness > > annotation. > > > > I'm not sure that this is a problem, at least not for all cases. When > reading seekable streams, it is possible to have IO on demand provided > that all processing take place within the context of the Iteratee (see > Oleg's Tiff reader, http://okmij.org/ftp/Haskell/Iteratee/Tiff.hs, and > my wave reader, > http://inmachina.net/~jwlato/haskell/iteratee/src/Data/Iteratee/Codecs/Wave.hs). BTW, I've started (with his blessing) packaging up Oleg's Haskell code: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/liboleg-0.1.0.2 So you can use, e.g., the left-fold based TIFF parser: http://hackage.haskell.org/packages/archive/liboleg/0.1.0.2/doc/html/Codec-Image-Tiff.html I'm walking backwards over his released modules, adding a few each day. Enjoy. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
> John A. De Goes schrieb:
>
>> Elsewhere, laziness can be a real boon, so I don't understand your
>> question, "Why have laziness in Haskell at all?"
>
> As I have written, many libaries process their data lazily (or could be
> changed to do so without altering their interface) but their interface
> can forbid application to data that is fetched from the outside world.
> Say you are used to 'map', 'filter', 'foldr' - you cannot use them on
> data fetched by the iteratee/enumerator approach.
>
Thank you for replying to this; it's good to see what features people
would like to make iteratees more useful.
Where did you get the idea that you can't use 'map'? IterateeGM is a
Monad and Functor, so fmap is available for the resultant value. If
you want to map over the elements of the stream, use the 'mapStream'
function ('map_stream' in Oleg's writings).
I never thought about writing a general iteratee fold because that's
all iteratees are anyway; it's simple enough to make a foldl. It
should be possible to write foldr for seekable resources by
enumerating the stream backwards. That certainly wouldn't work for
network I/O, unfortunately. Note that this is essentially the
approach taken by lazy bytestrings, for example. There are other
elements I'd want to address before adding this, however. There are
also complications dealing with nested streams. I have some ideas,
but the way forward is far from clear.
filter should be fairly simple to implement with peek and drop. I'll
look into it.
Cheers,
John
___
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Tue, Mar 3, 2009 at 1:03 AM, Henning Thielemann wrote: > > On Mon, 2 Mar 2009, John Lato wrote: > > While I think that the Iteratee pattern has benefits, I suspect that it > can't be combined with regular lazy functions, e.g. of type [a] -> [a]. Say > I have a chain of functions: read a file, parse it into a tag soup, parse > that into an XML tree, transform that tree, format that into a string, write > that to a file, and all of these functions are written in a lazy way, which > is currently considered good style, I can't use them in conjunction with > iteratees. This means, almost all Haskell libraries have to be rewritten or > extended from lazy style to iteratee style. The question for me is then: Why > having laziness in Haskell at all? Or at least, why having laziness by > default, why not having laziness annotation instead of strictness > annotation. > I'm not sure that this is a problem, at least not for all cases. When reading seekable streams, it is possible to have IO on demand provided that all processing take place within the context of the Iteratee (see Oleg's Tiff reader, http://okmij.org/ftp/Haskell/Iteratee/Tiff.hs, and my wave reader, http://inmachina.net/~jwlato/haskell/iteratee/src/Data/Iteratee/Codecs/Wave.hs). Also, since the inner monad can be any monad, not just IO, you should be able to lift processing and computations into an iteratee in a fairly straightforward manner. File enumerators are only provided for IO, but it's fairly easy to create versions for other monads as necessary. I've got one for StateT s IO, for example. Now I do agree that this probably won't work in every case. I would suspect that parsers may have to be rewritten to use iteratees (although I don't know to what extent because I don't work with generic parsers). I'm not sure in what other cases this would also be true. The best way to figure it out would be to have more people using iteratees and reporting their findings. Cheers, John ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
John A. De Goes schrieb: > Elsewhere, laziness can be a real boon, so I don't understand your > question, "Why have laziness in Haskell at all?" As I have written, many libaries process their data lazily (or could be changed to do so without altering their interface) but their interface can forbid application to data that is fetched from the outside world. Say you are used to 'map', 'filter', 'foldr' - you cannot use them on data fetched by the iteratee/enumerator approach. Actually, Lazy I/O and exceptions can work together if you drop the exceptions that are baked into IO monad and use explicit exceptions (synchronous and asynchronous ones) as I develop them in the explicit-exception package. I'm however still searching for a good set of combinators. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Lazy IO is a complete disaster for "interactive IO", such as network and process IO. Moreover, it's somewhat of a failure even for non- interactive IO such as the use case you described, because it's very easy for partial evaluation to lead to unclosed files and lazy evaluation to lead to delayed resource acquisition. I can imagine a few use cases that might benefit from it, but the evidence suggests that most developers trying to solve "real world" problems work extra hard to get their programs working properly with lazy IO. Elsewhere, laziness can be a real boon, so I don't understand your question, "Why have laziness in Haskell at all?" Regards, John A. De Goes N-BRAIN, Inc. The Evolution of Collaboration http://www.n-brain.net|877-376-2724 x 101 On Mar 2, 2009, at 6:03 PM, Henning Thielemann wrote: On Mon, 2 Mar 2009, John Lato wrote: Hello, I am not a super-geek (at least, not compared to others on this list), but I'll take a try at this anyway. The benefits of iteratees mostly depend on differences between lazy and strict IO (see ch. 7 of Real World Haskell for more on this). Maybe a good text for http://www.haskell.org/haskellwiki/Enumerator_and_iteratee ? While I think that the Iteratee pattern has benefits, I suspect that it can't be combined with regular lazy functions, e.g. of type [a] - > [a]. Say I have a chain of functions: read a file, parse it into a tag soup, parse that into an XML tree, transform that tree, format that into a string, write that to a file, and all of these functions are written in a lazy way, which is currently considered good style, I can't use them in conjunction with iteratees. This means, almost all Haskell libraries have to be rewritten or extended from lazy style to iteratee style. The question for me is then: Why having laziness in Haskell at all? Or at least, why having laziness by default, why not having laziness annotation instead of strictness annotation. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Mon, 2 Mar 2009, John Lato wrote: Hello, I am not a super-geek (at least, not compared to others on this list), but I'll take a try at this anyway. The benefits of iteratees mostly depend on differences between lazy and strict IO (see ch. 7 of Real World Haskell for more on this). Maybe a good text for http://www.haskell.org/haskellwiki/Enumerator_and_iteratee ? While I think that the Iteratee pattern has benefits, I suspect that it can't be combined with regular lazy functions, e.g. of type [a] -> [a]. Say I have a chain of functions: read a file, parse it into a tag soup, parse that into an XML tree, transform that tree, format that into a string, write that to a file, and all of these functions are written in a lazy way, which is currently considered good style, I can't use them in conjunction with iteratees. This means, almost all Haskell libraries have to be rewritten or extended from lazy style to iteratee style. The question for me is then: Why having laziness in Haskell at all? Or at least, why having laziness by default, why not having laziness annotation instead of strictness annotation. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hi all, thanks again for the explanation. So I have the impression that this approach is something with recognized merit but in a way still considered work in progress as I do not really see it advertised for the general haskell audience. And there are some that study it further. Ok. My question now is: "Where do we take this from here?". Oleg seems to have published what he had to say on this years ago, takusen and hyena *use* it and beyond that there is only the occasional thread like this one. Anybody willing to take action so this stays on the radar? Günther ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
RE: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Mon, 2009-03-02 at 12:50 +, Bayley, Alistair wrote: > > From: haskell-cafe-boun...@haskell.org > > [mailto:haskell-cafe-boun...@haskell.org] On Behalf Of Duncan Coutts > > > > This can still be done using withFile and hGetContents. You > > just have to > > put the consumer inside the scope of withFile. The consumer > > can work in > > a streaming fashion. With lazy bytestrings this can be both efficient, > > work in constant memory and guarantee the file is closed. > > > > We guarantee the file is closed by using withFile. The only thing to > > watch out for is a consumer that doesn't consume as much as you were > > expecting before the file does get closed. You should notice > > that pretty > > quickly though since it should happen every time (whereas > > resource leaks > > are not so immediately visible). > > Sure. But this case is the one that typically causes problems for > beginners, who have not yet had the educating experience of being > bitten by lazy IO. The standard café response to "why do I get closed>" errors here?" is "you're using hGetContents and you haven't > forced/consumed all of your file". This is quite true, but I expect that's easier to explain to beginners than iterator IO, at least at the present time. Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On 2009 Mar 2, at 7:40, Manlio Perillo wrote: Bayley, Alistair ha scritto: All the file is consumed, before the result is returned. This only works if the entire file can reasonably fit into memory. It's not the entire file, but only the parsed data structure. ...which, depending on what you're doing with it, can be larger than the original file (consider lists). -- brandon s. allbery [solaris,freebsd,perl,pugs,haskell] allb...@kf8nh.com system administrator [openafs,heimdal,too many hats] allb...@ece.cmu.edu electrical and computer engineering, carnegie mellon universityKF8NH PGP.sig Description: This is a digitally signed message part ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
RE: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
> From: haskell-cafe-boun...@haskell.org > [mailto:haskell-cafe-boun...@haskell.org] On Behalf Of Duncan Coutts > > This can still be done using withFile and hGetContents. You > just have to > put the consumer inside the scope of withFile. The consumer > can work in > a streaming fashion. With lazy bytestrings this can be both efficient, > work in constant memory and guarantee the file is closed. > > We guarantee the file is closed by using withFile. The only thing to > watch out for is a consumer that doesn't consume as much as you were > expecting before the file does get closed. You should notice > that pretty > quickly though since it should happen every time (whereas > resource leaks > are not so immediately visible). Sure. But this case is the one that typically causes problems for beginners, who have not yet had the educating experience of being bitten by lazy IO. The standard café response to "why do I get " errors here?" is "you're using hGetContents and you haven't forced/consumed all of your file". Alistair * Confidentiality Note: The information contained in this message, and any attachments, may contain confidential and/or privileged material. It is intended solely for the person(s) or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer. * ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Bayley, Alistair ha scritto: From: haskell-cafe-boun...@haskell.org [mailto:haskell-cafe-boun...@haskell.org] On Behalf Of Manlio Perillo Sent: 02 March 2009 11:01 Eugene Kirpichov ha scritto: I'm not considering the lazy IO approach, as it doesn't involve any form of control over resources. This is not always true. I'm using lazy IO, still having full control over the resources. parse path = withFile path ReadMode parse' where parse' :: Handle -> IO (UArr Xxx) parse' handle = do contents <- L.hGetContents handle let v = toU $ xxx $ L.lines contents rnf v `seq` return v All the file is consumed, before the result is returned. This only works if the entire file can reasonably fit into memory. It's not the entire file, but only the parsed data structure. If you want to process something really big, then you need some sort of streaming approach, Yes, this is a more general solution. > [...] Manlio Perillo ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
RE: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
On Mon, 2009-03-02 at 11:50 +, Bayley, Alistair wrote: > > From: haskell-cafe-boun...@haskell.org > > [mailto:haskell-cafe-boun...@haskell.org] On Behalf Of Manlio Perillo > > Sent: 02 March 2009 11:01 > > > > Eugene Kirpichov ha scritto: > > > I'm not considering the lazy IO approach, as it doesn't involve any > > > form of control over resources. > > > > This is not always true. > > I'm using lazy IO, still having full control over the resources. > > > > parse path = withFile path ReadMode parse' > > where > >parse' :: Handle -> IO (UArr Xxx) > >parse' handle = do > > contents <- L.hGetContents handle > > let v = toU $ xxx $ L.lines contents > > rnf v `seq` return v > > > > All the file is consumed, before the result is returned. > This only works if the entire file can reasonably fit into memory. If > you want to process something really big, then you need some sort of > streaming approach, where you only look at a small part of the file (a > line, or a block) at a time. And this is where the enumerator-iteratee > approach looks better, because the IO is strict, but you still take a > stream-like approach to processing the contents. This can still be done using withFile and hGetContents. You just have to put the consumer inside the scope of withFile. The consumer can work in a streaming fashion. With lazy bytestrings this can be both efficient, work in constant memory and guarantee the file is closed. We guarantee the file is closed by using withFile. The only thing to watch out for is a consumer that doesn't consume as much as you were expecting before the file does get closed. You should notice that pretty quickly though since it should happen every time (whereas resource leaks are not so immediately visible). Duncan ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
strictly and released as soon as possible. I hope this is helpful. Cheers, John Lato > Message: 12 > Date: Mon, 02 Mar 2009 01:31:02 +0100 > From: G??nther Schmidt > Subject: [Haskell-cafe] Re: Left fold enumerator - a real pearl > overlooked? > To: haskell-cafe@haskell.org > Message-ID: > Content-Type: text/plain; charset=windows-1252; format=flowed > > Hi everyone, > > after reading all the responses I would like to ask someone, anyone, to > kind of summarize the merits of the left-fold-enumerator approach. > > From all that I read so far about it all I was able to gather was that > it has significance but I'm still not even sure what for and what not for. > > Apparently Oleg has done various CS work, this particular piece just > being one. But he also broaches the topic at very high level, ok, too > high for me, ie. no CS or higher math background. > > Would one of the super geeks please summarize it up? (In RWH kind of > style if possible) > > Günther > > > > John Lato schrieb: >> Hi Don, >> >> Would you please elaborate on what features or capabilities you think >> are missing from left-fold that would elevate it out of the special >> purpose category? I think that the conception is so completely >> different from bytestrings that just saying it's not a bytestring >> equivalent doesn't give me any ideas as to what would make it more >> useful. Since the technique is being actively developed and >> researched, IMO this is a good time to be making changes. >> >> Incidentally, in my package I've made newtypes that read data into >> strict bytestrings. It would be relatively simple to use >> unsafeInterleaveIO in an enumerator to create lazy bytestrings using >> this technique. I don't see why anyone would want to do so, however, >> since it would have all the negatives of lazy IO and be less efficient >> than simply using lazy bytestrings directly. >> >> Cheers, >> John >> >> On Sat, Feb 28, 2009 at 10:54 PM, Don Stewart wrote: >> >>>> There are a few iteratee/enumerator design questions that remain, >>>> which Oleg and others would like to explore more fully. The results >>>> of that research will likely find there way into this library. >>> >>> I agree. There's no left-fold 'bytestring' equivalent. So it remains a >>> special purpose technique. >>> >>> -- Don >>> > > > ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
RE: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
> From: haskell-cafe-boun...@haskell.org > [mailto:haskell-cafe-boun...@haskell.org] On Behalf Of Manlio Perillo > Sent: 02 March 2009 11:01 > > Eugene Kirpichov ha scritto: > > I'm not considering the lazy IO approach, as it doesn't involve any > > form of control over resources. > > This is not always true. > I'm using lazy IO, still having full control over the resources. > > parse path = withFile path ReadMode parse' > where >parse' :: Handle -> IO (UArr Xxx) >parse' handle = do > contents <- L.hGetContents handle > let v = toU $ xxx $ L.lines contents > rnf v `seq` return v > > All the file is consumed, before the result is returned. This only works if the entire file can reasonably fit into memory. If you want to process something really big, then you need some sort of streaming approach, where you only look at a small part of the file (a line, or a block) at a time. And this is where the enumerator-iteratee approach looks better, because the IO is strict, but you still take a stream-like approach to processing the contents. BTW, does this discussion remind anyone (else) of Peter Simon's Block-IO proposal? http://cryp.to/blockio/fast-io.html Alistair * Confidentiality Note: The information contained in this message, and any attachments, may contain confidential and/or privileged material. It is intended solely for the person(s) or entity to which it is addressed. Any review, retransmission, dissemination, or taking of any action in reliance upon this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer. * ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Eugene Kirpichov ha scritto: OK, I'm far from being a supergeek, but anyways. I'm not considering the lazy IO approach, as it doesn't involve any form of control over resources. This is not always true. I'm using lazy IO, still having full control over the resources. parse path = withFile path ReadMode parse' where parse' :: Handle -> IO (UArr Xxx) parse' handle = do contents <- L.hGetContents handle let v = toU $ xxx $ L.lines contents rnf v `seq` return v All the file is consumed, before the result is returned. > [...] Manlio Perillo ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
OK, I'm far from being a supergeek, but anyways. I'm not considering the lazy IO approach, as it doesn't involve any form of control over resources. With the traditional approach, you manually ask a stream do to something (read a block of bytes, seek to a position etc.), and your program is a mixture of stuff that asks the stream to do something and stuff that deals with the results. With the iteratee approach, you split the program into two parts: Iteratee (a thing that encapsulates the state of your work with the stream: either you're done and you don't need any new data, or you need another block of bytes to do more work (and you know *which* work), or you need to seek to a different position and you know what you're going to do after that) and Enumerator : a thing that 'runs' an iteratee, looking at what is is demanding, performing the corresponding action, giving its result to the iteratee and seeing what it wants next repeatedly. Simplifying all the monadic stuff, we end up with something like this: data Iteratee a = Done a | NeedAnotherChunk (Maybe Chunk -> Iteratee a) -- Will be given Nothing if we're at EOF | NeedToSeek Int (Maybe Chunk -> Iteratee a) type Enumerator a = Iteratee a -> Iteratee a The key point is that you, by construction, *always* know whether you need the stream or not. Thus, since the data processing loop is concentrated in one place, namely the particular enumerator, this loop *always* knows whether it's time to close the stream or not. It is time, if the iteratee has become a Done, or if the stream was closed or encountered an error. Another key point is that both the iteratees and enumerators are highly composable; and iteratees also form a monad, thus becoming suitable for easily writing parsers. Also, there's no recursion in the iteratees, and they are fusable and thus extremely performant. Further, you better read Oleg's article and code. 2009/3/2 Gü?nther Schmidt : > Hi everyone, > > after reading all the responses I would like to ask someone, anyone, to kind > of summarize the merits of the left-fold-enumerator approach. > > From all that I read so far about it all I was able to gather was that it > has significance but I'm still not even sure what for and what not for. > > Apparently Oleg has done various CS work, this particular piece just being > one. But he also broaches the topic at very high level, ok, too high for me, > ie. no CS or higher math background. > > Would one of the super geeks please summarize it up? (In RWH kind of style > if possible) > > Günther > > > > John Lato schrieb: >> >> Hi Don, >> >> Would you please elaborate on what features or capabilities you think >> are missing from left-fold that would elevate it out of the special >> purpose category? I think that the conception is so completely >> different from bytestrings that just saying it's not a bytestring >> equivalent doesn't give me any ideas as to what would make it more >> useful. Since the technique is being actively developed and >> researched, IMO this is a good time to be making changes. >> >> Incidentally, in my package I've made newtypes that read data into >> strict bytestrings. It would be relatively simple to use >> unsafeInterleaveIO in an enumerator to create lazy bytestrings using >> this technique. I don't see why anyone would want to do so, however, >> since it would have all the negatives of lazy IO and be less efficient >> than simply using lazy bytestrings directly. >> >> Cheers, >> John >> >> On Sat, Feb 28, 2009 at 10:54 PM, Don Stewart wrote: >> There are a few iteratee/enumerator design questions that remain, which Oleg and others would like to explore more fully. The results of that research will likely find there way into this library. >>> >>> I agree. There's no left-fold 'bytestring' equivalent. So it remains a >>> special purpose technique. >>> >>> -- Don >>> > > > ___ > Haskell-Cafe mailing list > Haskell-Cafe@haskell.org > http://www.haskell.org/mailman/listinfo/haskell-cafe > -- Eugene Kirpichov Web IR developer, market.yandex.ru ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hi everyone, after reading all the responses I would like to ask someone, anyone, to kind of summarize the merits of the left-fold-enumerator approach. From all that I read so far about it all I was able to gather was that it has significance but I'm still not even sure what for and what not for. Apparently Oleg has done various CS work, this particular piece just being one. But he also broaches the topic at very high level, ok, too high for me, ie. no CS or higher math background. Would one of the super geeks please summarize it up? (In RWH kind of style if possible) Günther John Lato schrieb: Hi Don, Would you please elaborate on what features or capabilities you think are missing from left-fold that would elevate it out of the special purpose category? I think that the conception is so completely different from bytestrings that just saying it's not a bytestring equivalent doesn't give me any ideas as to what would make it more useful. Since the technique is being actively developed and researched, IMO this is a good time to be making changes. Incidentally, in my package I've made newtypes that read data into strict bytestrings. It would be relatively simple to use unsafeInterleaveIO in an enumerator to create lazy bytestrings using this technique. I don't see why anyone would want to do so, however, since it would have all the negatives of lazy IO and be less efficient than simply using lazy bytestrings directly. Cheers, John On Sat, Feb 28, 2009 at 10:54 PM, Don Stewart wrote: There are a few iteratee/enumerator design questions that remain, which Oleg and others would like to explore more fully. The results of that research will likely find there way into this library. I agree. There's no left-fold 'bytestring' equivalent. So it remains a special purpose technique. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hi Don, Would you please elaborate on what features or capabilities you think are missing from left-fold that would elevate it out of the special purpose category? I think that the conception is so completely different from bytestrings that just saying it's not a bytestring equivalent doesn't give me any ideas as to what would make it more useful. Since the technique is being actively developed and researched, IMO this is a good time to be making changes. Incidentally, in my package I've made newtypes that read data into strict bytestrings. It would be relatively simple to use unsafeInterleaveIO in an enumerator to create lazy bytestrings using this technique. I don't see why anyone would want to do so, however, since it would have all the negatives of lazy IO and be less efficient than simply using lazy bytestrings directly. Cheers, John On Sat, Feb 28, 2009 at 10:54 PM, Don Stewart wrote: >> There are a few iteratee/enumerator design questions that remain, >> which Oleg and others would like to explore more fully. The results >> of that research will likely find there way into this library. > > > I agree. There's no left-fold 'bytestring' equivalent. So it remains a > special purpose technique. > > -- Don > ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hello, I'm not sure that I would call it a general-purpose resource preserving technique. As I understand it, the general concept is a means to handle strict data processing in a functional manner. Any "resource preserving" that comes from this is actually from the use of strict IO rather than lazy. I actually think it's rather like foldl' compared to foldl. John On Sat, Feb 28, 2009 at 11:16 PM, G?uenther Schmidt wrote: > So I was hopeful the above mentioned left-fold-enumerator was some sort of > general purpose resource preserving technique. I'm looking forward to study > its implementation in the future release of takusen and see if I grasp it > enough to translate it to other problems as well. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hi, thank you all for your responses. I see now that the subject did indeed register with some haskellers. :-) I had hoped that is will eventually become the tested and approved method for certain types of problems that do arise. Maybe some of you have read my earlier posts that I was developing an application, which is now finished btw. An initial design was using HAppS-IxSet instead of an SQL database, but it had hit performance problems, one a stack overflow and another a not enough RAM problem. The stack overflow problem I was eventually able to solve, though all kinds of strictness primitives would not help (!, seq and strict data constructors), following several different leads here on this list. But finally the not enough RAM problem occurred which I can't really explain why that happened, the memory consumption was so not linear to the input size, I may have caused that with my tricks of avoiding the stack overflow. Anyway, I eventually gave up and used SQLite. So I was hopeful the above mentioned left-fold-enumerator was some sort of general purpose resource preserving technique. I'm looking forward to study its implementation in the future release of takusen and see if I grasp it enough to translate it to other problems as well. And special thanks to you Don and your co-authors, your book saved my neck, with its help I was actually able to develop the application in an entirely new to me language and with a mere 3 months past deadline. Günther Don Stewart schrieb: jwlato: Hello Günther, I think the largest reason Haskellers don't use left-fold enumerators is that there isn't a ready-to-use package on Hackage. Oleg's code is extremely well commented and easy to follow, but it's not cabalized. In addition to Takusen, Johan Tibbe's hyena application server uses enumerators for IO: http://github.com/tibbe/hyena/tree/master There is a darcs repo of a cabalized iteratee package available at http://inmachina.net/~jwlato/haskell/iteratee/ This is essentially Oleg's code, slightly modified and reorganized. If anyone is interested in using left-fold enumerators for IO, please give it a look and let me know what you think. I'd like to put this on hackage in about a week or so, if possible. I would especially appreciate build reports. There are a few iteratee/enumerator design questions that remain, which Oleg and others would like to explore more fully. The results of that research will likely find there way into this library. I agree. There's no left-fold 'bytestring' equivalent. So it remains a special purpose technique. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
jwlato: > Hello Günther, > > I think the largest reason Haskellers don't use left-fold enumerators > is that there isn't a ready-to-use package on Hackage. Oleg's code is > extremely well commented and easy to follow, but it's not cabalized. > > In addition to Takusen, Johan Tibbe's hyena application server uses > enumerators for IO: > http://github.com/tibbe/hyena/tree/master > > There is a darcs repo of a cabalized iteratee package available at > http://inmachina.net/~jwlato/haskell/iteratee/ > This is essentially Oleg's code, slightly modified and reorganized. > If anyone is interested in using left-fold enumerators for IO, please > give it a look and let me know what you think. I'd like to put this > on hackage in about a week or so, if possible. I would especially > appreciate build reports. > > There are a few iteratee/enumerator design questions that remain, > which Oleg and others would like to explore more fully. The results > of that research will likely find there way into this library. I agree. There's no left-fold 'bytestring' equivalent. So it remains a special purpose technique. -- Don ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] Re: Left fold enumerator - a real pearl overlooked?
Hello Günther, I think the largest reason Haskellers don't use left-fold enumerators is that there isn't a ready-to-use package on Hackage. Oleg's code is extremely well commented and easy to follow, but it's not cabalized. In addition to Takusen, Johan Tibbe's hyena application server uses enumerators for IO: http://github.com/tibbe/hyena/tree/master There is a darcs repo of a cabalized iteratee package available at http://inmachina.net/~jwlato/haskell/iteratee/ This is essentially Oleg's code, slightly modified and reorganized. If anyone is interested in using left-fold enumerators for IO, please give it a look and let me know what you think. I'd like to put this on hackage in about a week or so, if possible. I would especially appreciate build reports. There are a few iteratee/enumerator design questions that remain, which Oleg and others would like to explore more fully. The results of that research will likely find there way into this library. Sincerely, John Lato > Hi all, > > in the last few months I was looking for haskell database library, > eventually settling for HDBC (thanks John btw). > > Takusen also caught my eye although I even failed installing it. > > Nevertheless a particular property of takusen, left-fold-enumerator, > also sparked my interest and I tried to follow it up. > > I had the impression this is something of relevance but something that > is not mentioned or made use of often. There are actually only a few > references on the net and most of those by one person only, Oleg. > > There was one post from John Goerzen about Haskell HTTP libraries in > which he hinted using left-fold enumerators. > > Anyway what I'm saying is that the whole topic is somewhat off the radar > of the larger haskell community, how come, or am I merely overestimating > its relevance and usefulness? > > Günther ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
