Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
Excerpts from Henning Thielemann's message of Sun Mar 22 23:58:44 +0100 2009: On Sun, 22 Mar 2009, nicolas.pouillard wrote: It sounds like a nice idea, it would be great to have a straight-io package to play a bit more with explicit exceptions in things like 'IO'. Maybe I should then restrict lifting to LazyIO to SIO actions. That would not make LazyIO safe, but reduces surprises. By SIO you actually mean straight-io right? I was confused because I also have an SIO monad in the strict-io package. -- Nicolas Pouillard ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
On Mon, 23 Mar 2009, nicolas.pouillard wrote: Excerpts from Henning Thielemann's message of Sun Mar 22 23:58:44 +0100 2009: On Sun, 22 Mar 2009, nicolas.pouillard wrote: It sounds like a nice idea, it would be great to have a straight-io package to play a bit more with explicit exceptions in things like 'IO'. Maybe I should then restrict lifting to LazyIO to SIO actions. That would not make LazyIO safe, but reduces surprises. By SIO you actually mean straight-io right? Yes I was confused because I also have an SIO monad in the strict-io package. Sorry ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
Excerpts from Henning Thielemann's message of Mon Mar 23 11:06:20 +0100 2009: On Mon, 23 Mar 2009, nicolas.pouillard wrote: Excerpts from Henning Thielemann's message of Sun Mar 22 23:58:44 +0100 2009: On Sun, 22 Mar 2009, nicolas.pouillard wrote: It sounds like a nice idea, it would be great to have a straight-io package to play a bit more with explicit exceptions in things like 'IO'. Maybe I should then restrict lifting to LazyIO to SIO actions. That would not make LazyIO safe, but reduces surprises. By SIO you actually mean straight-io right? Yes Then what do you mean by lifting to LazyIO to SIO actions? Do you mean liftSIO :: SIO a - LazyIO.T a which says that we only lift computations that explicitly throws exceptions. In that case it be actually safer, but all of this greatly depends on how reasonable is the explicit exception handling. In particular in the case 'IO', using explicit exception is maybe too heavy. -- Nicolas Pouillard ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
On Mon, 23 Mar 2009, nicolas.pouillard wrote: Excerpts from Henning Thielemann's message of Mon Mar 23 11:06:20 +0100 2009: Yes Then what do you mean by lifting to LazyIO to SIO actions? Do you mean liftSIO :: SIO a - LazyIO.T a which says that we only lift computations that explicitly throws exceptions. Yes. In that case it be actually safer, but all of this greatly depends on how reasonable is the explicit exception handling. If it does not fit, you can change it. :-) That's the advantage over built-in IO exceptions. In particular in the case 'IO', using explicit exception is maybe too heavy. I think it's precisely the best thing to do, given all the problems with asynchronous, imprecise and what-know-I exceptions. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
Excerpts from Henning Thielemann's message of Sat Mar 21 22:27:08 +0100 2009: On Fri, 20 Mar 2009, Nicolas Pouillard wrote: Hi folks, We have good news (nevertheless we hope) for all the lazy guys standing there. Since their birth, lazy IOs have been a great way to modularly leverage all the good things we have with *pure*, *lazy*, *Haskell* functions to the real world of files. Maybe you know of my packages lazy-io and explicit-exception which also aim at lazy I/O and asynchronous exception handling. I was indeed aware of these two packages but I think they hold orthogonal ideas. About the lazy-io package, as explained in the documentation one has to carefully choose which operations can be lifted. In safe-lazy-io I try to choose a set of well behaving combinators to replace 'getContents' in the IO monad. Moreover if I take the three problems of standard lazy IO in turn: 1/ Control of resources: One advantage over standard lazy IO is that the file opening can also be done lazily, avoiding an immediate resource exhaustion. However one still relies on evaluation and garbage collection to take care of closing handles, which is not satisfying since handles are scarce resources. 2/ Control of exceptions: If one writes a 'getContents' function such that it no longer hides I/O errors during reading, how do you guarantee that exceptions will happen during the LazyIO.run and not after? 3/ Determinism: when freely combining multiple inputs one risks the problem mentioned by Oleg [1], when using your package it will depend on the 'getContents' function we use: a) if we 'liftIO' the standard 'getContents' function, we can have the issue. b) if we write a new 'getContents' as below [2], then (if I got right your lazy IO monad) all reads are chained. And then one has to process inputs in the same order. However I've found the underlying idea of your monad brilliant. I've tried a little to use something similar as a base for the implementation but didn't succeed. With lazy-io, you are able to write more complicated things than getContents. I needed this for HTTP communication that is run by demand. That is when the HTTP response header is requested, then the function could send a HTTP request first. Is it possible and sensible to combine this with safe-lazy-io? While currently focusing only on reading file handles, the long term purpose for this technique is to have new primitives like reading on sockets, using bytestrings... I have also code that demonstrates the usage of explicit asynchronous exceptions. I have however still not a set of combinators that makes working with asynchronous exceptions as simple as working with synchronous ones: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/spreadsheet I also think that explicit asynchronous exceptions could be part of the equation, however I currently don't know how to mix them well. Best regards, [1]: http://www.haskell.org/pipermail/haskell/2009-March/021064.html [2] hGetContents :: Handle - LIO.T String hGetContents h = lazyRead where lazyRead = do isEOF - liftIO $ hIsEOF h if isEOF then do unit - liftIO $ hClose h return $ unit `seq` [] else do c - liftIO $ hGetChar h cs- lazyRead return $ c : cs -- Nicolas Pouillard ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
On Sun, 22 Mar 2009, nicolas.pouillard wrote: Excerpts from Henning Thielemann's message of Sat Mar 21 22:27:08 +0100 2009: Maybe you know of my packages lazy-io and explicit-exception which also aim at lazy I/O and asynchronous exception handling. I was indeed aware of these two packages but I think they hold orthogonal ideas. About the lazy-io package, as explained in the documentation one has to carefully choose which operations can be lifted. In safe-lazy-io I try to choose a set of well behaving combinators to replace 'getContents' in the IO monad. Moreover if I take the three problems of standard lazy IO in turn: 1/ Control of resources: One advantage over standard lazy IO is that the file opening can also be done lazily, avoiding an immediate resource exhaustion. However one still relies on evaluation and garbage collection to take care of closing handles, which is not satisfying since handles are scarce resources. 2/ Control of exceptions: If one writes a 'getContents' function such that it no longer hides I/O errors during reading, how do you guarantee that exceptions will happen during the LazyIO.run and not after? Currently I cannot guarantee anything. However my idea is to stay away from built-in exceptions in IO. In explicit-exception there is an experimental hidden module which provides an IO monad wrapper called SIO which cannot throw any IO exception. http://code.haskell.org/explicit-exception/src/System/IO/Straight.hs Actually, I think it's the wrong way round to build an exception-free monad on top of one with exceptions. Instead IO should be built on top of SIO, but that's not possible for historical reasons. The only safe operation to get into SIO is ioToExceptionalSIO :: IO a - ExceptionalT IOException SIO a That is, it makes exceptions explicit and SIO operations can never throw IO exceptions. You should convert synchronous explicit exceptions of atomic operations like getChar into asynchronous explicit exceptions, combine them lazily to big operations like getContents. Then you get getContents :: SIO (Asynchronous.Exception IOException String) If you run lazy SIO operations you can't become surprised by exceptions. 3/ Determinism: when freely combining multiple inputs one risks the problem mentioned by Oleg [1], when using your package it will depend on the 'getContents' function we use: a) if we 'liftIO' the standard 'getContents' function, we can have the issue. b) if we write a new 'getContents' as below [2], then (if I got right your lazy IO monad) all reads are chained. And then one has to process inputs in the same order. I wouldn't build hClose into getContents, because you never know, whether the file is read until it's end. If you call a LazyIO.getContents twice, the contents are read sequential. In order to read file contents simultaneously you must call (LazyIO.run LazyIO.getContents) twice in the IO monad. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
Excerpts from Henning Thielemann's message of Sun Mar 22 22:52:48 +0100 2009: On Sun, 22 Mar 2009, nicolas.pouillard wrote: Excerpts from Henning Thielemann's message of Sat Mar 21 22:27:08 +0100 2009: Maybe you know of my packages lazy-io and explicit-exception which also aim at lazy I/O and asynchronous exception handling. I was indeed aware of these two packages but I think they hold orthogonal ideas. About the lazy-io package, as explained in the documentation one has to carefully choose which operations can be lifted. In safe-lazy-io I try to choose a set of well behaving combinators to replace 'getContents' in the IO monad. Moreover if I take the three problems of standard lazy IO in turn: 1/ Control of resources: One advantage over standard lazy IO is that the file opening can also be done lazily, avoiding an immediate resource exhaustion. However one still relies on evaluation and garbage collection to take care of closing handles, which is not satisfying since handles are scarce resources. 2/ Control of exceptions: If one writes a 'getContents' function such that it no longer hides I/O errors during reading, how do you guarantee that exceptions will happen during the LazyIO.run and not after? Currently I cannot guarantee anything. However my idea is to stay away from built-in exceptions in IO. In explicit-exception there is an experimental hidden module which provides an IO monad wrapper called SIO which cannot throw any IO exception. http://code.haskell.org/explicit-exception/src/System/IO/Straight.hs Actually, I think it's the wrong way round to build an exception-free monad on top of one with exceptions. Instead IO should be built on top of SIO, but that's not possible for historical reasons. The only safe operation to get into SIO is ioToExceptionalSIO :: IO a - ExceptionalT IOException SIO a That is, it makes exceptions explicit and SIO operations can never throw IO exceptions. You should convert synchronous explicit exceptions of atomic operations like getChar into asynchronous explicit exceptions, combine them lazily to big operations like getContents. Then you get getContents :: SIO (Asynchronous.Exception IOException String) If you run lazy SIO operations you can't become surprised by exceptions. It sounds like a nice idea, it would be great to have a straight-io package to play a bit more with explicit exceptions in things like 'IO'. For safe-lazy-io I wanted to keep the exception management as light as possible. In particular when writing programs where most of the 'IO' errors are considered fatals---EOF is not fatal of course but using getContents one do not see it. 3/ Determinism: when freely combining multiple inputs one risks the problem mentioned by Oleg [1], when using your package it will depend on the 'getContents' function we use: a) if we 'liftIO' the standard 'getContents' function, we can have the issue. b) if we write a new 'getContents' as below [2], then (if I got right your lazy IO monad) all reads are chained. And then one has to process inputs in the same order. I wouldn't build hClose into getContents, because you never know, whether the file is read until it's end. If you call a LazyIO.getContents twice, the contents are read sequential. In order to read file contents simultaneously you must call (LazyIO.run LazyIO.getContents) twice in the IO monad. Right but one of the purposes of safe-lazy-io is to provides a good management of file handles in particular closing them. Actually the implementation of lazy inputs focus particularly on that---through the 'Finalized' values. http://hackage.haskell.org/packages/archive/safe-lazy-io/0.1/doc/html/src/System-IO-Lazy-Input-Internals.html -- Nicolas Pouillard ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
On Sun, 22 Mar 2009, nicolas.pouillard wrote: It sounds like a nice idea, it would be great to have a straight-io package to play a bit more with explicit exceptions in things like 'IO'. Maybe I should then restrict lifting to LazyIO to SIO actions. That would not make LazyIO safe, but reduces surprises. ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] [ANN] Safe Lazy IO in Haskell
On Fri, 20 Mar 2009, Nicolas Pouillard wrote: Hi folks, We have good news (nevertheless we hope) for all the lazy guys standing there. Since their birth, lazy IOs have been a great way to modularly leverage all the good things we have with *pure*, *lazy*, *Haskell* functions to the real world of files. Maybe you know of my packages lazy-io and explicit-exception which also aim at lazy I/O and asynchronous exception handling. With lazy-io, you are able to write more complicated things than getContents. I needed this for HTTP communication that is run by demand. That is when the HTTP response header is requested, then the function could send a HTTP request first. Is it possible and sensible to combine this with safe-lazy-io? http://hackage.haskell.org/cgi-bin/hackage-scripts/package/lazyio http://hackage.haskell.org/cgi-bin/hackage-scripts/package/explicit-exception I have also code that demonstrates the usage of explicit asynchronous exceptions. I have however still not a set of combinators that makes working with asynchronous exceptions as simple as working with synchronous ones: http://hackage.haskell.org/cgi-bin/hackage-scripts/package/spreadsheet ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
[Haskell-cafe] [ANN] Safe Lazy IO in Haskell
Hi folks, We have good news (nevertheless we hope) for all the lazy guys standing there. Since their birth, lazy IOs have been a great way to modularly leverage all the good things we have with *pure*, *lazy*, *Haskell* functions to the real world of files. We are happy to present the safe-lazy-io package [1] that does exactly this and is going to be explained and motivated in the rest of this post. === The context === Although these times were hard with the Lazy/IO technique, some people continue to defend them arguing that all discovered problems about it was not that harmful and that taking care was sufficient. Indeed some issues have been discovered about Lazy/IOs, some have been fixed in the underlying machinery, some have just been hidden and some others are still around. == An alternative == An alternative design has been proposed --and is still evolving--, it is called Iteratee [2] and has been designed by Oleg Kiselyov. This new design has tons of advantages over standard imperative IOs, and shares some of the goals of Lazy/IOs. Iteratee provides a way to do incremental processing in a high-level style. Indeed both processed data (via enumerators) and processing code (called iteratee) can be modularly composed. The handling of file-system resources is precise and safe. Catching errors can be done precisely and can be interleaved with the processing. In spite of all this, there is an important drawback: a lot of code has to be re-written and thought in another way. Processing becomes explicitly chunked which is not always needed and, even worse, exceptions handling also becomes very explicit. While this makes sense in a wide range of applications it makes things less natural than the general case of pure functions. We think that Iteratee have too be studied more, and we recommend them when you have incrementally react to IO errors. == Issues of Standard Lazy/IO == We think that we can save Lazy/IO cheaply, but before explaining the way we solve such and such issue, let's first expose Lazy/IO and its issues. One of the main Lazy/IO functions is 'readFile': it takes a file path opens it and returns the list of characters until the end of the file is reached. The characteristic of 'readFile' is that only the opening is done strictly, while the reading is performed lazily as much as the output list is processed. Cousins of 'readFile' are 'hGetContents' that takes a file handle and 'getContents' that reads on the standard input. This technique enables to process a file as if the file was completely stored in memory. Because it is read lazily one knows that only the required part of the file will be read. Even better, if the input is consumed to produce a small output or the output is emitted incrementally, then the processing can be done in constant memory space. Examples: -- Prints the number of words read on stdin countWords = print . length . words = getContents -- Prints the length of the longest line maxLineLen = print . maximum . map length . lines = getContents -- Prints in lower case the text read on stdin lowerText = interact (map toLower) -- Alternatively lowerText = putStr . map toLower = getContents All these examples are pretty idiomatic Haskell code and make a simple use of Lazy/IOs. Each of them runs in constant memory space even if they are declared as if the whole contents were available at once. By using stream fusion or 'ByteString''s one can get even faster code while keeping the code almost the same. Here we will stay with the default list of 'Char''s data type. However one goal of our approach is to be trivially adaptable to those data types. Using our library will be rougly a matter of namespace switch plus a running function: lowerText = LI.run' (SIO.putStr . map toLower $ LI.getContents) However we will introducing this library as one goes along. Here is another example where the Lazy/IO are still easy to use but no longer scales well. This program counts the lines of all the files given in arguments: countLines = print . length . lines . concat = mapM readFile = getArgs Here the problem is the limitation of simultaneous opened files. Indeed, all the files are opened at the beginning therefore reaching the limit easily. It's time to recall when the files are closed. With standard Lazy/IOs the handle is closed when you reach the end of the file, and so when you've explored the whole list returned by 'readFile'. Note also that if you manually open the file and get a handle, then you can manually close the file, however if by misfortune you close the file and then still consume the lazy list you will get a truncated list, observing how much of the file has been read. This last point is due to the fact that 'readFile' considers the reading error as the end of the file. In particular one can fix this program, by simply counting the number of lines of each file separately and then compute the sum to get the final result. countLines = print . sum =
