On laziness and with-open
I wanted to grab bytes out of a stream, and didn't see an analogue to reader from duck-streams, so I made my own: (defn byte-seq Returns the bytes from stream as a lazy sequence of ints. stream must implement java.io.InputStream. [#^java.io.InputStream stream] (lazy-seq (let [b (. stream (read))] (if (= b 0) (cons b (byte-seq stream)) Then I did a simple lazy operation on a stream of bytes, say, to drop the first 5: (with-open [st (FileInputStream. mike.clj)] (drop 5 (byte-seq st))) I was a little surprised at first getting java.io.IOException: Bad file descriptor, but of course it hit me: the laziness persists well beyond the .close() in the with-open. I modified my byte-seq to close the stream when EOF is reached, but this is an awful ugly solution having the inner thing know when to close the outer concern's thing. (What if the outer thing wants to rewind? etc.) Is there a pattern out there in Clojure for handling laziness at the same time as handling resource lifecycle (with-open, etc.)? Thanks again in advance, Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: On laziness and with-open
On Thu, Jul 9, 2009 at 12:10, Mikecki...@gmail.com wrote: I wanted to grab bytes out of a stream, and didn't see an analogue to reader from duck-streams, so I made my own: (defn byte-seq Returns the bytes from stream as a lazy sequence of ints. stream must implement java.io.InputStream. [#^java.io.InputStream stream] (lazy-seq (let [b (. stream (read))] (if (= b 0) (cons b (byte-seq stream)) Then I did a simple lazy operation on a stream of bytes, say, to drop the first 5: (with-open [st (FileInputStream. mike.clj)] (drop 5 (byte-seq st))) I was a little surprised at first getting java.io.IOException: Bad file descriptor, but of course it hit me: the laziness persists well beyond the .close() in the with-open. I modified my byte-seq to close the stream when EOF is reached, but this is an awful ugly solution having the inner thing know when to close the outer concern's thing. (What if the outer thing wants to rewind? etc.) Is there a pattern out there in Clojure for handling laziness at the same time as handling resource lifecycle (with-open, etc.)? I'd just like to pile-on to say that I ran into the same question last night in a slightly different form. I have a file containing a sequence of clojure forms, which I want to read in sequentially. I don't have to have them all in memory at the same time for my task. A lazy sequence which yielded parsed forms while consuming bytes from the underlying file would be a perfect solution, but I ran into the same issue. I close the file when EOF is reached. I can (but don't yet, because it was a quick hack late at night) catch IOExceptions and close the underlying file under those circumstances, but this is ugly. What happens if the caller grabs such a lazy sequence and then just drops it on the floor? Does the file ever get closed before the VM exits? Do InputStreams register a finalizer? Needed: a pattern for handing a lazy sequence backed by a closeable resource. // ben --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: On laziness and with-open
Hi Mike, On Thu, Jul 9, 2009 at 12:10 PM, Mikecki...@gmail.com wrote: I wanted to grab bytes out of a stream, and didn't see an analogue to reader from duck-streams, so I made my own: (defn byte-seq Returns the bytes from stream as a lazy sequence of ints. stream must implement java.io.InputStream. [#^java.io.InputStream stream] (lazy-seq (let [b (. stream (read))] (if (= b 0) (cons b (byte-seq stream)) Then I did a simple lazy operation on a stream of bytes, say, to drop the first 5: (with-open [st (FileInputStream. mike.clj)] (drop 5 (byte-seq st))) I was a little surprised at first getting java.io.IOException: Bad file descriptor, but of course it hit me: the laziness persists well beyond the .close() in the with-open. I modified my byte-seq to close the stream when EOF is reached, but this is an awful ugly solution having the inner thing know when to close the outer concern's thing. (What if the outer thing wants to rewind? etc.) Is there a pattern out there in Clojure for handling laziness at the same time as handling resource lifecycle (with-open, etc.)? I think you have two ways to deal with this: (1) Include all your processing inside the with-open body, or (2) realize the seq by calling doall. With (1) you can take advantage of the laziness by using only the parts of the seq you really need, but you have to have all your processing in there which may seem a bit messy when there is a lot of processing code. With (2) you can return the complete seq for processing in other places, so you can encapsulate the reading in a function, but you have all the data in memory. You could also combine these approaches, by doing some preprocessing (like dropping 5 elements) and then returning the realized result seq. (with-open [st (FileInputStream. msghub.clj)] (doall (drop 5 (byte-seq st I hope this helps Janico --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: On laziness and with-open
I ran the code you pasted here. It didn't throw an IOException for me. I am running 1.0. On Jul 9, 5:10 am, Mike cki...@gmail.com wrote: I wanted to grab bytes out of a stream, and didn't see an analogue to reader from duck-streams, so I made my own: (defn byte-seq Returns the bytes from stream as a lazy sequence of ints. stream must implement java.io.InputStream. [#^java.io.InputStream stream] (lazy-seq (let [b (. stream (read))] (if (= b 0) (cons b (byte-seq stream)) Then I did a simple lazy operation on a stream of bytes, say, to drop the first 5: (with-open [st (FileInputStream. mike.clj)] (drop 5 (byte-seq st))) I was a little surprised at first getting java.io.IOException: Bad file descriptor, but of course it hit me: the laziness persists well beyond the .close() in the with-open. I modified my byte-seq to close the stream when EOF is reached, but this is an awful ugly solution having the inner thing know when to close the outer concern's thing. (What if the outer thing wants to rewind? etc.) Is there a pattern out there in Clojure for handling laziness at the same time as handling resource lifecycle (with-open, etc.)? Thanks again in advance, Mike --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: On laziness and with-open
On Thu, Jul 9, 2009 at 16:43, eyerisdrewpvo...@gmail.com wrote: I ran the code you pasted here. It didn't throw an IOException for me. I am running 1.0. I suspect that You're Doing it Wrong. You'll see the exception only if you actually try to evaluate the lazy sequence returned by byte-seq. (import [java.io InputStream FileInputStream]) (defn byte-seq [#^java.io.InputStream stream] (lazy-seq (let [b (. stream (read))] (if (= b 0) (cons b (byte-seq stream)) (defn most [] (with-open [st (FileInputStream. /home/smithma/.bashrc)] (drop 5 (byte-seq st (def x (most)) ;; this is OK, it just binds x to the lazy seq returned by (most) (first x) ;; not ok, as it tries to get the first element of x, which dies ;; since the file we're reading from is already closed. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: On laziness and with-open
That pages says the scopes system is already designed. To you have any preliminary design docs posted somewhere? On Jul 9, 2:59 pm, Stuart Sierra the.stuart.sie...@gmail.com wrote: On Jul 9, 6:10 am, Mike cki...@gmail.com wrote: Is there a pattern out there in Clojure for handling laziness at the same time as handling resource lifecycle (with-open, etc.)? Not yet, but it is planned, in something called scopes. http://clojure.org/todo -SS --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
