Clojure lazy-seq over Java iterative code
I also posted this to StackOverflow, so sorry if you saw it there too. If you want some rep points over there you can answer there too ( http://stackoverflow.com/questions/12427518/clojure-lazy-seq-over-java-iterative-code ). I'm trying to use create a Clojure seq from some iterative Java library code that I inherited. Basically what the Java code does is read records from a file using a parser, sends those records to a processor and returns an ArrayList of result. In Java this is done by calling parser.readData(), then parser.getRecord() to get a record then passing that record into processor.processRecord(). Each call to parser.readData() returns a single record or null if there are no more records. Pretty common pattern in Java. So I created this next-record function in Clojure that will get the next record from a parser. (defn next-record Get the next record from the parser and process it. [parser processor] (let [datamap (.readData parser) row (.getRecord parser datamap)] (if (nil? row) nil (.processRecord processor row 100 The idea then is to call this function and accumulate the records into a Clojure seq (preferably a lazy seq). So here is my first attempt which works great as long as there aren't too many records: (defn datamap-seq Returns a lazy seq of the records using the given parser and processor [parser processor] (lazy-seq (when-let [records (next-record parser processor)] (cons records (datamap-seq parser processor) I can create a parser and processor, and do something like (take 5 (datamap-seq parser processor)) which gives me a lazy seq. And as expected getting the (first) of that seq only realizes one element, doing count realizes all of them, etc. Just the behavior I would expect from a lazy seq. Of course when there are a lot of records I end up with a StackOverflowException. So my next attempt was to use loop-recur to do the same thing. (defn datamap-seq Returns a lazy seq of the records using the given parser and processor [parser processor] (lazy-seq (loop [records (seq '())] (if-let [record (next-record parser processor)] (recur (cons record records)) records Now using this the same way and defing it using (def results (datamap-seq parser processor)) gives me a lazy seq and doesn't realize any elements. However, as soon as I do anything else like (first results) it forces the realization of the entire seq. Can anyone help me understand where I'm going wrong in the second function using loop-recur that causes it to realize the entire thing? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure lazy-seq over Java iterative code
On Fri, Sep 14, 2012 at 8:37 AM, Dave Kincaid kincaid.d...@gmail.com wrote: I also posted this to StackOverflow, so sorry if you saw it there too. If you want some rep points over there you can answer there too (http://stackoverflow.com/questions/12427518/clojure-lazy-seq-over-java-iterative-code). Looks like there's already a good answer over there. Of course when there are a lot of records I end up with a StackOverflowException. Did you dig into where the StackOverflowException originated from? As noted in a comment on SO, the lazy-seq can handle millions of items without a problem so the problem isn't inherently with that part of your code. Perhaps it's the parser or the processor? -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Clojure lazy-seq over Java iterative code
Thanks, Sean. It turns out the exception is being thrown from one of the Java classes (I should have looked closer). I just assumed that doing the recursion without loop/recur would always blow the stack eventually. I'll have to do some research to understand when it does and when it doesn't. I also pasted in the wrong original function that I was using when I was getting the exception. The code I pasted does work. The version that's making the Java code throw the exception has an extra (remove) in it to remove empty lists. It looks like this: (defn datamap-seq Returns a lazy seq of the records using the given parser and processor [parser processor] (lazy-seq (when-let [records (next-record parser processor)] (cons records (remove empty? (datamap-seq parser processor)) If I take out that remove and do it elsewhere it works fine. Still puzzling over that one. On Friday, September 14, 2012 11:48:23 AM UTC-5, Sean Corfield wrote: On Fri, Sep 14, 2012 at 8:37 AM, Dave Kincaid kincai...@gmail.comjavascript: wrote: I also posted this to StackOverflow, so sorry if you saw it there too. If you want some rep points over there you can answer there too ( http://stackoverflow.com/questions/12427518/clojure-lazy-seq-over-java-iterative-code). Looks like there's already a good answer over there. Of course when there are a lot of records I end up with a StackOverflowException. Did you dig into where the StackOverflowException originated from? As noted in a comment on SO, the lazy-seq can handle millions of items without a problem so the problem isn't inherently with that part of your code. Perhaps it's the parser or the processor? -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
