I believe that what you are seeing, with up to 32 more elements being evaluated than is necessary to execute your code, is due precisely to chunked sequences, as returned by (range n), preserved by map, and I believe also for, doseq, and other Clojure functions and macros.
Chunked sequences can cause evaluation of things that you do not need. In the cases where you do want those things evaluated eventually anyway, chunked sequences can save memory and time in producing those results. They can also cause larger memory use than you want, and forcing full laziness back into a chunked sequence can sometimes help, e.g. see the issue description and my proposed pach here: http://dev.clojure.org/jira/browse/MCOMB-2 Andy On Sat, Jul 13, 2013 at 6:52 PM, Daniel Dinnyes <dinny...@gmail.com> wrote: > Seems these guys below have the same problem, and everyone thinks it is > `apply` or `concat` to blame. One of the answerer even explains how apply > is the issue, but in the reimplemented version of `mapcat` beside throwing > out `apply` he does not use `map` either: > > > http://stackoverflow.com/questions/4290665/does-concat-break-the-laziness-of-line-seq > > http://stackoverflow.com/questions/16194841/clojure-lazy-sequences-in-math-combinatorics-results-in-outofmemory-oom-error/16270113#16270113 > > > On Sunday, July 14, 2013 2:33:56 AM UTC+1, Daniel Dinnyes wrote: >> >> Hiya, check this code out guys: >> ---- >> >> (defn point [x y] >> (println "x:" x "y:" y) >> [x y]) >> >> (defn gen-data [n m] >> (for [i (range n)] >> (for [j (range m)] >> (point i j)))) >> >> (def data (apply concat (gen-data 100 100))) >> >> (nth data 5) >> >> "The output was the following:" >> >> "x: 0 y: 0 >> x: 0 y: 1 >> x: 0 y: 2 >> x: 0 y: 3 >> x: 0 y: 4 >> x: 0 y: 5 >> x: 0 y: 6 >> x: 0 y: 7 >> x: 0 y: 8 >> x: 0 y: 9 >> x: 0 y: 10 >> x: 0 y: 11 >> x: 0 y: 12 >> x: 0 y: 13 >> x: 0 y: 14 >> x: 0 y: 15 >> x: 0 y: 16 >> x: 0 y: 17 >> x: 0 y: 18 >> x: 0 y: 19 >> x: 0 y: 20 >> x: 0 y: 21 >> x: 0 y: 22 >> x: 0 y: 23 >> x: 0 y: 24 >> x: 0 y: 25 >> x: 0 y: 26 >> x: 0 y: 27 >> x: 0 y: 28 >> x: 0 y: 29 >> x: 0 y: 30 >> x: 0 y: 31 >> [0 5]" >> >> "Seems like other people have similar problems but the issue was >> mis-attributed, >> as they thought it has to do with `apply` and/or `concat` (read further >> to find out why not): >> https://groups.google.com/**forum/#!topic/clojure/**vzhFmpGkWTo<https://groups.google.com/forum/#!topic/clojure/vzhFmpGkWTo> >> http://clojurian.blogspot.co.**uk/2012/11/beware-of-mapcat.**html<http://clojurian.blogspot.co.uk/2012/11/beware-of-mapcat.html> >> " >> >> "First I too was suspicious about `concat` and `apply`, so I wrote a >> version of concat which was not using varargs." >> >> (defn concat2 [coll] >> (lazy-seq >> (if-let [s (seq coll)] >> (if-let [ss (seq (first s))] >> (cons (first ss) (concat2 (cons (rest ss) (rest s)))) >> (concat2 (rest s))) >> nil))) >> >> (def data (apply concat (gen-data 100 100))) >> >> (nth data 5) >> >> "The issue was still there unfortunately, exactly the same printout like >> the with the first example" >> >> "So next i became suspicious of `for`. Maybe it has to do with the way it >> is evaluated. So I rewrote `gen-data` using `map`" >> >> (defn gen-data [n m] >> (map (fn [x] >> (map (fn [y] (point x y)) >> (range m))) >> (range n))) >> >> "Even with `map` the issue was still present. Maybe both `map` and `for` >> has the same problem? Let's rewrite `map` then" >> >> (defn map2 [f coll] >> (lazy-seq >> (if-let [s (seq coll)] >> (cons (f (first s)) >> (map2 f (rest s))) >> nil))) >> >> (defn gen-data [n m] >> (map2 (fn [x] >> (map2 (fn [y] (point x y)) >> (range m))) >> (range n))) >> >> (def data (apply concat (gen-data 100 100))) >> >> (nth data 5) >> >> "x: 0 y: 0 >> x: 0 y: 1 >> x: 0 y: 2 >> x: 0 y: 3 >> x: 0 y: 4 >> x: 0 y: 5 >> [0 5]" >> >> "GOTCHA!!! WORKS CORRECTLY!!! BUG FOUND!!!" >> >> "...seems like both `map` and `for` are affected, possibly because `for` >> depends on `map` (just assumption sorry, I was L.A.Z.Y. to check)" >> >> "Also, the bug was present while testing with both Clojure versions 1.4 >> and 1.5.1" >> >> "Finally, just to emphasize how serious the issue is try the same with >> ONLY three levels of nested \"mapcatting\":" >> >> (defn point3d [x y z] >> (println "x:" x "y:" y "z:" z) >> [x y z]) >> >> (defn gen-data3d [n m k] >> (mapcat (fn [x] >> (mapcat (fn [y] >> (mapcat (fn [z] [(point3d x y z)]) >> (range k))) >> (range m))) >> (range n))) >> >> (def data3d (gen-data3d 100 100 100)) >> >> (nth data3d 5) >> >> "Not gonna copy the output here... seems like if the number of nested >> level of mapcats is `n` then the total number of elements evaluating >> non-lazily in one go is 32^n." >> >> --- >> BTW, the `map2` implementation above is the quite matching >> clojure.core/map, except the core version has special handling for >> ChunkedSeqs. Most probably the issue comes from there. Can someone look >> into this please? >> >> Thanks, >> Daniel >> >> -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
