Seems these guys below have the same problem, and everyone thinks it is `apply` or `concat` to blame. One of the answerer even explains how apply is the issue, but in the reimplemented version of `mapcat` beside throwing out `apply` he does not use `map` either:
http://stackoverflow.com/questions/4290665/does-concat-break-the-laziness-of-line-seq http://stackoverflow.com/questions/16194841/clojure-lazy-sequences-in-math-combinatorics-results-in-outofmemory-oom-error/16270113#16270113 On Sunday, July 14, 2013 2:33:56 AM UTC+1, Daniel Dinnyes wrote: > > Hiya, check this code out guys: > ---- > > (defn point [x y] > (println "x:" x "y:" y) > [x y]) > > (defn gen-data [n m] > (for [i (range n)] > (for [j (range m)] > (point i j)))) > > (def data (apply concat (gen-data 100 100))) > > (nth data 5) > > "The output was the following:" > > "x: 0 y: 0 > x: 0 y: 1 > x: 0 y: 2 > x: 0 y: 3 > x: 0 y: 4 > x: 0 y: 5 > x: 0 y: 6 > x: 0 y: 7 > x: 0 y: 8 > x: 0 y: 9 > x: 0 y: 10 > x: 0 y: 11 > x: 0 y: 12 > x: 0 y: 13 > x: 0 y: 14 > x: 0 y: 15 > x: 0 y: 16 > x: 0 y: 17 > x: 0 y: 18 > x: 0 y: 19 > x: 0 y: 20 > x: 0 y: 21 > x: 0 y: 22 > x: 0 y: 23 > x: 0 y: 24 > x: 0 y: 25 > x: 0 y: 26 > x: 0 y: 27 > x: 0 y: 28 > x: 0 y: 29 > x: 0 y: 30 > x: 0 y: 31 > [0 5]" > > "Seems like other people have similar problems but the issue was > mis-attributed, > as they thought it has to do with `apply` and/or `concat` (read further to > find out why not): > https://groups.google.com/forum/#!topic/clojure/vzhFmpGkWTo > http://clojurian.blogspot.co.uk/2012/11/beware-of-mapcat.html"; > > "First I too was suspicious about `concat` and `apply`, so I wrote a > version of concat which was not using varargs." > > (defn concat2 [coll] > (lazy-seq > (if-let [s (seq coll)] > (if-let [ss (seq (first s))] > (cons (first ss) (concat2 (cons (rest ss) (rest s)))) > (concat2 (rest s))) > nil))) > > (def data (apply concat (gen-data 100 100))) > > (nth data 5) > > "The issue was still there unfortunately, exactly the same printout like > the with the first example" > > "So next i became suspicious of `for`. Maybe it has to do with the way it > is evaluated. So I rewrote `gen-data` using `map`" > > (defn gen-data [n m] > (map (fn [x] > (map (fn [y] (point x y)) > (range m))) > (range n))) > > "Even with `map` the issue was still present. Maybe both `map` and `for` > has the same problem? Let's rewrite `map` then" > > (defn map2 [f coll] > (lazy-seq > (if-let [s (seq coll)] > (cons (f (first s)) > (map2 f (rest s))) > nil))) > > (defn gen-data [n m] > (map2 (fn [x] > (map2 (fn [y] (point x y)) > (range m))) > (range n))) > > (def data (apply concat (gen-data 100 100))) > > (nth data 5) > > "x: 0 y: 0 > x: 0 y: 1 > x: 0 y: 2 > x: 0 y: 3 > x: 0 y: 4 > x: 0 y: 5 > [0 5]" > > "GOTCHA!!! WORKS CORRECTLY!!! BUG FOUND!!!" > > "...seems like both `map` and `for` are affected, possibly because `for` > depends on `map` (just assumption sorry, I was L.A.Z.Y. to check)" > > "Also, the bug was present while testing with both Clojure versions 1.4 > and 1.5.1" > > "Finally, just to emphasize how serious the issue is try the same with > ONLY three levels of nested \"mapcatting\":" > > (defn point3d [x y z] > (println "x:" x "y:" y "z:" z) > [x y z]) > > (defn gen-data3d [n m k] > (mapcat (fn [x] > (mapcat (fn [y] > (mapcat (fn [z] [(point3d x y z)]) > (range k))) > (range m))) > (range n))) > > (def data3d (gen-data3d 100 100 100)) > > (nth data3d 5) > > "Not gonna copy the output here... seems like if the number of nested > level of mapcats is `n` then the total number of elements evaluating > non-lazily in one go is 32^n." > > --- > BTW, the `map2` implementation above is the quite matching > clojure.core/map, except the core version has special handling for > ChunkedSeqs. Most probably the issue comes from there. Can someone look > into this please? > > Thanks, > Daniel > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
