By the way, you seem to misunderstand some of the workings of GC. In the kind of generational scheme used in virtually all modern algorithms, a collection of the nursery (where allocations are first hatched) will only be forced when the memory assigned to the nursery is exhausted--not the memory of any higher generations.
-Per On Tue, Mar 23, 2010 at 9:49 PM, aria42 <ari...@gmail.com> wrote: > Whoops duh. That was silly, far to early on the west coast. > > On Mar 23, 7:46 am, Per Vognsen <per.vogn...@gmail.com> wrote: >> It doesn't seem very accidental: the namespace binding for 'trees' is >> retaining the head. You probably want to wrap it in a function: >> >> (defn trees [] >> ...) >> >> -Per >> >> On Tue, Mar 23, 2010 at 9:40 PM, aria42 <ari...@gmail.com> wrote: >> > Hi, >> > I was experimenting with some code and I had an largish sequence I >> > did a doseq over. The memory hit the ceiling, which you expect since >> > even though the head isn't retained GC doesn't happen until you hit >> > your memory limit (is there a way to change that). Once it hit the >> > memory limit, 1.2 gigs in this case, the doseq slows to a halt. >> >> > On the other hand if rather than make a long lazy seq, I do it >> > implicitly in the doseq itself (see FAST snippet below), the >> > performance is great. Is there anyway to get good performance using a >> > single lazy seq? Relevant snippets below. >> >> > Thanks, Aria >> >> > (defn lines >> > "get lines from gz file" >> > [#^String path] >> > (-> path >> > java.io.FileInputStream. >> > java.util.zip.GZIPInputStream. >> > java.io.InputStreamReader. >> > java.io.BufferedReader. >> > line-seq)) >> >> > ; the tree-from-str does some processing, but doesn't have state >> > (def trees (for [l (lines "/usr/local/corpora//NANC/003.gz") >> > :when (not (empty? l)) >> > :let [[t _] (tree/tree-from-str l)]] >> > t)) >> >> > ; SLOW: hits memory limit and becomes slow b/c of constant >> > ; GC hits >> > (doseq [t trees] >> > (println (str t))) >> >> > ; FAST: low memory >> > (doseq [l (lines "/usr/local/corpora//NANC/003.gz") >> > :when (not (empty? l)) >> > :let [[t _] (tree/tree-from-str l)]] >> > (println (str t))) >> >> > -- >> > You received this message because you are subscribed to the Google >> > Groups "Clojure" group. >> > To post to this group, send email to clojure@googlegroups.com >> > Note that posts from new members are moderated - please be patient with >> > your first post. >> > To unsubscribe from this group, send email to >> > clojure+unsubscr...@googlegroups.com >> > For more options, visit this group at >> >http://groups.google.com/group/clojure?hl=en >> >> > To unsubscribe from this group, send email to >> > clojure+unsubscribegooglegroups.com or reply to this email with the words >> > "REMOVE ME" as the subject. >> >> > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > > To unsubscribe from this group, send email to > clojure+unsubscribegooglegroups.com or reply to this email with the words > "REMOVE ME" as the subject. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en To unsubscribe from this group, send email to clojure+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
