Re: Closures eat permgen?
With those settings, it does not OOME even with 'eval'. It looks like those settings allow infinite sequential function creation without leaks. I did not closely evaluate GC performance. However I must suspect there to be some reasons why those settings aren't the default. I wouldn't mind knowing what those reasons are. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
Ken~ The CMS settings are best for low latency applications or applications that want more predictable pause times. They can cause a drop in throughput for scientific computing or for Extract-Transform-Load style programs. CMS is not the default for historical reasons. The permgen sweeping is needed for programs that dynamically load and unload classes. Because this is a somewhat uncommon use case in the world of java, they are not the default. Matt On Thu, Nov 18, 2010 at 3:07 AM, Ken Wesson kwess...@gmail.com wrote: With those settings, it does not OOME even with 'eval'. It looks like those settings allow infinite sequential function creation without leaks. I did not closely evaluate GC performance. However I must suspect there to be some reasons why those settings aren't the default. I wouldn't mind knowing what those reasons are. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
Even with Matt's suggested GC settings, keyword creation still leaks permgen: (def kwseq (map (comp keyword str) (iterate inc 0))) #'user/kwseq user= (domany 1000 kwseq) ##OutOfMemoryError java.lang.OutOfMemoryError: PermGen space To top it off, the Java process continues to use 20-30% CPU when supposedly idle after this message, and typing (keyword zoob) at the REPL makes it hang. The same thing happens with symbols. It doesn't with normal strings. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
I did a bit more torture testing of clojure 1.2 and those JVM settings
this time for speed.
user= (time (domany 10 (repeatedly gensym)))
Elapsed time: 1362.33344 msecs
{:foo
clojure.lang.Symbol}
user= (time (pmap #(assoc % :bar 1) (take 10 (repeatedly #(domany
1 (repeatedly gensym))
Elapsed time: 210.43172 msecs
({:bar 1,
:foo
clojure.lang.Symbol}
{:bar 1,
... yadda yadda yadda
(This is after a couple of prior runs to JIT everything.)
This is interesting. The machine's only dual core; the roughly 6x
speedup therefore tells me that repeated gensymming is not CPU-bound.
On the other hand I would expect no speedup at all if gensymming
wasn't CPU bound because it spent lots of time waiting on a lock on
some counter used to generate the next gensym's numerical part while
avoiding collisions. Ten threads wouldn't be able to generate gensyms
any faster than one thread, in that case, unless there was a long wait
for something other than a global lock in gensym.
I tried testing this because I suspected that runtime use of gensym,
besides leaking permgen, might create a bottleneck at a global lock if
done in a concurrent app; seems that's not the case at least up to a
parallelism factor of 10, for whatever reason.
Changing (take 10 (repeatedly #(...))) to (repeat 10 (...)) results in
a 2x further speedup as well as the gensym operation being done only
10,000 times instead of 100,000. So 90,000 gensyms done in parallel
takes ~100ms, 10,000 would take ~110, and the other ~100ms is consumed
by pmap overhead. Compared with
user= (time (domany 9 (repeatedly gensym)))
Elapsed time: 1164.5704 msecs
that's a nearly 12x speedup from parallelism after adjusting for
pmap's overhead!
The amount of CPU spent on gensym creation per ms can have doubled at
most, so again most of the time is spent waiting on something and
since this is certainly not I/O bound -- no printing occurs until
after the part that's timed has been timed -- it's got to be locks and
synchronization of some sort (unless there's a gratuitous Thread/sleep
buried in clojure somewhere, which seems highly unlikely), but it
can't be a global lock in gensym creation or parallelization wouldn't
produce any speedup to speak of. (If there's a global lock, most of
the time must be spent elsewhere than waiting on that particular lock,
or all 10 threads would spend most of their time queued up on that one
lock and wouldn't get things done any faster than one thread would.)
user= (time (pmap #(assoc % :bar 1) (take 5 (repeatedly #(domany
2 (repeatedly gensym))
Elapsed time: 555.41208 msecs
The speedup is only 2x instead of 6x with half as many parallel
threads. Subtracting the estimated 100ms pmap overhead gives 455 which
makes the speedup closer to 3x (vs. 12x with 10 threads). Quintupling
the thread count from 1 yields a 3x speedup but a further doubling
yields a 4x speedup? That seems strange.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
I ran some tests:
(defn domany [n s] (reduce (fn [a b] (assoc a :foo (.getClass b))) {}
(take n s)))
(def fnseq (iterate (fn [_] (fn [x] (+ 2 x))) 0))
(domany 1000 fnseq)
With these, the last operation grinds away for a long time (a lot more
than 10x what it takes with only 100 iterations) and I suspect the
garbage collector is working hard (the Java process reaches the -Xmx
size and stays there) but it works.
But:
(def fnseq (iterate (eval '(fn [_] (fn [x] (+ 2 x 0))
(domany 1000 fnseq)
spews an OOME after a couple of minutes.
It looks like eval-generated functions, unlike closures, don't get
(completely) garbage collected if all references to them are dropped.
The message didn't actually say it ran out of permgen but it looks
like using eval as a run-time plugin loader or similarly has caveats,
but using ordinary closures a lot does not.
It would be unloading and reloading plugins on demand that would have
the caveat; loading plugins on demand but keeping them for session
lifetime thereafter won't leak permgen but letting plugins become
garbage and reloading them if needed again would leak permgen. It'd
be a tradeoff; I assume the partly-unloaded plugin would take up less
memory than a fully loaded one, but repeated reloads would chew up
more and more.
And loading identical code repeatedly still would leak, as the example
(which has a fixed s-expression as the argument to eval) shows.
Long story short: if you're going to dynamically load code at runtime,
and may want to load, discard, and load more constantly in a
long-running task, you'll need to write an interpreter unless/until
eval and/or Java's classloader mechanism are fixed so that
unreferenced eval-generated objects and/or dynamically-loaded Java
classes in general are fully garbage collectable.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
On Wed, Nov 17, 2010 at 10:12 PM, Matt Fowles matt.fow...@gmail.com wrote: Ken~ Not sure what jvm args you are running with, but not all GC settings will sweep or clear the permgen. You should try it with: -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled Maybe I will. (Use a mark-sweep GC instead of the generational one that all newer JVMs use by default, though? That will really hurt GC performance.) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
Ken~ CMS (Concurrent Mark Sweep) is part of a multi-stage generational GC. It is the newest GC in a released version of the JVM (the G1 GC not having been released yet). With the below settings, the young gen is divided into Eden and two survivor spaces. The survivor spaces act as generations for young objects before they are tenured into the old gen. The concurrent mark sweep is only used for the old generation and is far more than a simple mark-sweep GC. If you are curious I can provide a somewhat more detailed explanation of the different collectors and phases, but I can assure you that these settings are very good defaults for high performance systems. Matt On Wed, Nov 17, 2010 at 10:37 PM, Ken Wesson kwess...@gmail.com wrote: On Wed, Nov 17, 2010 at 10:12 PM, Matt Fowles matt.fow...@gmail.com wrote: Ken~ Not sure what jvm args you are running with, but not all GC settings will sweep or clear the permgen. You should try it with: -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled Maybe I will. (Use a mark-sweep GC instead of the generational one that all newer JVMs use by default, though? That will really hurt GC performance.) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.comclojure%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
If you look at the bytecode for the closures, you'll see that the Var that *ns*/a points to is resolved at clinit time, and the Java reference is stored as a static final class member. That's a small use of additional permgen. In your example, my-generator isn't the concern. It's the call to my- generator that creates functions, each of which creates bytecode, is loaded as a class, then is instantiated, and finally invoked. That temporarily uses permgen. There should be no problem collecting permgen in this trivial example. However, more complex scenarios will cause classes to live far beyond you would expect. If there is a leak or long-lived object, you'd rather it be in the regular heap instead. My point in the other thread was more of a question of using closures at all vs. not using closures. All functions lead to a loaded class, which by definition uses permgen. Unless there's a real win in terms of design or maintainability, you should use regular functions and just pass in your state/identity. The use case in the other thread was a bad one as far as using closures go; I was recommending against that. On Nov 15, 2:52 pm, Ken Wesson kwess...@gmail.com wrote: In another thread, someone just indicated that a closed-over variable chews up permgen. Is this true? I had been under the impression that keywords didn't get garbage-collected, so runtime generation with (keyword foo) ought to be used sparingly, but that was it. Perhaps the scenario was something along the lines of (defn make-generator [] (let [a (atom 0)] (fn [] (swap! a inc) @a))) (def my-generator (make-generator)) user= (my-generator) 1 user= (my-generator) 2 user= In this case, a closed-over atom is referenced by a function that's bound to a global var, my-generator. This won't be garbage-collected so long as my-generator isn't un/redef'd. Even so it shouldn't be permgen, technically speaking; just indirectly referenced by a loaded class and so ineligible for GC by more ordinary criteria. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
On Nov 15, 12:12 pm, Alyssa Kwan alyssa.c.k...@gmail.com wrote: In your example, my-generator isn't the concern. It's the call to my- generator that creates functions, each of which creates bytecode, is loaded as a class, then is instantiated, and finally invoked. Not true. Compiling my-generator creates two classes, which at run time are simply instantiated as needed. Here's an example: (ns permgen.core (:gen-class)) (defn -my-generator [] (let [x (atom 0)] (fn [] (swap! x inc (dotimes [_ 1000] (-my-generator)) $ cake compile ls classes/permgen/* classes/permgen/core.class classes/permgen/core__init.class classes/permgen/core$loading__4410__auto__.class classes/permgen/core$_my_generator.class classes/permgen/core$_my_generator$fn__2474.class -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
On Mon, Nov 15, 2010 at 4:24 PM, Alan a...@malloys.org wrote: On Nov 15, 12:12 pm, Alyssa Kwan alyssa.c.k...@gmail.com wrote: In your example, my-generator isn't the concern. It's the call to my- generator that creates functions, each of which creates bytecode, is loaded as a class, then is instantiated, and finally invoked. Not true. Compiling my-generator creates two classes, which at run time are simply instantiated as needed. I thought so. Now if your code has stuff like: (defn foo [x] (eval `(fn [quux] blah blah blah quux blah blah ~x blah))) then every call to foo generates new classes and loads them at runtime. Part of why eval should be used sparingly. :) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Closures eat permgen?
I totally misunderstood the role of the EVAL context flag in the compile method of ObjExpr. Is there a general writeup anywhere of how the compiler works, especially the interaction of parse and emit? On Nov 15, 4:59 pm, Ken Wesson kwess...@gmail.com wrote: On Mon, Nov 15, 2010 at 4:24 PM, Alan a...@malloys.org wrote: On Nov 15, 12:12 pm, Alyssa Kwan alyssa.c.k...@gmail.com wrote: In your example, my-generator isn't the concern. It's the call to my- generator that creates functions, each of which creates bytecode, is loaded as a class, then is instantiated, and finally invoked. Not true. Compiling my-generator creates two classes, which at run time are simply instantiated as needed. I thought so. Now if your code has stuff like: (defn foo [x] (eval `(fn [quux] blah blah blah quux blah blah ~x blah))) then every call to foo generates new classes and loads them at runtime. Part of why eval should be used sparingly. :) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
