Re: Closures eat permgen?

2010-11-18 Thread Ken Wesson
I did a bit more torture testing of clojure 1.2 and those JVM settings this time for speed. user=> (time (domany 10 (repeatedly gensym))) "Elapsed time: 1362.33344 msecs" {:foo clojure.lang.Symbol} user=> (time (pmap #(assoc % :bar 1) (take 10 (repeatedly #(domany 1 (repeatedly gensym

Re: Closures eat permgen?

2010-11-18 Thread Ken Wesson
Even with Matt's suggested GC settings, keyword creation still leaks permgen: (def kwseq (map (comp keyword str) (iterate inc 0))) #'user/kwseq user=> (domany 1000 kwseq) #<# To top it off, the Java process continues to use 20-30% CPU when supposedly idle after this message, and typing (keywo

Re: Closures eat permgen?

2010-11-18 Thread Matt Fowles
Ken~ The CMS settings are best for low latency applications or applications that want more predictable pause times. They can cause a drop in throughput for scientific computing or for Extract-Transform-Load style programs. CMS is not the default for historical reasons. The permgen sweeping is n

Re: Closures eat permgen?

2010-11-18 Thread Ken Wesson
With those settings, it does not OOME even with 'eval'. It looks like those settings allow infinite sequential function creation without leaks. I did not closely evaluate GC performance. However I must suspect there to be some reasons why those settings aren't the default. I wouldn't mind knowing

Re: Closures eat permgen?

2010-11-17 Thread Matt Fowles
Ken~ CMS (Concurrent Mark Sweep) is part of a multi-stage generational GC. It is the newest GC in a released version of the JVM (the G1 GC not having been released yet). With the below settings, the young gen is divided into Eden and two survivor spaces. The survivor spaces act as generations f

Re: Closures eat permgen?

2010-11-17 Thread Ken Wesson
On Wed, Nov 17, 2010 at 10:12 PM, Matt Fowles wrote: > Ken~ > Not sure what jvm args you are running with, but not all GC settings will > sweep or clear the permgen.  You should try it with: > -XX:+CMSClassUnloadingEnabled > -XX:+CMSPermGenSweepingEnabled > -XX:+UseParNewGC > -XX:+UseConcMarkSweep

Re: Closures eat permgen?

2010-11-17 Thread Matt Fowles
Ken~ Not sure what jvm args you are running with, but not all GC settings will sweep or clear the permgen. You should try it with: -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled Matt PS - http://blogs.sun.com/

Re: Closures eat permgen?

2010-11-17 Thread Ken Wesson
I ran some tests: (defn domany [n s] (reduce (fn [a b] (assoc a :foo (.getClass b))) {} (take n s))) (def fnseq (iterate (fn [_] (fn [x] (+ 2 x))) 0)) (domany 1000 fnseq) With these, the last operation grinds away for a long time (a lot more than 10x what it takes with only 100 iteratio

Re: Closures eat permgen?

2010-11-15 Thread Alyssa Kwan
I totally misunderstood the role of the EVAL context flag in the compile method of ObjExpr. Is there a general writeup anywhere of how the compiler works, especially the interaction of parse and emit? On Nov 15, 4:59 pm, Ken Wesson wrote: > On Mon, Nov 15, 2010 at 4:24 PM, Alan wrote: > > On No

Re: Closures eat permgen?

2010-11-15 Thread Ken Wesson
On Mon, Nov 15, 2010 at 4:24 PM, Alan wrote: > On Nov 15, 12:12 pm, Alyssa Kwan wrote: >> In your example, my-generator isn't the concern.  It's the call to my- >> generator that creates functions, each of which creates bytecode, is >> loaded as a class, then is instantiated, and finally invoked.

Re: Closures eat permgen?

2010-11-15 Thread Alan
On Nov 15, 12:12 pm, Alyssa Kwan wrote: > In your example, my-generator isn't the concern.  It's the call to my- > generator that creates functions, each of which creates bytecode, is > loaded as a class, then is instantiated, and finally invoked. Not true. Compiling my-generator creates two clas

Re: Closures eat permgen?

2010-11-15 Thread Alyssa Kwan
If you look at the bytecode for the closures, you'll see that the Var that *ns*/a points to is resolved at time, and the Java reference is stored as a static final class member. That's a small use of additional permgen. In your example, my-generator isn't the concern. It's the call to my- gener

Closures eat permgen?

2010-11-15 Thread Ken Wesson
In another thread, someone just indicated that a closed-over variable chews up permgen. Is this true? I had been under the impression that keywords didn't get garbage-collected, so runtime generation with (keyword foo) ought to be used sparingly, but that was it. Perhaps the scenario was somethin