Thanks, but those sadly don't help me. I tried your configuration but did not change of fix the behaviour I'm seeing.
I investigated a bit further and figured out that after a full garbage collection I seem to get a problem with the scala Stream API. Using jmap -histo I can see a very quickly growing count of scala.collection.immutable.Stream$Cons, Stream$$anonfun$takeWhile$1 and Stream$$anonfun$continually$1. Cons grows with a rate of about 50,000/s and the other two with half of that. At about 3,500,000 the GC kicks in again and it begins anew. During normal operation I get a growth rate of maybe 2,000/s and never go above about 75,000 instances total. I found a scala bug / pull request about Stream.continually / Stream.cons that might point to the problem I'm experiencing: https://github.com/scala/scala/pull/3450 I don't really now scala though, so maybe there is an alternative to using Stream.continually that does not suddenly leak memory? Has anyone ever experienced this and knows how to fix it? Thanks, Florian On Fri, Jan 31, 2014 at 5:48 PM, Jun Rao <[email protected]> wrote: > You can take a look at the GC setting described in > https://cwiki.apache.org/confluence/display/KAFKA/Operations > > Thanks, > > Jun > > > On Fri, Jan 31, 2014 at 2:26 AM, Florian Ollech <[email protected] > >wrote: > > > Hi, > > > > I am currently trying to setup Kafka 0.8 and ran into a problem with the > > producer. Every time the server triggers a garbage collection, because > > permgen space is full, the server descends into a very unusual memory > state > > where a lot of garbage collection is suddenly happening (~5-10% gc cpu > > time, when normally it is 0-0.5%). Additionally a lot of objects suddenly > > get promoted from eden space to survivor space, old gen and perm gen. > This > > obviously slows down the whole server and turns up the system load by a > > lot. > > This behaviour only appears when the producer is active during a perm gen > > gc and can be remedied by reloading the producer(s). Since reloading the > > producer(s) every 5 seconds really is not how it is supposed to work, I > am > > wondering if any one has any pointers as to where the problem may be? > > > > My system has the following specs: > > - oracle jdk 1.6.0_22-b04 > > - sending about 12000 messages per second, about 60 bytes each > > - producer counts tried: 1, 10, 25, 50 > > - async producer > > - ack makes no difference > > - batch sizes tried: 100 - 5000 > > > > I made screenshot of the behaviour: http://i.imgur.com/xtlVNaK.png > > at ~10:24:30 the full gc happens, surivor space fills up and old gen > > becomes a sawtooth. > > at ~10:34 I reloaded the producers and everything was fine again. > > > > Cheers, > > Flo > > > -- Florian Ollech Software Engineer - Yieldlab GmbH Mail: [email protected] Web: http://www.yieldlab.de Geschäftsführer: Marco Klimkeit Sitz: Hamburg HRB 115082
