Re: Garbage collection freezes cassandra node

2012-01-20 Thread Peter Schuller
> Thanks for this very helpful info. It is indeed a production site which I > cannot easily upgrade. I will try the various gc knobs and post any positive > results. *IF* your data size, or at least hot set, is small enough that you're not extremely reliant on the current size of page cache, and

RE: Garbage collection freezes cassandra node

2012-01-20 Thread Rene Kochen
:23 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node > On node "172.16.107.46", I see the following: > > 21:53:27.192+0100: 1335393.834: [GC 1335393.834: [ParNew (promotion failed): > 319468K->324959K(345024K), 0.1304456 secs]13353

Re: Garbage collection freezes cassandra node

2012-01-19 Thread Peter Schuller
> On node "172.16.107.46", I see the following: > > 21:53:27.192+0100: 1335393.834: [GC 1335393.834: [ParNew (promotion failed): > 319468K->324959K(345024K), 0.1304456 secs]1335393.964: [CMS: > 6000844K->3298251K(8005248K), 10.8526193 secs] 6310427K->3298251K(8350272K), > [CMS Perm : 26355K->263

Re: Garbage collection freezes cassandra node

2012-01-19 Thread Mohit Anchlia
tested the "XX:+DisableExplicitGC" switch. > > Is the right thing to do to decrease the CMSInitiatingOccupancyFraction > setting? > > Thanks! > > Rene > > -Original Message- > From: sc...@scode.org [mailto:sc...@scode.org] On Behalf Of Peter Schuller > Sent: d

RE: Garbage collection freezes cassandra node

2012-01-19 Thread Rene Kochen
uot; switch. Is the right thing to do to decrease the CMSInitiatingOccupancyFraction setting? Thanks! Rene -Original Message- From: sc...@scode.org [mailto:sc...@scode.org] On Behalf Of Peter Schuller Sent: dinsdag 20 december 2011 6:38 To: user@cassandra.apache.org Subject: Re: Garb

RE: Garbage collection freezes cassandra node

2011-12-23 Thread Rene Kochen
] On Behalf Of Peter Schuller Sent: dinsdag 20 december 2011 6:36 To: user@cassandra.apache.org Subject: Re: Garbage collection freezes cassandra node > During the garbage collections, Cassandra freezes for about ten seconds. I > observe the following log entries: > > > > “GC for C

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Peter Schuller
I should add: If you are indeed actually pausing due to "promotion failed" or "concurrent mode failure" (which you will see in the GC log if you enable it with the options I suggested), the first thing I would try to mitigate is: * Decrease the occupancy trigger (search for "occupancy") of CMS to

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Peter Schuller
> During the garbage collections, Cassandra freezes for about ten seconds. I > observe the following log entries: > > > > “GC for ConcurrentMarkSweep: 11597 ms for 1 collections, 1887933144 used; max > is 8550678528” Ok, first off: Are you certain that it is actually pausing, or are you assumin

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Mohit Anchlia
Increasing memory in this case may not solve the problem. Share some information about your workload. Cluster configuration, cache sizes etc. You can also try getting java heap historgram to get more info on what's on the heap. On Mon, Dec 19, 2011 at 7:35 AM, Rene Kochen wrote: > I recently se

AW: Garbage collection freezes cassandra node

2011-12-19 Thread Roland Gude
2011 16:35 An: user@cassandra.apache.org Betreff: Garbage collection freezes cassandra node I recently see the following garbage collect behavior in our performance tests (the attached chart shows the heap-size in MB): [cid:image001.jpg@01CCBE73.8B567F80] During the garbage collections, Cassandra

Re: Garbage collection freezes cassandra node

2011-12-19 Thread Paul Loy
Giving it more memory will simply increase the time between (and duration of) the freezes. Which JVM are you using? I think that Java 7 has some nice improvements to GC especially with large heaps. On Mon, Dec 19, 2011 at 7:35 AM, Rene Kochen wrote: > I recently see the following garbage collect