Re: Cassandra GC Settings

2011-01-21 Thread Robert Coli
On 1/17/11, Dan Hendry wrote: > Since applying these settings, the one time I saw the same type of behavior > as before, the following appeared in the GC log. > >(concurrent mode failure): 9418431K->6267709K(11841536K), 26.4973750 > secs] 9777393K->6267709K(12290944K), [CMS Perm : 20477K->2044

Re: Cassandra GC Settings

2011-01-17 Thread Peter Schuller
> Now, a full stop of the application was what I was seeing extensively before > (100-200 times over the course of a major compaction as reported by > gossipers on other nodes). I have also just noticed that the previous > instability (ie application stops) correlated with the compaction of a few >

Re: Cassandra GC Settings

2011-01-17 Thread SriSatish Ambati
Thanks, Dan: Yes, -Xmn512MB/1G sizes the Young Generation explicitly and removes the adaptive resizing out of the picture. (If at all possible send your gc log over & we can analyze the promotion failure a little bit more finely.) The low load implies that that you are able to use the parallel thr

Re: Cassandra GC Settings

2011-01-17 Thread Dan Hendry
Thanks for all the info, I think I have been able to sort out my issue. The new settings I am using are: -Xmn512M (Very important I think) -XX:SurvivorRatio=5 (Not very important I think) -XX:MaxTenuringThreshold=5 -XX:ParallelGCThreads=8 -XX:CMSInitiatingOccupancyFraction=75 Since applying these

Re: Cassandra GC Settings

2011-01-17 Thread Jonathan Ellis
On Mon, Jan 17, 2011 at 11:58 AM, Peter Schuller wrote: > 45 seconds is pretty significant even for a 12 gig heap Note that you really need to uncomment the -XX:PrintGC* arguments to get a detailed GC log from the jvm before taking guesses at this; the numbers GCInspector can get are NOT pause ti

Re: Cassandra GC Settings

2011-01-17 Thread Peter Schuller
> very quickly from the young generation to the old generation". Furthermore, > the CMSInitiatingOccupancyFraction of 75 (from a JVM default of 68) means > "start gc in the old generation later", presumably to allow Cassandra to use > more of the old generation heap without needlessly trying to fre

Re: Cassandra GC Settings

2011-01-17 Thread SriSatish Ambati
Dan, Please kindly attach your: 1) java -version 2) full commandline settings, heap sizes. 3) gc log from one of the nodes via: -XX:+PrintTenuringDistribution \ -XX:+PrintGCDetails \ -XX:+PrintGCTimeStamps \ -Xloggc:/var/log/cassandra/gc.log \ 4) number of cores on your system. How busy is the s

Cassandra GC Settings

2011-01-17 Thread Dan Hendry
I am having some reliability problems in my Cassandra cluster which I am almost certain is due to GC. I was about to start delving into the guts of the problem by turning on GC logging but I have never done any serious java GC tuning before (time to learn I guess). As a first step however, I was ho