> very quickly from the young generation to the old generation". Furthermore, > the CMSInitiatingOccupancyFraction of 75 (from a JVM default of 68) means > "start gc in the old generation later", presumably to allow Cassandra to use > more of the old generation heap without needlessly trying to free up used > space (?). Please correct me if I am misinterpreting these settings.
Note the use of -XX:+UseCMSInitiatingOccupancyOnly which causes the JVM to always trigger on that occupancy fraction rather than only do it for the first trigger (or something along those lines) and then switch to heuristics. Presumably (though I don't specifically know the history of this particular option being added) it is more important to avoid doing Full GC:s at all than super-optimally tweaking the trigger for maximum throughput. The heuristics tend to cut it pretty close, and setting a conservative fixed occupancy trigger probably greatly lessens the chance of falling back to a full gc in production. > One of the issues I have been having is extreme node instability when > running a major compaction. After 20-30 seconds of operation, the node > spends 30+ seconds in (what I believe to be) GC. Now I have tried halving > all memtable thresholds to reduce overall heap memory usage but that has not > seemed to help with the instability. After one of these blips, I often see > log entries as follows: > INFO [ScheduledTasks:1] 2011-01-17 10:41:21,961 GCInspector.java (line 133) > GC for ParNew: 215 ms, 45084168 reclaimed leaving 11068700368 used; max is > 12783583232 > INFO [ScheduledTasks:1] 2011-01-17 10:41:28,033 GCInspector.java (line 133) > GC for ParNew: 234 ms, 40401120 reclaimed leaving 12144504848 used; max is > 12783583232 > INFO [ScheduledTasks:1] 2011-01-17 10:42:15,911 GCInspector.java (line 133) > GC for ConcurrentMarkSweep: 45828 ms, 3350764696 reclaimed leaving > 9224048472 used; max is 12783583232 45 seconds is pretty significant even for a 12 gig heap unless you're really CPU loaded so that there is heavy contention over the CPU. While I don't see anything obviously extreme; are you sure you're not swapping a bit? Also, what do you mean by node instability - does it *completely* stop responding during these periods or does it flap in and out of the cluster but is still responding? Are you nodes disk bound or CPU bound during compaction? -- / Peter Schuller
