Hey Jeff, one of the nodes with high GC has 1400 SST tables, all other nodes have about 500-900 SST tables. the other node with high GC has 636 SST tables.
the average row size for compacted partitions is about 1640 bytes on all nodes. We have replication factor 3 but the problem is only on two nodes. the only other thing that stands out in cfstats is the read time and write time on the nodes with high GC is 5-7 times higher than other 5 nodes, but i think thats expected. thanks anishek On Wed, Mar 2, 2016 at 1:09 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > Compaction falling behind will likely cause additional work on reads (more > sstables to merge), but I’d be surprised if it manifested in super long GC. > When you say twice as many sstables, how many is that?. > > In cfstats, does anything stand out? Is max row size on those nodes larger > than on other nodes? > > What you don’t show in your JVM options is the new gen size – if you do > have unusually large partitions on those two nodes (especially likely if > you have rf=2 – if you have rf=3, then there’s probably a third node > misbehaving you haven’t found yet), then raising new gen size can help > handle the garbage created by reading large partitions without having to > tolerate the promotion. Estimates for the amount of garbage vary, but it > could be “gigabytes” of garbage on a very wide partition (see > https://issues.apache.org/jira/browse/CASSANDRA-9754 for work in progress > to help mitigate that type of pain). > > - Jeff > > From: Anishek Agarwal > Reply-To: "user@cassandra.apache.org" > Date: Tuesday, March 1, 2016 at 11:12 PM > To: "user@cassandra.apache.org" > Subject: Lot of GC on two nodes out of 7 > > Hello, > > we have a cassandra cluster of 7 nodes, all of them have the same JVM GC > configurations, all our writes / reads use the TokenAware Policy wrapping > a DCAware policy. All nodes are part of same Datacenter. > > We are seeing that two nodes are having high GC collection times. Then > mostly seem to spend time in GC like about 300-600 ms. This also seems to > result in higher CPU utilisation on these machines. Other 5 nodes don't > have this problem. > > There is no additional repair activity going on the cluster, we are not > sure why this is happening. > we checked cfhistograms on the two CF we have in the cluster and number of > reads seems to be almost same. > > we also used cfstats to see the number of ssttables on each node and one > of the nodes with the above problem has twice the number of ssttables than > other nodes. This still doesnot explain why two nodes have high GC > Overheads. our GC config is as below: > > JVM_OPTS="$JVM_OPTS -XX:+UseParNewGC" > > JVM_OPTS="$JVM_OPTS -XX:+UseConcMarkSweepGC" > > JVM_OPTS="$JVM_OPTS -XX:+CMSParallelRemarkEnabled" > > JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8" > > JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=50" > > JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=70" > > JVM_OPTS="$JVM_OPTS -XX:+UseCMSInitiatingOccupancyOnly" > > JVM_OPTS="$JVM_OPTS -XX:+UseTLAB" > > JVM_OPTS="$JVM_OPTS -XX:MaxPermSize=256m" > > JVM_OPTS="$JVM_OPTS -XX:+AggressiveOpts" > > JVM_OPTS="$JVM_OPTS -XX:+UseCompressedOops" > > JVM_OPTS="$JVM_OPTS -XX:+CMSScavengeBeforeRemark" > > JVM_OPTS="$JVM_OPTS -XX:ConcGCThreads=48" > > JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=48" > > JVM_OPTS="$JVM_OPTS -XX:-ExplicitGCInvokesConcurrent" > > JVM_OPTS="$JVM_OPTS -XX:+UnlockDiagnosticVMOptions" > > JVM_OPTS="$JVM_OPTS -XX:+UseGCTaskAffinity" > > JVM_OPTS="$JVM_OPTS -XX:+BindGCTaskThreadsToCPUs" > > # earlier value 131072 = 32768 * 4 > > JVM_OPTS="$JVM_OPTS -XX:ParGCCardsPerStrideChunk=131072" > > JVM_OPTS="$JVM_OPTS -XX:CMSScheduleRemarkEdenSizeThreshold=104857600" > > JVM_OPTS="$JVM_OPTS -XX:CMSRescanMultiple=32768" > > JVM_OPTS="$JVM_OPTS -XX:CMSConcMarkMultiple=32768" > > #new > > JVM_OPTS="$JVM_OPTS -XX:+CMSConcurrentMTEnabled" > > We are using cassandra 2.0.17. If anyone has any suggestion as to how what > else we can look for to understand why this is happening please do reply. > > > > Thanks > anishek > > >
