I have a three node cluster that has been sitting at a load of 4 (for each
node), 100% CPI utilization (although 92% nice) for that last 12 hours,
ever since some significant writes finished. I'm trying to determine what
tuning I should be doing to get it out of this state. The debug log is just
an endless series of:
DEBUG [ScheduledTasks:1] 2014-12-16 19:03:35,042 GCInspector.java (line
118) GC for ParNew: 166 ms for 10 collections, 4400928736 used; max is
8000634880
DEBUG [ScheduledTasks:1] 2014-12-16 19:03:36,043 GCInspector.java (line
118) GC for ParNew: 165 ms for 10 collections, 4440011176 used; max is
8000634880
DEBUG [ScheduledTasks:1] 2014-12-16 19:03:37,043 GCInspector.java (line
118) GC for ParNew: 135 ms for 8 collections, 4402220568 used; max is
8000634880
iostat shows virtually no I/O.
Compaction may enter into this, but i don't really know what to make of
compaction stats since they never change:
[root@cassandra-37919c3a ~]# nodetool compactionstats
pending tasks: 10
compaction type keyspace table completed
total unit progress
Compaction mediamedia_tracks_raw 271651482
563615497 bytes 48.20%
Compaction mediamedia_tracks_raw 30308910
21676695677 bytes 0.14%
Compaction mediamedia_tracks_raw 1198384080
1815603161 bytes 66.00%
Active compaction remaining time : 0h22m24s
5 minutes later:
[root@cassandra-37919c3a ~]# nodetool compactionstats
pending tasks: 9
compaction type keyspace table completed
total unit progress
Compaction mediamedia_tracks_raw 271651482
563615497 bytes 48.20%
Compaction mediamedia_tracks_raw 30308910
21676695677 bytes 0.14%
Compaction mediamedia_tracks_raw 1198384080
1815603161 bytes 66.00%
Active compaction remaining time : 0h22m24s
Sure the pending tasks went down by one, but the rest is identical.
media_tracks_raw likely has a bunch of tombstones (can't figure out how to
get stats on that).
Is this behavior something that indicates that i need more Heap, larger new
generation? Should I be manually running compaction on tables with lots of
tombstones?
Any suggestions or places to educate myself better on performance tuning
would be appreciated.
arne
