Lots of possible "issues" with high write load and not sure if it means we
need more nodes or if the nodes aren't tuned correctly.
Were using 4 EC2 xlarge instances to support 4 medium instances. We're
getting about 10k inserts/sec, but after about 10 minutes it goes down to
about 7k/sec which seems to time well with these messages:
WARN [MemoryMeter:1] 2013-05-29 21:04:05,462 Memtable.java (line 222)
setting live ratio to minimum of 1.0 instead of 0.08283890630659889
and
WARN [ScheduledTasks:1] 2013-05-29 21:24:07,732 GCInspector.java (line 142)
Heap is 0.7554059480798656 full. You may need to reduce memtable and/or
cache sizes. Cassandra will now flush up to the two largest memtables to
free up memory. Adjust flush_largest_memtables_at threshold in
cassandra.yaml if you don't want Cassandra to do this automatically
Weve insured were using JNA and have left most defaults as defaults. No row
cache, default key/bloom caches.
Also, we start to get timeouts on the clients after about 15 minutes of
hammering.
Were using the latest JNA and separate ephemeral drives for commit log and
data directories.
Were using 1.2.5.
ALso we see compations backing up:
pending tasks: 6
compaction type keyspace column family completed
total unit progress
Compaction svbks ttevents
691259241 3621003064 bytes 19.09%
Compaction svbks ttevents
135890464 505047776 bytes 26.91%
Compaction svbks ttevents
225229105 2531271538 bytes 8.90%
Compaction svbks ttevents
1312041410 5409348928 bytes 24.26%
Active compaction remaining time : 0h09m38s
OpsCenter says were only using 20% of the disk. The overall compaction's
remaining time is going up :{
Advise welcome/TIA
--Darren
