after running my cluster for a while performance has become unacceptable, 200+
ms for reads. if running well, i see reads <10ms. when i run iostat the disk
is being hammered by reads. seems like i/o caching isn't even being used
avg-cpu: %user %nice %system %iowait %steal %idle
2.81 0.00 1.41 13.62 0.00 82.16
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz
await svctm %util
cciss/c0d0p1
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00
cciss/c0d1p1
0.00 0.00 848.50 0.00 13.66 0.00 32.98 21.50
25.23 1.18 100.05
i run top and i see cassandra's memory usage as follows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31510 bburruss 19 0 359g 37g 27g S 48.8 80.1 2137:30 java
i set -Xmx10g so it isn't java using the memory. is it mmap i/o? what would
be causing the huge memory usage? it seems reasonable that the performance is
bad because the i/o cache can't be used properly.
