We're troubleshooting a memory usage problem during batch reads. We've spent the last few days profiling and trying different GC settings. The symptoms are that after a certain amount of time during reads one or more nodes in the cluster will exhibit extreme memory pressure followed by a gc storm. We've tried every possible JVM setting and different GC methods and the issue persists. This is pointing towards something instantiating a lot of objects and keeping references so that they can't be cleaned up.
Typically nothing is ever logged other than the GC failures however just now one of the nodes emitted logs we've never seen before: INFO [ScheduledTasks:1] 2011-05-06 15:04:55,085 StorageService.java (line 2218) Unable to reduce heap usage since there are no dirty column families We have tried increasing the heap on these nodes to large values, eg 24GB and still run into the same issue. We're running 8GB of heap normally and only one or two nodes will ever exhibit this issue, randomly. We don't use key/row caching and our memtable sizing is 64mb/0.3. Larger or smaller memtables make no difference in avoiding the issue. We're on 0.7.5, mmap, jna and jdk 1.6.0_24 We've somewhat hit the wall in troubleshooting and any advice is greatly appreciated. -- Adam
