> How much of your physical RAM is dedicatd to the JVM? > > I forgot to say that you probably should consider lowering it > significantly (to be continued, getting off the subway...).
So, it occurred to be you reported a 16 GB maximum heap size. If that is a substantial portion of your total physical RAM, that would probably be the reason why you see swapping. The operating system will tend to use some heuristics to figure out what to keep in memory, and whether to evict pages from cache or actually swap out applications. The greater the memory pressure the greater the chance that the operating system will start swapping you out. Swappiness 0 won't help in this case. (For some reason it never occurred to me before, but the JVM should, if it doesn't already, provide a command line argument to make it use mlock()/mlockall() to lock data in memory...) Anyways: The solution in this case, if your 16 GB heap size is a significant portion of system memory, is probably not to force the OS not to swap, but rather lowering the JVM heap size. Consider that all memory used by the JVM will be used for its heap, instead of caching disk I/O. I am spekulating now on what your settings are, but suppose: (1) Your *actual* need for JVM heap size is roughly 1 GB (I picked something random and low). (2) Your cassandra is configured to not keep extreme amounts of data cached in-JVM (thus causing the real need to be roughly 1 GB, among other things). (3) Your maximum heap size is 16 GB. (3) means that garbage collection becomes more efficient, because you have lots more memory than you need, if looked at in isolation. But if you are wasting 15 GB of would-be disk caching on maintaining the oversize JVM heap, you will instead completely kill performance. Probably there are two extremes, and a spectrum in between: (1) Tune cassandra and the JVM to gobble up a *lot* of memory and cache data in the JVM/cassandra. (2) Tune cassandra and the JVM to gobble up an appropriate amount of memory, and leave as much as possible for the operating system cache mechanism. (2) is likely to be a lot more performant than (1), assuming it works and you don't swap. If you do have a significantly oversized heap, one potential resulting behavior is overall poor performance due to lack of caching, and periodically much worse performance in relation to swap storms during GC:s. -- / Peter Schuller
