Just an update here. We're now entirely on standard IO mode, and everything is stable and happy. There hasn't been much of a performance hit, if at all.
- James On Mon, Jun 21, 2010 at 3:30 AM, Peter Schuller <peter.schul...@infidyne.com > wrote: > > How much of your physical RAM is dedicatd to the JVM? > > > > I forgot to say that you probably should consider lowering it > > significantly (to be continued, getting off the subway...). > > So, it occurred to be you reported a 16 GB maximum heap size. If that > is a substantial portion of your total physical RAM, that would > probably be the reason why you see swapping. The operating system will > tend to use some heuristics to figure out what to keep in memory, and > whether to evict pages from cache or actually swap out applications. > The greater the memory pressure the greater the chance that the > operating system will start swapping you out. Swappiness 0 won't help > in this case. > > (For some reason it never occurred to me before, but the JVM should, > if it doesn't already, provide a command line argument to make it use > mlock()/mlockall() to lock data in memory...) > > Anyways: The solution in this case, if your 16 GB heap size is a > significant portion of system memory, is probably not to force the OS > not to swap, but rather lowering the JVM heap size. Consider that all > memory used by the JVM will be used for its heap, instead of caching > disk I/O. I am spekulating now on what your settings are, but suppose: > > (1) Your *actual* need for JVM heap size is roughly 1 GB (I picked > something random and low). > (2) Your cassandra is configured to not keep extreme amounts of data > cached in-JVM (thus causing the real need to be roughly 1 GB, among > other things). > (3) Your maximum heap size is 16 GB. > > (3) means that garbage collection becomes more efficient, because you > have lots more memory than you need, if looked at in isolation. But if > you are wasting 15 GB of would-be disk caching on maintaining the > oversize JVM heap, you will instead completely kill performance. > > Probably there are two extremes, and a spectrum in between: > > (1) Tune cassandra and the JVM to gobble up a *lot* of memory and > cache data in the JVM/cassandra. > (2) Tune cassandra and the JVM to gobble up an appropriate amount of > memory, and leave as much as possible for the operating system cache > mechanism. > > (2) is likely to be a lot more performant than (1), assuming it works > and you don't swap. > > If you do have a significantly oversized heap, one potential resulting > behavior is overall poor performance due to lack of caching, and > periodically much worse performance in relation to swap storms during > GC:s. > > -- > / Peter Schuller >
