Re: Instability and memory problems

2010-06-21 Thread James Golick
On Mon, Jun 21, 2010 at 12:24 PM, Peter Schuller < peter.schul...@infidyne.com> wrote: > > Just an update here. We're now entirely on standard IO mode, and > everything > > is stable and happy. There hasn't been much of a performance hit, if at > all. > > Cool. Just be aware that if my speculation

Re: Instability and memory problems

2010-06-21 Thread Peter Schuller
> Just an update here. We're now entirely on standard IO mode, and everything > is stable and happy. There hasn't been much of a performance hit, if at all. Cool. Just be aware that if my speculation was correct that you're (1) dedicating a very large portion of system memory to cassandra, but (2)

Re: Instability and memory problems

2010-06-21 Thread James Golick
Just an update here. We're now entirely on standard IO mode, and everything is stable and happy. There hasn't been much of a performance hit, if at all. - James On Mon, Jun 21, 2010 at 3:30 AM, Peter Schuller wrote: > > How much of your physical RAM is dedicatd to the JVM? > > > > I forgot to s

Re: Instability and memory problems

2010-06-21 Thread Peter Schuller
> How much of your physical RAM is dedicatd to the JVM? > > I forgot to say that you probably should consider lowering it > significantly (to be continued, getting off the subway...). So, it occurred to be you reported a 16 GB maximum heap size. If that is a substantial portion of your total physi

Re: Instability and memory problems

2010-06-21 Thread Peter Schuller
>> (1) Is the machine swapping? (Actively swapping in/out as reported by >> e.g. vmstat) > > Yes, somewhat, although swappiness is set to 0. Ok. While I have no good suggestion to fix it other than moving away from mmap(), given that a low swappiness didn't help, I'd say that as long as you're swa

Re: Instability and memory problems

2010-06-20 Thread James Golick
I opened #1214 about this. I hope people will take a look and provide their feedback. https://issues.apache.org/jira/browse/CASSANDRA-1214 Thanks. On Sun, Jun 20, 2010 at 3:58 PM, James Golick wrote: > uh. wow. I just read up on all this again, and read the code, and I'm a > little surprised,

Re: Instability and memory problems

2010-06-20 Thread James Golick
uh. wow. I just read up on all this again, and read the code, and I'm a little surprised, to be honest. There's no attempt to manage the total size of the mmap()'d IO, and the default buffer allocation is quite sizeable. So, basically, if you have any data, over time, you will run out of memory, a

Re: Instability and memory problems

2010-06-20 Thread James Golick
Thanks for your thoughts. Answers below: On Sun, Jun 20, 2010 at 2:21 PM, Peter Schuller wrote: > > The memory problems I've posted about before have gotten much worse and > our > > nodes are becoming incredibly slow/unusable every 24 hours or so. > Basically, > > the JVM reports that only 14GB

Re: Instability and memory problems

2010-06-20 Thread Peter Schuller
> The memory problems I've posted about before have gotten much worse and our > nodes are becoming incredibly slow/unusable every 24 hours or so. Basically, > the JVM reports that only 14GB is committed, but the RSS of the process is > 22GB, and cassandra is completely unresponsive, but still havin

Re: Instability and memory problems

2010-06-20 Thread Ran Tavory
I don't have the answer but if you provide jmap output, cfstats output that may help. Are you using mmap files? Do you see swap? Gc in the logs? On Jun 20, 2010 7:25 PM, "James Golick" wrote: As I alluded to in another post, we just moved from 2-4 nodes. Since then, the cluster has been incredib

Instability and memory problems

2010-06-20 Thread James Golick
As I alluded to in another post, we just moved from 2-4 nodes. Since then, the cluster has been incredibly The memory problems I've posted about before have gotten much worse and our nodes are becoming incredibly slow/unusable every 24 hours or so. Basically, the JVM reports that only 14GB is comm