Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-21 Thread S Ahmed
curious how did things turn out? On Tue, May 18, 2010 at 1:38 PM, Curt Bererton c...@zipzapplay.com wrote: We only have a few CFs (6 or 7). I've increased the MemtableThroughputInMB and MemtableOperationsInMillions as per your suggestions. Do we really need a swap file though? I suppose it

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-21 Thread Curt Bererton
We can get Cassandra to run great for a few hours now. Writing to and reading from cassandra work well and the read/write times are good etc. We also changed our config to enable row caching (we're hoping to ditch our memcache server layer entirely). Unfortunately, running on an EC2 High Memory

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-18 Thread Curt Bererton
We only have a few CFs (6 or 7). I've increased the MemtableThroughputInMB and MemtableOperationsInMillions as per your suggestions. Do we really need a swap file though? I suppose it can't hurt, but with my problem in particular we weren't maxing out main memory. We'll be running another test

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Mark Greene
Can you provide us with the current JVM args? Also, what type of work load you are giving the ring (op/s)? On Mon, May 17, 2010 at 6:39 PM, Curt Bererton c...@zipzapplay.com wrote: Hello Cassandra users+experts, Hopefully someone will be able to point me in the correct direction. We have

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Brandon Williams
On Mon, May 17, 2010 at 6:02 PM, Curt Bererton c...@zipzapplay.com wrote: So pretty much the defaults aside from the 7Gig max heap. CPU is totally hammered right now, and it is receiving 0 ops/sec from me since I disconnected it from our application right now until I can figure out what's

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Lee Parker
What are your storage-conf settings for Memtable thresholds? One thing that could cause lots of CPU usage is dumping the memtables too frequently and then having to do lots of compaction. With that much available heap space you could definitely go larger than the default thresholds. Also, do

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Lee Parker
Also, I am using batch_mutate for all of my writes. Lee Parker On Mon, May 17, 2010 at 7:11 PM, Lee Parker l...@socialagency.com wrote: What are your storage-conf settings for Memtable thresholds? One thing that could cause lots of CPU usage is dumping the memtables too frequently and then

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Curt Bererton
Thanks for the help guys: First answering the first question: both cores are pegged: Cpu0 : 43.8%us, 34.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 22.1%st Cpu1 : 40.5%us, 36.2%sy, 0.0%ni, 0.4%id, 0.0%wa, 0.0%hi, 0.2%si, 22.6%st Mem: 7872040k total, 3620180k used, 4251860k

Re: Problems running Cassandra 0.6.1 on large EC2 instances.

2010-05-17 Thread Curt Bererton
Agreed, and I just saw that in storage conf that a higher value for the MemtableFlushAfterMinutes is suggested otherwise you might get a flush storm: of all your memtables flushing at once. I've changed that as well. -- Curt, ZipZapPlay Inc., www.PlayCrafter.com,