I am having major issues with stability on my cassandra nodes. Here is the setup: Cassandra Cluster - 2 EC2 small instances (1.7G RAM, single 32 bit core) with an EBS for the cassandra sstables Cassandra 0.6.0 w/ 1G heap space and 128M/1mil Memtable Thresholds Clients are also small EC2 webservers running php 5.3.2 and using Thrift PHP to access the cluster
I am trying to migrate data from mysql into the cluster using the following methodology: 1. get 500 rows (12 columns each) from mysql 2. build a batch_mutate to insert these rows into one CF (1 row = 1 row ) 3. build a second batch_mutate to insert an index of those rows into a second CF ( 1 row = 1 column ) 4. loop around and do it again until all data has migrated. This process is running on two clients each working on a separate part of the mysql data which totals to about 70G. Each time I start it up, it will work fine for about 1 hour and then it will crash the servers. The error message on the servers is usually an out of memory error. I will get several time out errors on the clients and occasionally get an error telling me that i was missing the timestamp. The timestamp error is accompanied by a server crashing if I use framed transport instead of buffered. I wasn't having the out of memory errors with 0.5.0, but had lots of timeouts and some "unknown result" errors. So we upgraded to 0.6.0 when it became the stable release. I have a development VM which has all the same code on one machine and have had stability issues there as well. I upgraded to the newest build of Java (b20) on the VM and it hasn't helped. If anything, it is more unstable. I am at the point of abandoning cassandra as a solution for us, but wanted to see if you all could offer me some advice first. One of the reasons we were trying cassandra was to scale out with smaller nodes rather than having to run larger instances for mysql. Lee Parker
