single node capacity
Hi, How much data load can a single typical cassandra instance handle? It seems like we are getting into trouble when one of our node's load grows to bigger than 200g. Both read latency and write latency are increasing, varying from 10 to several thousand milliseconds. machine config is 16*cpu 32G RAM Heap size is 10G Any suggestion of tuning? Or should I start considering adding more nodes when the data grows to this big? Thanks
Best way of adding new nodes
Hi, guys The 2 ways of adding new nodes, when add with bootstrapping, since we've already got lots of data, often it will take many hours to complete the bootstrapping and probably affect the performance of existing nodes. But if we add without bootstrapping, the data load on the new node could be quite unbalanced. Is there a better way of adding a new node with less cost while more balanced? Maybe a steady approach at a cost of longer time while not affecting current nodes much. Thanks
Skipping corrupted rows when doing compaction
Hi, Is there a way to skip corrupted rows when doing compaction? We are currently deploying 2 nodes with replicationfactor=2 but one node reports lots of exceptions like java.io.UTFDataFormatException: malformed input around byte 72. My guess is that some of the data in the SSTable is corrupted but not all because I can still read data out of the related CF but for some keys. It's OK for us to throw away a small portion of the data to get the nodes working normal. If there is no such way to skip corrupted rows can I just clean all the data in the corrupted node and then add it back to the cluster? Will it automatically migrating data from the other node? Thanks. Ivan
Re: Skipping corrupted rows when doing compaction
Thanks, Jonathan I'm using 0.6.1 And another thing is that I get lots of zero-sized tmp files in the data directory. When I restarted cassandra those tmp files will be deleted then new empty tmp files will be generated gradually, while still lots of UTFDataFormatException in the system.log Using 0.6.2 and DiskAccessMode=standard will skip corrupted rows? On Tue, Jun 1, 2010 at 9:08 PM, Jonathan Ellis jbel...@gmail.com wrote: If you're on a version earlier than 0.6.1, you might be running into https://issues.apache.org/jira/browse/CASSANDRA-866. Upgrading will fix it, you don't need to reload data. It's also worth trying 0.6.2 and DiskAccessMode=standard, in case you've found another similar bug. On Tue, Jun 1, 2010 at 7:37 AM, hive13 Wong hiv...@gmail.com wrote: Hi, Is there a way to skip corrupted rows when doing compaction? We are currently deploying 2 nodes with replicationfactor=2 but one node reports lots of exceptions like java.io.UTFDataFormatException: malformed input around byte 72. My guess is that some of the data in the SSTable is corrupted but not all because I can still read data out of the related CF but for some keys. It's OK for us to throw away a small portion of the data to get the nodes working normal. If there is no such way to skip corrupted rows can I just clean all the data in the corrupted node and then add it back to the cluster? Will it automatically migrating data from the other node? Thanks. Ivan -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of Riptano, the source for professional Cassandra support http://riptano.com
