> The other problem is: if I keep mixed write and read (e.g, 8 > write threads > plus 7 read threads) against the 2-nodes cluster > continuously, the read > latency will go up gradually (along with the size of > Cassandra data file), > and at the end it will become ~40ms (up from ~20ms) even with only 15 > threads. During this process the data file grew from 1.6GB to > over 3GB even > if I kept writing the same key/values to Cassandra. It seems > that Cassandra > keeps appending to sstable data files and will only clean up > them during > node cleanup or compact (please correct me if this is incorrect).
In my tests I have observed that good read latency depends on keeping the number of data files low. In my current test setup, I have stored 1.9 TB of data on a single node, which is in 21 data files, and read latency is between 10 and 60ms (for small reads, larger read of course take more time). In earlier stages of my test, I had up to 5000 data files, and read performance was quite bad: my configured 10-second RPC timeout was regularly encountered. The number of data files is reduced whenever Cassandra compacts them, which is either automatically, when enough datafiles are generated by continuous writing, or when triggered by nodeprobe compact, cleanup etc. So my advice is to keep the write throughput low enough so that Cassandra can keep up compacting the data files. For high write throughput, you need fast drives, if possible on different RAIDs, which are configured as different DataDirectories for Cassandra. On my setup (6 drives in a single RAID-5 configuration), compaction is quite slow: sequential reads/writes are done at 150 MB/s, whereas during compaction, read/write-performance drops to a few MB/s. You definitively want more than one logical drive, so that Cassandra can alternate between them when flushin memtables and when compacting. I would really be interested whether my observations are shared by other people on this list. Thanks! Martin
