Hi! We have unfortunately managed to put ourselves in a situation where we are really close to full disks on our existing 27 nodes.
We are now trying to add 15 more nodes, but running into problems with out of disk space on the new nodes while joining. We're using vnodes, on Cassandra 1.2.18 (yes, I know that's old, and I'll upgrade as soon as I'm out of this problematic situation). I've added all the 15 nodes, with some time inbetween - definitely more than the 2-minute rule. But it seems like compaction is not keeping up with the incoming data. Or at least that's my theory. What are the recommended settings to avoid this problem? I have now set compaction threshold to 0 for unlimited compaction bandwidth, hoping that will help (will it?) Will it help to lower the streaming throughput too? I'm unsure about the latter since from observation it seems that compaction will not start until it has finished streaming from a node. With 27 nodes sharing the incoming bandwidth, all of them will take equally long time to finish and then the compaction can occur. I guess I could limit streaming bandwidth on some of the source nodes too. Or am I completely wrong here? Other ideas most welcome. Regards, \EF
