Hi, we are currently in the process of migrating from C* 3.11 to C* 4.1 and we want to start using incremental repairs after the upgrade has been completed. It seems like all the really bad bugs that made using incremental repairs dangerous in C* 3.x have been fixed in 4.x, and for our specific workload, incremental repairs should offer a significant performance improvement.
Therefore, I am currently devising a plan how we could migrate to using incremental repairs. I am aware of the guide from DataStax (https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/operations/opsRepairNodesMigration.html), but this guide is quite old and was written with C* 3.0 in mind, so I am not sure whether this still fully applies to C* 4.x. In addition to that, I am not sure whether this approach fits our workload. In particular, I am wary about disabling autocompaction for an extended period of time (if you are interested in the reasons why, they are at the end of this e-mail). Therefore, I am wondering whether a slighly different process might work better for us: 1. Run a full repair (we periodically run those anyway). 2. Mark all SSTables as repaired, even though they will include data that has not been repaired yet because it was added while the repair process was running. 3. Run another full repair. 4. Start using incremental repairs (and the occasional full repair in order to handle bit rot etc.). If I understood the interactions between full repairs and incremental repairs correctly, step 3 should repair potential inconsistencies in the SSTables that were marked as repaired in step 2 while avoiding the problem of overstreaming that would happen when only marking those SSTables as repaired that already existed before step 1. Does anyone see a flaw in this concept or has experience with a similar scenario (migrating to incremental repairs in an environment with high-density nodes, where a single table contains most of the data)? I am also interested in hearing about potential problems other C* users experienced when migrating to incremental repairs, so that we get a better idea what to expect. Thanks, Sebastian Here is the explanation why I am being cautious: More than 95 percent of our data is stored in a single table, and we use high density nodes (storing about 3 TB of data per node). This means that a full repair for the whole cluster takes about a week. The reason for this layout is that most of our data is “cold”, meaning that it is written once, never updated, and rarely deleted or read. However, new data is added continuously, so disabling autocompaction for the duration of a full repair would lead to a high number of small SSTables accumulating over the course of the week, and I am not sure how well the cluster would handle such a situation (and the increased load when autocompaction is enabled again).
smime.p7s
Description: S/MIME cryptographic signature
