Alex, you write:
> We can implement CEP-40 using a similar approach: we can leave the source > node as both a read and write target, and allow the new node to be a target > for (pending) writes. Unfortunately, this does not help with availability > (in fact, it decreases write availability, since we will have to collect > 2+1 mandatory write responses instead of just 2), but increases durability, > and I think helps to fully eliminate the second phase. This also increases > read availability when the source node is up, since we can still use the > source node as a part of read quorum. Would it be possible to create a new type of write target node? The new write target node is notified of writes (like any other write node) but does not participate in the write availability calculation. In this way a node this is being migrated to could receive writes and have minimal impact on the current operation of the cluster? Claude On Wed, May 1, 2024 at 12:33 PM Alex Petrov <al...@coffeenco.de> wrote: > Thank you for submitting this CEP! > > Wanted to discuss this point from the description: > > > How to bring up/down Cassandra/Sidecar instances or making/applying > config changes are outside the scope of this document. > > One advantage of doing migration via sidecar is the fact that we can > stream sstables to the target node from the source node while the source > node is down. Also if the source node is down, it does not matter if we > can’t use it as a write target However, if we are replacing a live node, we > do lose both durability and availability during the second copy phase. > There are copious other advantages described by others in the thread above. > > For example, we have three adjacent nodes A,B,C and simple RF 3. C > (source) is up and is being replaced with live-migrated D (destination). > According to the described process in CEP-40, we perform streaming in 2 > phases: first one is a full copy (similar to bootstrap/replacement in > cassandra), and the second one is just a diff. The second phase is still > going to take a non-trivial amount of time, and is likely to last at very > least minutes. During this time, we only have nodes A and B as both read > and write targets, with no alternatives: we have to have both of them > present for any operation, and losing either one of them leaves us with > only one copy of data. > > To contrast this, TCM bootstrap process is 4-step: between the old owner > being phased out and the new owner brought in, we always ensure r/w quorum > consistency and liveness of at least 2 nodes for the read quorum, 3 nodes > available for reads in best case, and 2+1 pending replica for the write > quorum, with 4 nodes (3 existing owners + 1 pending) being available for > writes in best case. Replacement in TCM is implemented similarly, with the > old node remaining an (unavailable) read target, but new node already being > the target for (pending) writes. > > We can implement CEP-40 using a similar approach: we can leave the source > node as both a read and write target, and allow the new node to be a target > for (pending) writes. Unfortunately, this does not help with availability > (in fact, it decreases write availability, since we will have to collect > 2+1 mandatory write responses instead of just 2), but increases durability, > and I think helps to fully eliminate the second phase. This also increases > read availability when the source node is up, since we can still use the > source node as a part of read quorum. > > I think if we want to call this feature "live migration", since this term > is used in hypervisor community to describe an instant and uninterrupted > instance migration from one host to the other without guest instance being > able to notice as much as the time jump, we may want to provide similar > guarantees. > > I am also not against to have this to be done post-factum, after > implementation of CEP in its current form, but I think it would be good to > have good understanding of availability and durability guarantees we want > to provide with it, and have it stated explicitly, for both "source node > down" and "source node up" cases. That said, since we will have to > integrate CEP-40 with TCM, and will have to ensure correctness of sstable > diffing for the second phase, it might make sense to consider reusing some > of the existing replacement logic from TCM. Just to make sure this is > mentioned explicitly, my proposal is only concerned with the second copy > phase, without any implications about the first. > > Thank you, > --Alex > > On Fri, Apr 5, 2024, at 12:46 PM, Venkata Hari Krishna Nukala wrote: > > Hi all, > > I have filed CEP-40 [1] for live migrating Cassandra instances using the > Cassandra Sidecar. > > When someone needs to move all or a portion of the Cassandra nodes > belonging to a cluster to different hosts, the traditional approach of > Cassandra node replacement can be time-consuming due to repairs and the > bootstrapping of new nodes. Depending on the volume of the storage service > load, replacements (repair + bootstrap) may take anywhere from a few hours > to days. > > Proposing a Sidecar based solution to address these challenges. This > solution proposes transferring data from the old host (source) to the new > host (destination) and then bringing up the Cassandra process at the > destination, to enable fast instance migration. This approach would help to > minimise node downtime, as it is based on a Sidecar solution for data > transfer and avoids repairs and bootstrap. > > Looking forward to the discussions. > > [1] > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-40%3A+Data+Transfer+Using+Cassandra+Sidecar+for+Live+Migrating+Instances > > Thanks! > Hari > > >
