> Alex, just want to make sure that I understand your point correctly. Are you > suggesting this sequence of operations with TCM? > > * Make config changes > * Do the initial data copy > * Make destination part of write placements (same as source) > * Start destination instance > * Decommission the source > * Enable reads for destination by making it part of read placements (as > source)
Almost. I am suggesting reuse the logic we have in TCM and already use for bootstraps and replacements. I think the way it'll be sequencing will be something like: * Make config changes * Start destination instance * Make destination part of write placements (same as source) * Do the initial data copy * Load sstables from the initial data copy * Enable reads for destination by making it part of read placements * Decommission the source We've also had a short discussion offline, and brought up a good point that this may require extra care for making sure that initial data copy sstables aren't involved in the regular node sstable lifecycle, since in that case we may inadvertently remove or compact them. > It is a fair point. It is good to have the understanding of availability and > durability guarantees during migration. I can create a JIRA for it later. Sounds good. As I mentioned, I'm fine either way: if we do it as a part of CEP, or as a follow-up. On Sun, May 12, 2024, at 8:18 PM, Venkata Hari Krishna Nukala wrote: > Replies from my side for the other points of the discussion: > *Managing C* life cycle with Sidecar* > > >lifecycle / orchestration portion is the more challenging aspect. It would > >be nice to address that as well so we don’t end up with something like > >repair where the building blocks are there but the hard parts are left to > >the operator > > CEP-1 has lifecycle operations under scope. > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652224#CEP1:ApacheCassandraManagementProcess(es)-3.Lifecycle(safelystart,stop,restartC*). > I think it can be leveraged when implemented as part of CEP-1. > > *On backup & restore use case* > > I see similarities between backup/restore & this migration. But I feel there > will be considerable differences while implementing it and we might need to > tailor the API to make it usable for backup & restore. I think making the > code/logic reusable can be an implicit goal. Does calling backup & restore - > a stretch goal or creating a separate CEP sounds fair? > > *Migrate the largest keyspace to a dedicated cluster* > > Parick, proposed API can help to copy specific keyspace data to another > cluster. "No chance of doing this manually without some serious brain surgery > on c* and downtime." - sounds a bit tricky to me. Since the clusters are > independent, doing it without any coordination between clusters and downtime > sounds like a case this CEP is not targeting at the moment. > > *Live migration + TCM* > > >We can implement CEP-40 using a similar approach: we can leave the source > >node as both a read and write target, and allow the new node to be a target > >for (pending) writes. Unfortunately, this does not help with availability > >(in fact, it decreases write availability, since we will have to collect 2+1 > >mandatory write responses instead of just 2), but increases durability, and > >I think helps to fully eliminate the second phase. This also increases read > >availability when the source node is up, since we can still use the source > >node as a part of the read quorum. > > Alex, just want to make sure that I understand your point correctly. Are you > suggesting this sequence of operations with TCM? > > * Make config changes > * Do the initial data copy > * Make destination part of write placements (same as source) > * Start destination instance > * Decommission the source > * Enable reads for destination by making it part of read placements (as > source) > > >I am also not against to have this to be done post-factum, after > >implementation of CEP in its current form, but I think it would be good to > >have good understanding of availability and durability guarantees we want to > >provide with it, and have it stated explicitly, for both "source node down" > >and "source node up" cases. > > It is a fair point. It is good to have the understanding of availability and > durability guarantees during migration. I can create a JIRA for it later. > > Thanks! > Hari > > On Thu, May 2, 2024 at 12:30 PM Alex Petrov <al...@coffeenco.de> wrote: >> __ >> Thank you for input! >> >> > Would it be possible to create a new type of write target node? The new >> > write target node is notified of writes (like any other write node) but >> > does not participate in the write availability calculation. >> >> We could make a some kind of optional write, but unfortunately this way we >> can not codify our consistency level. Since we already use a notion of >> pending ranges that requires 1 extra ack, and we as a community are OK with >> it, I think for simplicity we should stick to the same notion. >> >> If there is a lot of interest in this kind of availability/durability >> tradeoff, we should discuss all implications in a separate CEP, but then it >> probably would make sense to make it available for all operations. >> >> My personal opinion is that if we can't guarantee/rely on the number of >> acks, this may accidentally mislead people as they would expect it to work >> and lead to surprises when it does not. >> >> On Wed, May 1, 2024, at 4:38 PM, Claude Warren, Jr via dev wrote: >>> Alex, >>> >>> you write: >>>> We can implement CEP-40 using a similar approach: we can leave the source >>>> node as both a read and write target, and allow the new node to be a >>>> target for (pending) writes. Unfortunately, this does not help with >>>> availability (in fact, it decreases write availability, since we will have >>>> to collect 2+1 mandatory write responses instead of just 2), but increases >>>> durability, and I think helps to fully eliminate the second phase. This >>>> also increases read availability when the source node is up, since we can >>>> still use the source node as a part of read quorum. >>> >>> Would it be possible to create a new type of write target node? The new >>> write target node is notified of writes (like any other write node) but >>> does not participate in the write availability calculation. In this way a >>> node this is being migrated to could receive writes and have minimal impact >>> on the current operation of the cluster? >>> >>> Claude >>> >>> >>> >>> On Wed, May 1, 2024 at 12:33 PM Alex Petrov <al...@coffeenco.de> wrote: >>>> __ >>>> Thank you for submitting this CEP! >>>> >>>> Wanted to discuss this point from the description: >>>> >>>> > How to bring up/down Cassandra/Sidecar instances or making/applying >>>> > config changes are outside the scope of this document. >>>> >>>> One advantage of doing migration via sidecar is the fact that we can >>>> stream sstables to the target node from the source node while the source >>>> node is down. Also if the source node is down, it does not matter if we >>>> can’t use it as a write target However, if we are replacing a live node, >>>> we do lose both durability and availability during the second copy phase. >>>> There are copious other advantages described by others in the thread above. >>>> >>>> For example, we have three adjacent nodes A,B,C and simple RF 3. C >>>> (source) is up and is being replaced with live-migrated D (destination). >>>> According to the described process in CEP-40, we perform streaming in 2 >>>> phases: first one is a full copy (similar to bootstrap/replacement in >>>> cassandra), and the second one is just a diff. The second phase is still >>>> going to take a non-trivial amount of time, and is likely to last at very >>>> least minutes. During this time, we only have nodes A and B as both read >>>> and write targets, with no alternatives: we have to have both of them >>>> present for any operation, and losing either one of them leaves us with >>>> only one copy of data. >>>> >>>> To contrast this, TCM bootstrap process is 4-step: between the old owner >>>> being phased out and the new owner brought in, we always ensure r/w quorum >>>> consistency and liveness of at least 2 nodes for the read quorum, 3 nodes >>>> available for reads in best case, and 2+1 pending replica for the write >>>> quorum, with 4 nodes (3 existing owners + 1 pending) being available for >>>> writes in best case. Replacement in TCM is implemented similarly, with the >>>> old node remaining an (unavailable) read target, but new node already >>>> being the target for (pending) writes. >>>> >>>> We can implement CEP-40 using a similar approach: we can leave the source >>>> node as both a read and write target, and allow the new node to be a >>>> target for (pending) writes. Unfortunately, this does not help with >>>> availability (in fact, it decreases write availability, since we will have >>>> to collect 2+1 mandatory write responses instead of just 2), but increases >>>> durability, and I think helps to fully eliminate the second phase. This >>>> also increases read availability when the source node is up, since we can >>>> still use the source node as a part of read quorum. >>>> >>>> I think if we want to call this feature "live migration", since this term >>>> is used in hypervisor community to describe an instant and uninterrupted >>>> instance migration from one host to the other without guest instance being >>>> able to notice as much as the time jump, we may want to provide similar >>>> guarantees. >>>> >>>> I am also not against to have this to be done post-factum, after >>>> implementation of CEP in its current form, but I think it would be good to >>>> have good understanding of availability and durability guarantees we want >>>> to provide with it, and have it stated explicitly, for both "source node >>>> down" and "source node up" cases. That said, since we will have to >>>> integrate CEP-40 with TCM, and will have to ensure correctness of sstable >>>> diffing for the second phase, it might make sense to consider reusing some >>>> of the existing replacement logic from TCM. Just to make sure this is >>>> mentioned explicitly, my proposal is only concerned with the second copy >>>> phase, without any implications about the first. >>>> >>>> Thank you, >>>> --Alex >>>> >>>> On Fri, Apr 5, 2024, at 12:46 PM, Venkata Hari Krishna Nukala wrote: >>>>> Hi all, >>>>> >>>>> I have filed CEP-40 [1] for live migrating Cassandra instances using the >>>>> Cassandra Sidecar. >>>>> >>>>> When someone needs to move all or a portion of the Cassandra nodes >>>>> belonging to a cluster to different hosts, the traditional approach of >>>>> Cassandra node replacement can be time-consuming due to repairs and the >>>>> bootstrapping of new nodes. Depending on the volume of the storage >>>>> service load, replacements (repair + bootstrap) may take anywhere from a >>>>> few hours to days. >>>>> >>>>> Proposing a Sidecar based solution to address these challenges. This >>>>> solution proposes transferring data from the old host (source) to the new >>>>> host (destination) and then bringing up the Cassandra process at the >>>>> destination, to enable fast instance migration. This approach would help >>>>> to minimise node downtime, as it is based on a Sidecar solution for data >>>>> transfer and avoids repairs and bootstrap. >>>>> >>>>> Looking forward to the discussions. >>>>> >>>>> [1] >>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-40%3A+Data+Transfer+Using+Cassandra+Sidecar+for+Live+Migrating+Instances >>>>> >>>>> Thanks! >>>>> Hari >>>> >>
