On Wed, 3 Apr 2019 at 18:38, Oleksandr Shulgin <oleksandr.shul...@zalando.de> wrote: > > On Wed, Apr 3, 2019 at 12:28 AM Saleil Bhat (BLOOMBERG/ 731 LEX) > <sbha...@bloomberg.net> wrote: >> >> >> The standard procedure for doing this seems to be add a 3rd datacenter to >> the cluster, stream data to the new datacenter via nodetool rebuild, then >> decommission the old datacenter. A more detailed review of this procedure >> can be found here: >> http://thelastpickle.com/blog/2019/02/26/data-center-switch.html >> >> However, I see two problems with the above protocol. First, it requires >>changes on the application layer because of the datacenter name change; e.g. >>all applications referring to the datacenter ‘Orlando’ will now have to be >>changed to refer to ‘Tampa’. > > > Alternatively, you may omit DC specification in the client and provide > internal network addresses as the contact points.
I am afraid you are mixing two things together. I believe OP means that he has to change local dc in DCAwareRoundRobinPolicy. I am not sure what contact points have to do with that. If there is at least one contact point from DC nobody removes all should be fine. The process in the article is right. Before transitioning to new DC one has to be sure that all writes and reads still target old dc too after you alter a keyspace and add new dc there so you dont miss any write when something goes south and you have to switch it back. Thats achieved by local_one / local_quorum and DCAwareRoundRobinPolicy with localDc pointing to the old one. Then you do rebuild and you restart your app in such way that new DC will be in that policy so new writes and reads are going primarily to that DC and once all is fine you drop the old one (you can do maybe additional repair to be sure). I think the rolling restart of the app is inevitable but if services are in some kind of HA setup I dont see a problem with that. From outside it would look like there is not any downtime. OP has a problem with repair on nodes and it is true that can be time consuming, even not doable, but there are workarounds around that and I do not want to go into here. You can speed this process significantly when you are smart about that and you repair in smaller chunks so you dont clog your cluster completely, its called subrange repair. >> As such, I was wondering what peoples’ thoughts were on the following >> alternative procedure: >> 1) Kill one node in the old datacenter >> 2) Add a new node in the new datacenter but indicate that it is to REPLACE >> the one just shutdown; this node will bootstrap, and all the data which it >> is supposed to be responsible for will be streamed to it > > > I don't think this is going to work. First, I believe streaming for > bootstrap or for replacing a node is DC-local, so the first node won't have > any peers to stream from. Even if it would stream from the remote DC, this > single node will own 100% of the ring and will most likely die of the load > well before it finishes streaming. > > Regards, > -- > Alex > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
