Re: Disconnecting two data centers
On 2017-03-08 07:57 (-0800), Chuck Reynolds wrote: > I was hoping I could do the following > > · Change seeds Definitely. > > · Change the topology back to simply > Not necessary, can just remove the "other" datacenter from the replication strategy. > · Stop nodes in datacenter 2 > > · Remove nodes in datacenter 2 > >From where? system.peers? > · Restart nodes in datacenter 2 Then dc2 connects back to dc1 and you end up messed up again. BOTH SIDES are going to try to reconnect, and it's going to be ugly. If you used an internode authenticator, you could make that lock out the other cluster. If you used a property file snitch (NOT GPFS, plain old PFS), you could remove the other datacenter from each topology file. You can use a firewall. You'll defintiely need to change seeds, and probably need to stop most of the nodes all at once to enable this. > > Somehow Cassandra holds on to the information about who was in the cluster. > system.peers keeps a list of all members in the cluster. System is local strategy (not replicated), so each node has its own copy, so IF you try hacking at it, you MUST hack at it on all nodes (basically at the same time, because they'll repopulate via gossip). > What if I also changed the cluster name in the Cassandra.yaml before > restarting? > Changing the name is very difficult (decidedly nontrivial). Given that you need to stop all the nodes to do this (downtime), I'd be pretty tempted to tell you to nuke one of the clusters, and use sstableloader to repopulate it as a brand new cluster after you've nuked it. That is - Don't try to split a cluster in half, create it, then kill half, then make a new cluster, and use sstableloader to repopulate it quickly.
Re: Disconnecting two data centers
Do not change the cluster name - the cassandra service will not start on the same sstables if the cluster name is changed. Arvydas On Wed, Mar 8, 2017 at 4:57 PM, Chuck Reynolds wrote: > I was hoping I could do the following > > · Change seeds > > · Change the topology back to simply > > · Stop nodes in datacenter 2 > > · Remove nodes in datacenter 2 > > · Restart nodes in datacenter 2 > > > > Somehow Cassandra holds on to the information about who was in the cluster. > > > > What if I also changed the cluster name in the Cassandra.yaml before > restarting? > > > > Is the data bound to the cluster name? > > > > *From: *Ryan Svihla > *Reply-To: *"user@cassandra.apache.org" > *Date: *Wednesday, March 8, 2017 at 7:38 AM > *To: *user > *Subject: *Re: Disconnecting two data centers > > > > it's a bit tricky and I don't advise it, but the typical pattern is (say > you have DC1 and DC2): > > > > 1. partition the data centers from one another..kill the routing however > you can (firewall, etc) > > 2. while partitioned log onto DC1 alter schema so that DC2 is not > replicating), repeat for other. > > 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property > files and vice versa > > 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's > shouldn't have any seeds from DC2, etc) > > 3. rolling restart to account for this. > > 4,. run repair (not even sure how necessary this step is, but after doing > RF changes I do this to prevent hiccups) > > > > I've done this a couple of times but really failing all of that, the more > well supported and harder to mess up but more work approach is: > > > > 1. Set DC2 to RF 0 > > 2. remove all nodes from DC2 > > 3. change yamls for seed files (update property file if need be) > > 4. create new cluster in DC2, > > 5. use sstableloader to stream DC1 data to DC2. > > > > On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds > wrote: > > I’m running C* 2.1.13 and I have two rings that are replicating data from > our data center to one in AWS. > > > > We would like to keep both of them for a while but we have a need to > disconnect them. How can this be done? > > > > > > -- > > Thanks, > > Ryan Svihla >
Re: Disconnecting two data centers
I guess it depends on the experience one has. This is a common process to bring up, move, build full prod copies, etc. What is outlined is pretty much exactly what I have done 20-50 times (too many to remember). FYI, some of this should be done with nodes DOWN. *...* *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Wed, Mar 8, 2017 at 6:38 AM, Ryan Svihla wrote: > it's a bit tricky and I don't advise it, but the typical pattern is (say > you have DC1 and DC2): > > 1. partition the data centers from one another..kill the routing however > you can (firewall, etc) > 2. while partitioned log onto DC1 alter schema so that DC2 is not > replicating), repeat for other. > 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property > files and vice versa > 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's > shouldn't have any seeds from DC2, etc) > 3. rolling restart to account for this. > 4,. run repair (not even sure how necessary this step is, but after doing > RF changes I do this to prevent hiccups) > > I've done this a couple of times but really failing all of that, the more > well supported and harder to mess up but more work approach is: > > 1. Set DC2 to RF 0 > 2. remove all nodes from DC2 > 3. change yamls for seed files (update property file if need be) > 4. create new cluster in DC2, > 5. use sstableloader to stream DC1 data to DC2. > > On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds > wrote: > >> I’m running C* 2.1.13 and I have two rings that are replicating data from >> our data center to one in AWS. >> >> >> >> We would like to keep both of them for a while but we have a need to >> disconnect them. How can this be done? >> > > > > -- > > Thanks, > Ryan Svihla > >
Re: Disconnecting two data centers
I was hoping I could do the following · Change seeds · Change the topology back to simply · Stop nodes in datacenter 2 · Remove nodes in datacenter 2 · Restart nodes in datacenter 2 Somehow Cassandra holds on to the information about who was in the cluster. What if I also changed the cluster name in the Cassandra.yaml before restarting? Is the data bound to the cluster name? From: Ryan Svihla Reply-To: "user@cassandra.apache.org" Date: Wednesday, March 8, 2017 at 7:38 AM To: user Subject: Re: Disconnecting two data centers it's a bit tricky and I don't advise it, but the typical pattern is (say you have DC1 and DC2): 1. partition the data centers from one another..kill the routing however you can (firewall, etc) 2. while partitioned log onto DC1 alter schema so that DC2 is not replicating), repeat for other. 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property files and vice versa 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's shouldn't have any seeds from DC2, etc) 3. rolling restart to account for this. 4,. run repair (not even sure how necessary this step is, but after doing RF changes I do this to prevent hiccups) I've done this a couple of times but really failing all of that, the more well supported and harder to mess up but more work approach is: 1. Set DC2 to RF 0 2. remove all nodes from DC2 3. change yamls for seed files (update property file if need be) 4. create new cluster in DC2, 5. use sstableloader to stream DC1 data to DC2. On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds mailto:creyno...@ancestry.com>> wrote: I’m running C* 2.1.13 and I have two rings that are replicating data from our data center to one in AWS. We would like to keep both of them for a while but we have a need to disconnect them. How can this be done? -- Thanks, Ryan Svihla
Re: Disconnecting two data centers
it's a bit tricky and I don't advise it, but the typical pattern is (say you have DC1 and DC2): 1. partition the data centers from one another..kill the routing however you can (firewall, etc) 2. while partitioned log onto DC1 alter schema so that DC2 is not replicating), repeat for other. 2a. If using propertyfilesnitch remove the DC2 from all the DC1 property files and vice versa 2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's shouldn't have any seeds from DC2, etc) 3. rolling restart to account for this. 4,. run repair (not even sure how necessary this step is, but after doing RF changes I do this to prevent hiccups) I've done this a couple of times but really failing all of that, the more well supported and harder to mess up but more work approach is: 1. Set DC2 to RF 0 2. remove all nodes from DC2 3. change yamls for seed files (update property file if need be) 4. create new cluster in DC2, 5. use sstableloader to stream DC1 data to DC2. On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds wrote: > I’m running C* 2.1.13 and I have two rings that are replicating data from > our data center to one in AWS. > > > > We would like to keep both of them for a while but we have a need to > disconnect them. How can this be done? > -- Thanks, Ryan Svihla
Disconnecting two data centers
I’m running C* 2.1.13 and I have two rings that are replicating data from our data center to one in AWS. We would like to keep both of them for a while but we have a need to disconnect them. How can this be done?
