Let's talk about the challenges, then talk about the strategy we'll use to
do this.
The logic cassandra uses to identify the rest of it's cluster comes down to
~3 things:
- Cluster name (in yaml and system.local)
- Seeds in your seed provider (probably a list of IPs in the yaml)
- The known peers in system.peers, which it will use to connect / discover
on cassandra restarts (also used by clients to connect to the rest of the
cluster, so it's important it's accurate)
The cluster name is meant to be mostly immutable - we don't provide an
obvious mechanism to change it, really, because if you change the yaml and
restart, the database should (last I checked) fail to startup when the yaml
doesn't match the data in system.local.
Cassandra keeps a list of all of the other hosts it knows about in
system.peers, and will attempt to reconnect to them as long as it's in
system.peers or gossip.
Most approaches to do this ignore the cluster name and rely on firewalls to
separate the two DCs, then nodetool assassinate to get the IPs from gossip,
but ultimately the two clusters have the same name, and if they EVER get
reminded of the old IP, they'll re-join each other, and you'll be unhappy.
For that reason, we probably want to change the cluster name in one side or
the other to make sure we protect ourself.
Strategy wise, I'd pick one cluster that can take downtime. It won't take
much, but it'll be more than zero, then approach it with something like the
following:
- Firewall off the two clusters so they cant talk.
- Figure out which cluster will keep the name (we'll call it 'old'), and
one which will change the name ('new')
- Push a yaml to old that removes all seeds in the 'new' dc
- Push a yaml to new that removes all seeds in 'old' dc, and has a 'new'
cluster name in it
- Alter the schema in old to remove the 'new' dc in replication settings
- Alter the schema in new to remove the 'old' dc in replication settings
- In the 'new' hosts, change the cluster name in every single instance of
system.local ( update system.local set cluster_name='new' where
key='local'; ) and flush (nodetool flush) on every host
- Restart the new hosts, they'll come up with a new cluster name, and at
this point if the firewall is turned off, both clusters will TRY to talk to
each other, but the different cluster names will prevent it.
- At that point, you can nodetool removenode / nodetool assassinate the
'old' IPs in 'new' and the 'new' IPs in 'old'
- Finally, check system.peers for any stray leftovers - there have been
times when system.peers leaked data. Cleanup anything that's wrong.
- Then remove the firewall rules
Obviously, you want to try this on a test cluster first.
On Thu, Jul 11, 2019 at 8:03 AM Voytek Jarnot <voytek.jar...@gmail.com>
wrote:
> My google-fu is failing me this morning. I'm looking for any tips on
> splitting a 2 DC cluster into two separate clusters. I see a lot of docs
> about decomissioning a datacenter, but not much in the way of disconnecting
> datacenters into individual clusters, but keeping each one as-is data-wise
> (aside from replication factor, of course).
>
> Our setup is simple: two DCs (dc1 and dc2), two seed nodes (both in dc1;
> yes, I know not the recommended config), one keyspace (besides the system
> ones) replicated in both DCs. I'm trying to end up with two clusters with 1
> DC in each.
>
> Would appreciate any input.
>
