> I believe that my solution isolates manual change to two ZK nodes in DC2, > while your requires config change to 1 in DC2 and manual start/stop of 1 in > DC1.
Answering my own statement here. Turns out that in order to flip the observer bit for one ZK node, you need to touch the config of all the others, quote from https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html <https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html>: "Secondly, in every server config file, you must add :observer to the server definition line of each Observer”. So basically you would do some sed magic or keep two versions of configs around and switch between them to toggle. Wonder what happens once you have modified the config of one ZK but not the rest, will they be confused and start printing warnings to the logs? Is it safe to to the observer flip as a rolling update or should you take down the whole ZK ensemble for it? And if you lose DC1, then modify the remaining ZKs in DC2 to form its own quorum, then when the DC1 ZKs come up again their config is not aware that some nodes in DC2 have changed from observer to follower - is there a risk that DC1 starts operating alone once it comes up again, or will it try repeatedly to contact the Zk nodes in DC2 and then discover that they disagree about the voting authority of some nodes or what? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com
