ZK 3.5 isn't officially released. It is alpha/beta for years. I wouldn't use it in production.
The setup I proposed : DC1 : 3 nodes, all are non observer's. DC2 : 3 nodes, 2 are non observer's and 1 is observer This means only 5 nodes participate in voting and 3 nodes make quorum. If DC1 goes down, you will have to bounce observer and bring it back as non observer. This will still cause some down time but it is easier to do than trying to add another node when you are already in a fire fighting mode. On May 25, 2017 5:50 PM, "Jan Høydahl" <jan....@cominvent.com> wrote: Thanks for the tip Pushkar, > A setup I have used in the past was to have an observer I DC2. If DC1 one I was not aware that ZK 3.4 supports observers, thought it was a 3.5 feature. So do you setup followers only in DC1 (3x), and observers only in DC2 (3x) and then point each Solr node to all 6 ZKs? Then you could get away by flipping the DC2 observers to followers in a rolling manner and avoid restarting Solr? > When I tested this solr survived just fine, but it been a while. Anirudha you say you tested this with 3.5. Does that mean that ZK3.5 works with Solr? > Whether ZK 3.5 is there or not, there is potential unknown behavior when > dc1 comes back online, unless you can have dc1 personnel shut the > servers down, or block communication between your servers in dc1 and dc2. Shawn, you are right that in the sketched setup, if DC1 is allowed to come back up and join the cluster, then various Solr nodes will use a different ZK connection string and the DC1 ZKs won’t talk to all the DC2 ones. But with the observer model, all nodes know about all ZKs all the time, and the “odd number of ZKs” requirement is only applying to the voting followers, see https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html < https://zookeeper.apache.org/doc/trunk/zookeeperObservers.html> Let’s do a thought experiment: - Have 2 ZKs in DC1, both being followers, and 3 ZKs in DC2, one being follower and the others observers - All Solr nodes have a ZK_HOST=zk1dc1,zk2dc1,zk1dc2,zk2dc2,zk3dc2 connection string, i.e. five nodes where initially 3 form the quorum - At loss of DC2 all is fine - At loss of DC1, you reconfigure zk2dc2 and zk3dc2 from observers to followers, meaning they now are in majority (3 of 5) and writes resume - When DC1 comes back up, zk1dc1 and zk2dc1 are now not a majority and need to sync up from the others - After all is stable again, you flip two DC2 ZKs back to observers I may have misunderstood the observer thing here, but if this is at all doable, this should be scriptable with ssh or ansible quite easily? -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com > 25. mai 2017 kl. 00.35 skrev Pushkar Raste <pushkar.ra...@gmail.com>: > > A setup I have used in the past was to have an observer I DC2. If DC1 one > goes boom you need manual intervention to change observer's role to make it > a follower. > > When DC1 comes back up change on instance in DC2 to make it a observer > again > > On May 24, 2017 6:15 PM, "Jan Høydahl" <jan....@cominvent.com> wrote: > >> Sure, ZK does by design not support a two-node/two-location setup. But >> still, users may want/need to deploy that, >> and my question was if there are smart ways to make such a setup as little >> painful as possible in case of failure. >> >> Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM. >> Without an active action DC2 would be read-only >> What if then the Ops personnel in DC2 could, with a single script/command, >> instruct DC2 to resume “master” role: >> - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up. >> - Rolling restart of Solr nodes with new ZK_HOST string >> Of course, they would also then need to make sure that DC1 does not boot >> up again before compatible change has been done there too. >> >> -- >> Jan Høydahl, search solution architect >> Cominvent AS - www.cominvent.com >> >>> 23. mai 2017 kl. 18.56 skrev Shawn Heisey <elyog...@elyograg.org>: >>> >>> On 5/23/2017 10:12 AM, Susheel Kumar wrote: >>>> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster >> in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 >> (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2. >> (I didn't have the availability of 3rd data center for ZK so went with only >> 2 data center with above configuration) and so far no issues. Its been >> running fine, indexing, replicating data, serving queries etc. So in my >> test, setting up single cluster across two zones/data center works without >> any issue when there is no or very minimal latency (in my case around 30ms >> one way >>> >>> With that setup, if dc2 goes down, you're all good, but if dc1 goes >> down, you're not. >>> >>> There aren't enough ZK servers in dc2 to maintain quorum when dc1 is >> unreachable, and SolrCloud is going to go read-only. Queries would most >> likely work, but you would not be able to change the indexes at all. >>> >>> ZooKeeper with N total servers requires int((N/2)+1) servers to be >> operational to maintain quorum. This means that with five total servers, >> three must be operational and able to talk to each other, or ZK cannot >> guarantee that there is no split-brain, so quorum is lost. >>> >>> ZK in two data centers will never be fully fault-tolerant. There is no >> combination of servers that will work properly. You must have three data >> centers for a geographically fault-tolerant cluster. Solr would be >> optional in the third data center. ZK must be installed in all three. >>> >>> Thanks, >>> Shawn >>> >> >>
