A setup I have used in the past was to have an observer I DC2. If DC1 one goes boom you need manual intervention to change observer's role to make it a follower.
When DC1 comes back up change on instance in DC2 to make it a observer again On May 24, 2017 6:15 PM, "Jan Høydahl" <jan....@cominvent.com> wrote: > Sure, ZK does by design not support a two-node/two-location setup. But > still, users may want/need to deploy that, > and my question was if there are smart ways to make such a setup as little > painful as possible in case of failure. > > Take the example of DC1: 3xZK and DC2: 2xZK again. And then DC1 goes BOOM. > Without an active action DC2 would be read-only > What if then the Ops personnel in DC2 could, with a single script/command, > instruct DC2 to resume “master” role: > - Add a 3rd DC2 ZK to the two existing, reconfigure and let them sync up. > - Rolling restart of Solr nodes with new ZK_HOST string > Of course, they would also then need to make sure that DC1 does not boot > up again before compatible change has been done there too. > > -- > Jan Høydahl, search solution architect > Cominvent AS - www.cominvent.com > > > 23. mai 2017 kl. 18.56 skrev Shawn Heisey <elyog...@elyograg.org>: > > > > On 5/23/2017 10:12 AM, Susheel Kumar wrote: > >> Hi Jan, FYI - Since last year, I have been running a Solr 6.0 cluster > in one of lower env with 6 shards/replica in dc1 & 6 shard/replica in dc2 > (each shard replicated cross data center) with 3 ZK in dc1 and 2 ZK in dc2. > (I didn't have the availability of 3rd data center for ZK so went with only > 2 data center with above configuration) and so far no issues. Its been > running fine, indexing, replicating data, serving queries etc. So in my > test, setting up single cluster across two zones/data center works without > any issue when there is no or very minimal latency (in my case around 30ms > one way > > > > With that setup, if dc2 goes down, you're all good, but if dc1 goes > down, you're not. > > > > There aren't enough ZK servers in dc2 to maintain quorum when dc1 is > unreachable, and SolrCloud is going to go read-only. Queries would most > likely work, but you would not be able to change the indexes at all. > > > > ZooKeeper with N total servers requires int((N/2)+1) servers to be > operational to maintain quorum. This means that with five total servers, > three must be operational and able to talk to each other, or ZK cannot > guarantee that there is no split-brain, so quorum is lost. > > > > ZK in two data centers will never be fully fault-tolerant. There is no > combination of servers that will work properly. You must have three data > centers for a geographically fault-tolerant cluster. Solr would be > optional in the third data center. ZK must be installed in all three. > > > > Thanks, > > Shawn > > > >
