On 5/14/2019 4:55 PM, Cody Burleson wrote:
I’m worried, for example, about spreading the Zookeper cluster between the two data centers because of potential latency across the pond. Maybe we keep the ZK ensemble on one side of the pond only? I imagined, for instance, 2 ZK nodes on one server, and one on the other (in at least one data center). But maybe we need 5 ZKs, with 1 on each server in the other data center? Then how about the Solr nodes, shards, and replicas? If anybody has done some remotely similar setup for production purposes, I would be grateful for any tips (and down-right giddy for a diagram).
If you're planning a geographically diverse ZooKeeper setup, you cannot do it with only two datacenters. You need at least three. This is inherent to the design of ZK and cannot be changed. With two data centers, you will always have one DC that if it goes down, ZK loses quorum. When ZK loses quorum, SolrCloud loses the ability to react to failures and goes read-only.
You mentioned CDCR. This involves two completely separate SolrCloud clusters -- a full ZK ensemble in each location. So you would have 3 ZK servers and at least two Solr servers in one data center, and 3 ZK servers plus at least two Solr servers in the other data center.
Thanks, Shawn
