Re: Using 100's of ZK Observers

Thanks Fangmin. That's an Interesting feature - allowing followers to host observers. but I assume the entire collection of servers is still considered part of the ensemble. If so, isn't the upper limit still capped to 256 - by the lowest 8 bits of the server id? On Fri, Apr 10, 2020 at 5:32 PM F

Re: Using 100's of ZK Observers

There is ObserverMaster feature contributed back in ZOOKEEPER-3140 could be used to scale the number of observers and traffics a single ensemble can support. It allows followers to serve observers as well, which relieves the fanout load on lea

Re: Using 100's of ZK Observers

If you have 100s of 1000s of ZK clients then having observer in each pod will presumably reduce traffic as most of the fan out traffic, from server to clients is localized to each pod. Observer is not part of quorum, and a quorum can't scale pass a few servers (typical just 5 or 7). Observers can

Re: Using 100's of ZK Observers

That was my instinct as well. I *think* any ZK writes would require a quorum before the transaction is committed. Getting a quorum over a several hundred/thousand node ensemble seems like a lot of traffic. Plus, from what I've read - though not 100% certain, it seems the number ZK nodes is capped a

Re: Using 100's of ZK Observers

On 10/04/2020 20:13, James Arbo wrote: > When we proposed this, there was great concern from the software architects > that network traffic between the kubernetes pods and the ZK ensemble must > be minimized. > This means that, at a minimum, we would be running at least 1 ZK ensemble > member on

Using 100's of ZK Observers

We are looking to use ZK as a discovery service. This ensemble will be read from/written to by Apache ignites event bus. The event bus allows multiple running processes to share data across nodes. This is all running in a K8s cluster. Our plan was to deploy a 5-7 instance ZK ensemble accessible vi