Thanks Colin, interesting KIP. I'm concerned that the KIP does not actually address its stated motivations. In particular, "Simpler Deployment and Configuration" are not really achieved, given that: 1) the proposal still requires quorums (now of controllers, instead of ZK nodes), with the same restrictions as ZK, e.g. at least three controllers and only an odd number of controllers, neither of which is easy to manage; 2) the proposal still requires separate processes with separate configuration (and indeed, no less configuration than ZK requires, namely a port to listen on); 3) configuration of brokers is not simplified, as they still require a list of servers to contact (now coordinators instead of ZK nodes). Is there any improvement to configuration and deployment I'm overlooking?
Second, single-broker clusters are mentioned as a motivation, but it is unclear how this change would move in that direction. Seems Raft requires three nodes, so perhaps the minimum number of hosts would be three? Third, "discrepancies between the controller state and the zookeeper state" are mentioned as a problem, and I understand that controllers coordinate amongst themselves rather than via zookeeper, but I'm not sure there is a functional difference? It seems controllers can still disagree amongst themselves for periods of time, with the same consequences as disagreeing with ZK. Finally, you say "there is no generic way for the controller to follow the ZooKeeper event log." I'm unsure this is a problem. Having a log is certainly powerful for consumers, but how would a controller use this log to do anything it can't without it? It seems only the latest compacted state is ever used, and there is nothing to undo or replay from the log. What future capabilities are you envisioning we would gain from carrying around log history? Ryanne On Thu, Aug 1, 2019, 4:05 PM Colin McCabe <cmcc...@apache.org> wrote: > Hi all, > > I've written a KIP about removing ZooKeeper from Kafka. Please take a > look and let me know what you think: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum > > cheers, > Colin >
