On Tue, Mar 3, 2020 at 4:02 PM Andy Taylor <andy.tayl...@gmail.com> wrote:
> Personally I wouldn't use Zookeeper, I think there are better options. Also > looks like Kafka are replacing it as well. Saying that, it doesn't really > matter what is used, the main thing is we need to remove the burden of > providing consensus away from the broker. https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum There is a blog from person who prototyped replacing Zookeeper with Atomix.io. https://medium.com/@lukasz.antoniak/apache-kafka-leaves-the-zoo-bef529ba82b7. It is afaik independent work to the KIP-500, as the proposal does not say Atomix.io is to be used. The blog links to other reimplementations/forks; one that replaces Zookeeper with etcd, and another one that does something with Go and Raft. Rabbitmq has implemented Raft in Erlang and they now have "mirrored queues" (legacy, the pre-raft version) and Quorum queues, the new thing ( https://www.rabbitmq.com/quorum-queues.html) There is a presentation that tries to explain it to people who don't know RabbitMQ https://www.slideshare.net/Pivotal/implementing-raft-in-rabbitmq. For some reason, they wanted to vote up a leader for each queue separately, and then they had to make it perform well. On Mon, Mar 9, 2020 at 4:02 PM Clebert Suconic <clebert.suco...@gmail.com> wrote: > I think we should have a management component, that runs outside of > the broker and would manage quorum. > > That way you can have the quorum running outside of the broker itself. > which would improve the need of multiple brokers to manage the quorum. > You just need Quorum Managers in distinct places. > > I had recently worked with a software called Ceph... And Ceph has the > concept of managers working away from their "broker" (it's not a > broker.. it's a DB, but in a sense it's the same concept here). I > think we should do the same. > One of the reasons for KIP-500 is simplification of deployment. From this point of view, taking inspiration for ActiveMQ from Ceph, which is notoriously hard to install, at least in folk memory, could be a step back. Also see section Rejected Alternatives -> Pluggable Consensus in the KIP. Btw, Kafka has controller brokers and ordinary brokers. Sounds a bit like Ceph manager.... Also, a Kafka broker is a DB of sorts (of events). -- Mit freundlichen Grüßen / Kind regards Jiri Daněk
