Hi Jason,
The KIP looks great. Thanks for all the work you've put into this.
I have a few lingering questions. I still don't like the fact that the leader
epoch / fetch epoch is 31 bits. What happens when this rolls over? Can we
just make this 63 bits now so that we never have to worry about it again? ZK
has some awful bugs surrounding 32 bit rollover, due to a similar decision to
use a 32 bit counter in their log structure. Doesn't seem like a good tradeoff.
For fsync, perhaps we can have a per-topic configuration to enable or disable
fsync on a per-topic basis. This would avoid adding a special hack just for
the metadata topic. It would also make life better for JUnit tests, since we
could disable fsync on the metadata topic in that case and get a performance
boost. To keep the scope small we could support this config only for Raft
topics for now, and only generalize it later if needed.
There are some references to setting the controller ID by using broker.id. I
think we should leave this out of this KIP and discuss it as part of KIP-631
instead. As you mention earlier in the KIP, the controller design should be
out of scope for KIP-595. It's enough to say that we get this value statically.
> quorum.voters: This is a connection map which contains the IDs of the voters
> and their respective endpoint. We use the following format for each voter in
> the list {broker-id}@{broker-host):{broker-port}. For example,
> `quorum.voters=1@kafka-1:9092, 2@kafka-2:9092, 3@kafka-3:9092`.
Just like in bootstrap.servers, I don't think we want to manually assign IDs
per hostname. The hosts know their own IDs, after all. Having to manually
specify the IDs also opens up the possibility of misconfigurations: what I say
the foobar server is node 2, but it's actually node 3? This would make the logs
extremely confusing. I realize this may require a little finesse to do, but
there's got to be a way we can avoid hard-coding IDs.
Also, here's another case where we are saying "broker" when we mean
"controller." It's really hard to break old habits. :)
best,
Colin
On Mon, Aug 3, 2020, at 11:19, Ben Stopford wrote:
> +1
>
> On Mon, 3 Aug 2020 at 19:03, Jason Gustafson <ja...@confluent.io> wrote:
>
> > Hi All, I'd like to start a vote on this proposal:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum
> > .
> > The discussion has been active for a bit more than 3 months and I think the
> > main points have been addressed. We have also moved some of the pieces into
> > follow-up proposals, such as KIP-630.
> >
> > Please keep in mind that the details are bound to change as all of
> > the pieces start coming together. As usual, we will keep this thread
> > notified of such changes.
> >
> > For me personally, this is super exciting since we have been thinking about
> > this work ever since I started working on Kafka! I am +1 of course.
> >
> > Best,
> > Jason
> >
>
>
> --
>
> Ben Stopford
>
> Lead Technologist, Office of the CTO
>
> <https://www.confluent.io>
>
