On Wed, Oct 5, 2022, at 10:06, Mickael Maison wrote: > Hi David, > > Thanks for starting this important KIP. > > I've just taken a quick look so far but I've got a couple of initial > questions: > > 1) What happens if a non KRaft compatible broker (or with > kafka.metadata.migration.enable set to false) joins the cluster after > the migration is triggered? >
Hi Mickael, Yes, I also asked this :) I think the answer is that it's an administrative error, and the broker will not be able to function. Maybe we should spell out that the broker should shut itself down in this scenario. Obviously this will not be possible for already released versions, but a broker running the new 3.4 code but with something that disqualifies it from upgrade-from-ZK (like the wrong IBP) could shut itself down if it knew it was in the wrong spot. > > 2) In the Failure Modes section you mention a scenario where a write > to ZK fails. What happens when the divergence limit is reached? Is > this a fatal condition? How much divergence should we allow? > Yes, this is one of the complexities of supporting async mirroring to ZK. I don't think it's quite as bad as it sounds because we still have the ordered __cluster_metadata log. I would suggest that if our backlog grows too high, we just block the kraft controller thread until it reduces. We really cannot afford to have an unbounded backlog. I would expected the performance to be comparable to the current controller's performance since we are doing the same stuff (less stuff actually, since we aren't doing reads...) best, Colin > Thanks, > Mickael > > On Wed, Oct 5, 2022 at 12:20 AM David Arthur <mum...@gmail.com> wrote: >> >> Hey folks, I wanted to get the ball rolling on the discussion for the >> ZooKeeper migration KIP. This KIP details how we plan to do an online >> migration of metadata from ZooKeeper to KRaft as well as a rolling >> upgrade of brokers to KRaft mode. >> >> The general idea is to keep KRaft and ZooKeeper in sync during the >> migration, so both types of brokers can exist simultaneously. Then, >> once everything is migrated and updated, we can turn off ZooKeeper >> writes. >> >> This is a pretty complex KIP, so please take a look :) >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration >> >> Thanks! >> David
