Hi, David, Thanks for the KIP. A few comments below.
10. It's still not very clear to me how the KRaft controller works in the dual writes mode to KRaft log and ZK when the brokers still run in ZK mode. Does the KRaft controller run a ZK based controller in parallel or do we derive what needs to be written to ZK based on KRaft controller logic? I am also not sure how the KRaft controller handles broker registration/deregistration, since brokers are still running in ZK mode and are not heartbeating to the KRaft controller. 11. When preparing the Cluster, each broker needs kafka.metadata.migration.enable set to “true”, right? Could we document that? 12. "A new set of nodes will be provisioned to host the controller quorum." I guess we don't support starting the KRaft controller quorum on existing brokers. It would be useful to make that clear. 13. "Once the quorum is established and a leader is elected, the controller will check the state of the cluster using the MigrationCheck RPC." How does the quorum controller detect other brokers? Does the controller node need to be configured with ZK connection string? If so, it would be useful to document the additional configs that the quorum controller needs to set. 14. "In order to prevent further writes to ZK, the first thing the new KRaft quorum must do is take over leadership of the ZK controller. " The ZK controller processing changes to /controller update asynchronously. How does the KRaft controller know when the ZK controller has resigned before it can safely copy the ZK data? 15. We have the following sentences. One says ControllerId is a random KRaft broker and the other says it's the active controller. Which one is correct? "UpdateMetadata: for certain metadata changes, the KRaft controller will need to send UpdateMetadataRequests to the ZK brokers. For the “ControllerId” field in this request, the controller should specify a random KRaft broker." "In the UpdateMetadataRequest sent by the KRaft controller to the ZK brokers, the ControllerId will point to the active controller which will be used for the inter-broker requests." 16. "Additionally, the controller must specify if a broker in “LiveBrokers” is KRaft or ZK." Does that require any protocol changes to UpdateMetadata? Thanks, Jun On Wed, Oct 5, 2022 at 10:07 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi David, > > Thanks for starting this important KIP. > > I've just taken a quick look so far but I've got a couple of initial > questions: > > 1) What happens if a non KRaft compatible broker (or with > kafka.metadata.migration.enable set to false) joins the cluster after > the migration is triggered? > > 2) In the Failure Modes section you mention a scenario where a write > to ZK fails. What happens when the divergence limit is reached? Is > this a fatal condition? How much divergence should we allow? > > Thanks, > Mickael > > On Wed, Oct 5, 2022 at 12:20 AM David Arthur <mum...@gmail.com> wrote: > > > > Hey folks, I wanted to get the ball rolling on the discussion for the > > ZooKeeper migration KIP. This KIP details how we plan to do an online > > migration of metadata from ZooKeeper to KRaft as well as a rolling > > upgrade of brokers to KRaft mode. > > > > The general idea is to keep KRaft and ZooKeeper in sync during the > > migration, so both types of brokers can exist simultaneously. Then, > > once everything is migrated and updated, we can turn off ZooKeeper > > writes. > > > > This is a pretty complex KIP, so please take a look :) > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration > > > > Thanks! > > David >
