Hi Kafka community, Two weeks ago, I got a reply from the community that our steps for an in-place KRaft topology change from dedicated controllers (static quorum) to combined broker/controller nodes is feasible, but we have some further questions.
Let's say we currently have a 6-broker + 3-dedicated-controller KRaft Kafka cluster, and we want to eventually sunset controllers 101, 102, 103 and use 1, 2, 3 as mixed-role (broker + controller) nodes. Current dedicated-controller topology: - Controllers: 101, 102, 103 with process.roles=controller - Brokers: 1, 2, 3, 4, 5, 6 with process.roles=broker - Current quorum: controller.quorum.voters=101@host1:9093,102@host2:9093,103@host3:9093 Target mixed-role topology: - Mixed-role nodes: 1, 2, 3 with process.roles=broker,controller - Broker nodes: 4, 5, 6 with process.roles=broker - Target quorum: controller.quorum.voters=1@host1:9093,2@host2:9093,3@host3:9093 Our questions: 1. Is stopping the whole Kafka cluster inevitable to complete the role migration? In other words, is it possible to keep the cluster running throughout the entire migration and still have it succeed? My understanding is that if we want to keep the cluster running, there must be a point in time where both dedicated controller nodes and mixed-role controller nodes coexist. 2. What negative impact could occur if a lagging node is added to the final quorum voters? We tested this scenario locally: one future voter (broker 3) was stopped before the migration, which caused it to lag on metadata. After performing the migration, the new combined-mode quorum still formed, and the lagging node caught up once the new quorum started. No data was lost in the Kafka topics, and publish/subscribe worked as expected. 3. Is it safe to perform the role migration without wiping/reformatting storage? All the official documentation I found states that a role change requires a storage format change (reformatting), but our local tests indicate this is not necessary. 4. Will the role migration have any negative impact on the data state (data integrity and publish/subscribe behavior) in Kafka? I believe the migration is only a controller-metadata change and should have no impact on the data in the Kafka brokers. Steps we tested (Kafka 3.9.1): 1. Before the change, brokers 1, 2, 3, 4, 5, 6 appeared as metadata-log observers. 2. Stopped the cluster, changed the config for brokers 1, 2, 3 to process.roles=broker,controller, and updated controller.quorum.voters to 1@host1:9093,2@host2:9093,3@host3:9093. 3. Started the cluster successfully. The new quorum became 1, 2, 3, and topic operations worked. Best Regards, Benson
