David Arthur created KAFKA-15825: ------------------------------------ Summary: KRaft controller writes empty state to ZK after migration Key: KAFKA-15825 URL: https://issues.apache.org/jira/browse/KAFKA-15825 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 3.6.0 Reporter: David Arthur Assignee: David Arthur Fix For: 3.7.0, 3.6.1
Immediately following the ZK migration, there is a race condition where the KRaftMigrationDriver can use an empty MetadataImage when performing the full "SYNC_KRAFT_TO_ZK" reconciliation. After the next controller failover, or when the controller loads a metadata snapshot, the correct state will be written to ZK. The symptom of this bug is that we see the migration complete, and then all the metadata removed from ZK. For example, {code} [KRaftMigrationDriver id=9990] Completed migration of metadata from ZooKeeper to KRaft. 573 records were generated in 2204 ms across 51 batches. The record types were {TOPIC_RECORD=41, PARTITION_RECORD=410, CONFIG_RECORD=121, PRODUCER_IDS_RECORD=1}. The current metadata offset is now 503794 with an epoch of 21. Saw 6 brokers in the migrated metadata [0, 1, 2, 3, 4, 5]. {code} immediately followed by: {code} [KRaftMigrationDriver id=9990] Made the following ZK writes when reconciling with KRaft state: {DeleteBrokerConfig=7, DeleteTopic=41, UpdateTopicConfig=41} {code} If affected by this, a quick workaround is to cause the controller to failover. -- This message was sent by Atlassian Jira (v8.20.10#820010)