Luke Chen created KAFKA-14197: --------------------------------- Summary: Kraft broker fails to startup after topic creation failure Key: KAFKA-14197 URL: https://issues.apache.org/jira/browse/KAFKA-14197 Project: Kafka Issue Type: Bug Components: kraft Reporter: Luke Chen
In kraft ControllerWriteEvent, we start by trying to apply the record to controller in-memory state, then sent out the record via raft client. But if there is error during sending the records, there's no way to revert the change to controller in-memory state[1]. The issue happened when creating topics, controller state is updated with topic and partition metadata (ex: broker to ISR map), but the record doesn't send out successfully (i.e. buffer allocation error). Then, when shutting down the node, the controlled shutdown will try to remove the broker from ISR by[2]: {code:java} generateLeaderAndIsrUpdates("enterControlledShutdown[" + brokerId + "]", brokerId, NO_LEADER, records, brokersToIsrs.partitionsWithBrokerInIsr(brokerId));{code} After we appending the partitionChangeRecords, and send to metadata topic successfully, it'll cause the brokers failed to "replay" these partition change since these topic/partitions didn't get created successfully previously. Even worse, after restarting the node, all the metadata records will replay again, and the same error happened again, cause the broker cannot start up successfully. The error and call stack is like this, basically, it complains the topic image can't be found {code:java} [2022-09-02 16:29:16,334] ERROR Encountered metadata loading fault: Error replaying metadata log record at offset 81 (org.apache.kafka.server.fault.LoggingFaultHandler) java.lang.NullPointerException at org.apache.kafka.image.TopicDelta.replay(TopicDelta.java:69) at org.apache.kafka.image.TopicsDelta.replay(TopicsDelta.java:91) at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:248) at org.apache.kafka.image.MetadataDelta.replay(MetadataDelta.java:186) at kafka.server.metadata.BrokerMetadataListener.$anonfun$loadBatches$3(BrokerMetadataListener.scala:239) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at kafka.server.metadata.BrokerMetadataListener.kafka$server$metadata$BrokerMetadataListener$$loadBatches(BrokerMetadataListener.scala:232) at kafka.server.metadata.BrokerMetadataListener$HandleCommitsEvent.run(BrokerMetadataListener.scala:113) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173) at java.base/java.lang.Thread.run(Thread.java:829) {code} [1] https://github.com/apache/kafka/blob/ef65b6e566ef69b2f9b58038c98a5993563d7a68/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L779-L804 [2] https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java#L1270 -- This message was sent by Atlassian Jira (v8.20.10#820010)