Luke Chen created KAFKA-16814: --------------------------------- Summary: KRaft broker cannot startup when `partition.metadata` is missing Key: KAFKA-16814 URL: https://issues.apache.org/jira/browse/KAFKA-16814 Project: Kafka Issue Type: Bug Affects Versions: 3.7.0 Reporter: Luke Chen
When starting up kafka logManager, we'll check stray replicas to avoid some corner cases. But this check might cause broker unable to startup if `partition.metadata` is missing because when startup kafka, we load log from file, and the topicId of the log is coming from `partition.metadata` file. So, if `partition.metadata` is missing, the topicId will be None, and the `LogManager#isStrayKraftReplica` will fail with no topicID error. The `partition.metadata` missing could be some storage failure, or another possible path is unclean shutdown after topic is created in the replica, but before data is flushed into `partition.metadata` file. This is possible because we do the flush, it's done async [here|https://github.com/apache/kafka/blob/5552f5c26df4eb07b2d6ee218e4a29e4ca790d5c/core/src/main/scala/kafka/log/UnifiedLog.scala#L229]. {code:java} ERROR Encountered fatal fault: Error starting LogManager (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) java.lang.RuntimeException: The log dir Log(dir=/tmp/kraft-broker-logs/quickstart-events-0, topic=quickstart-events, partition=0, highWatermark=0, lastStableOffset=0, logStartOffset=0, logEndOffset=0) does not have a topic ID, which is not allowed when running in KRaft mode. at kafka.log.LogManager$.$anonfun$isStrayKraftReplica$1(LogManager.scala:1609) at scala.Option.getOrElse(Option.scala:201) at kafka.log.LogManager$.isStrayKraftReplica(LogManager.scala:1608) at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1(BrokerMetadataPublisher.scala:294) at kafka.server.metadata.BrokerMetadataPublisher.$anonfun$initializeManagers$1$adapted(BrokerMetadataPublisher.scala:294) at kafka.log.LogManager.loadLog(LogManager.scala:359) at kafka.log.LogManager.$anonfun$loadLogs$15(LogManager.scala:493) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1623) {code} Because if we don't do the isStrayKraftReplica check, the topicID and the `partition.metadata` will get recovered after getting topic partition update and becoming leader or follower later. I'm proposing we skip the `isStrayKraftReplica` check if topicID is None, instead of throwing exception to terminate the kafka. `isStrayKraftReplica` check is just for a corner case only, it should be fine IMO. -- This message was sent by Atlassian Jira (v8.20.10#820010)