[
https://issues.apache.org/jira/browse/KAFKA-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113672#comment-18113672
]
Ramu commented on KAFKA-21022:
------------------------------
h3. Root Cause Analysis
I traced the failure path to the following:
{noformat}
BrokerMetadataPublisher.scala:338
|
+--> logManager.recoverAbandonedFutureLogs(...)
|
+--> LogManager.findAbandonedFutureLogs(...)
|
+--> futureLog.topicId().orElseThrow(...)
|
+--> RuntimeException
{noformat}
*Findings:*
# *Entry point:* During KRaft metadata processing, {{BrokerMetadataPublisher}}
invokes {{recoverAbandonedFutureLogs()}} to handle abandoned future logs.
# *Missing topic ID:* {{findAbandonedFutureLogs()}} iterates over future logs.
If a future log is missing {{partition.metadata}}, {{futureLog.topicId()}}
returns {{Optional.empty()}}.
# *Unhandled exception:* The code unconditionally throws a {{RuntimeException}}
when the topic ID is missing, causing the broker failure.
# *Related fix:* KAFKA-16814 addressed a similar missing-{{partition.metadata}}
scenario in the standard KRaft stray-replica handling path
({{isStrayKraftReplica}}), but the future-log recovery path still has this
unconditional exception.
This appears to be an inconsistency between the standard KRaft log handling and
future-log recovery paths.
*Proposed direction:* The future-log recovery path should likely handle a
missing topic ID without causing a broker failure, consistently with the
behavior introduced by KAFKA-16814.
Before proceeding with the implementation, I would appreciate maintainer
feedback on whether this RCA and proposed direction are correct, and what the
preferred handling should be for a future log without a topic ID.
> Broker Crash During ZK-to-KRaft Migration Due to Missing Topic ID in Empty
> Log Directory
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-21022
> URL: https://issues.apache.org/jira/browse/KAFKA-21022
> Project: Kafka
> Issue Type: Bug
> Components: kraft, migration
> Affects Versions: 3.9.1
> Reporter: Youssef Ismail
> Assignee: Ramu
> Priority: Critical
>
> While migrating a cluster from ZooKeeper to KRaft, during migration mode
> brokers can crash after receiving a full {{LeaderAndIsr}} request from the
> KRaft controller with the following error:
> {code:java}
> java.lang.RuntimeException: The log dir ... does not have a topic ID,{code}
>
> During migration, Kafka checks the broker’s existing log directories through
> {{{}findStrayReplicas{}}}. Unlike the normal KRaft startup path, this method
> throws an exception when a topic ID is missing which is not allowed when
> running in KRaft mode:
> * [Migration path in
> ReplicaManager|https://github.com/apache/kafka/blob/3.9/core/src/main/scala/kafka/server/ReplicaManager.scala#L1932-L1941]
> * [Exception in
> findStrayReplicas|https://github.com/apache/kafka/blob/3.9/core/src/main/scala/kafka/log/LogManager.scala#L1667-L1671]
> OSS addressed a similar problem in KAFKA-16814 and [PR
> #16165|https://github.com/apache/kafka/pull/16165]. That fix treats a log
> without a topic ID as stray during normal KRaft startup but the
> migration-specific {{findStrayReplicas}} path still throws.
> I recovered by stopping the affected broker, removing the empty log
> directory, and restarting it in migration mode.
> I propose updating {{findStrayReplicas}} to handle a missing topic ID the
> same way as the normal KRaft path: treat the empty log as stray instead of
> crashing the broker.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
