[
https://issues.apache.org/jira/browse/KAFKA-19371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17985728#comment-17985728
]
fujian commented on KAFKA-19371:
--------------------------------
[~yangpoan] hi. can you help to take I look? thanks!
> When a broker restarts, it should not attempt to create the
> __remote_log_metadata topic if it already exists.
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-19371
> URL: https://issues.apache.org/jira/browse/KAFKA-19371
> Project: Kafka
> Issue Type: Bug
> Components: Tiered-Storage
> Affects Versions: 3.9.0, 4.0.0
> Reporter: fujian
> Priority: Major
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> *[Precondition]*
> Kafka cluster already enabled the remote storage feature based on inner
> topic's implementation. The core inner topic "__remote_log_metadata" already
> created.
>
> *[Steps]*
> 1. Restart one broker of the Kafka cluster.
> 2. Check the log and the code logic for the "__remote_log_metadata"s creating
> when broker restarting
>
> *[Expect result]*
> The broker shouldn't attempt to call API to create the topic due to that it
> already existed.
>
> *[Actual result]*
> The results are different which depend on the start process' duration for
> broker:
> *Case 1: Happy Path when restarting take a short time*
> [2025-06-03 22:35:11,648] INFO Topic __remote_log_metadata
> {color:#00875a}exists{color}. TopicId: 4CT2TTC-R6u7fNo_njYlDA, numPartitions:
> 50,
> *Case 2: Unhappy path 1 when restarting take some time*
> [2025-06-03 23:59:40,505] INFO Topic __remote_log_metadata{color:#de350b}
> does not exist{color}. Error: Timed out waiting for a node assignment. Call:
> listNodes
> [2025-06-04 00:00:36,938] INFO Topic [__remote_log_metadata]
> {color:#de350b}already exists{color}
> *Case 3: Unhappy path 2 when restarting take a long time.*
> [2025-06-03 21:57:21,151] INFO Topic __remote_log_metadata
> {color:#de350b}does not exist{color}. Error: {color:#de350b}Timed out
> waiting{color} for a node assignment. Call: {color:#de350b}listNodes {color}at
> [2025-06-03 21:58:21,153] ERROR Encountered error while creating
> __remote_log_metadata topic. java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.{color:#de350b}TimeoutException{color}: Timed
> out waiting for a node assignment. Call: {color:#de350b}createTopics {color}at
>
> From the log and current code. we can know that {color:#de350b}case 2 and
> case 3 both give the prompt "the topic does not exist" and try to call topic
> creating API. In actually. it is useless and contradict the fact that the
> topic already existed. Especially. the case 2's log prompt the topic existed
> and not existed at the same time.{color}
>
> *[Root Cause analyst]*
> After reviewing the related code
> (TopicBasedRemoteLogMetadataManager#doesTopicExist). It is one
> {color:#de350b}wrong implement{color} to judge one topic existed or not.
> So let me create this [PR |#19899 · apache/kafka]to fix this minor bug. Thanks
>
> FYI:
> Why we got the timeout exception?
> It is normal case based on the fact:
> When restarting broker. The connection to query/create topic in
> "TopicBasedRemoteLogMetadataManager#initializeResources"will fail until the
> broker's self get ready.
> [2025-06-03 23:21:20,752] WARN [AdminClient clientId=adminclient-1]
> Connection to node -1 ([10.20.1.125:9559)|https://10-20-1-125/] could not be
> established. Node may not be available.
> [2025-06-03 23:21:21,282] INFO [BrokerServer id=2] Transition from STARTING
> to STARTED (kafka.server.BrokerServer)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
