C0urante commented on PR #14314: URL: https://github.com/apache/kafka/pull/14314#issuecomment-1701967620
Thanks @jolshan, great questions! > What was the original rationale for https://issues.apache.org/jira/browse/KAFKA-12339? That ticket was filed in response to a change in behavior caused by https://github.com/apache/kafka/pull/9780, which altered some parts of Kafka Connect to use an admin client instead of a consumer to get the end offsets of internal topics. Kafka Connect already had support for automatically creating these topics on startup, so the switch to using an admin client to list end offsets for internal topics caused some workers to fail on startup after trying to list end offsets for a topic they'd just created. This is because consumers retried when listing end offsets for topics that were unknown to the broker, whereas admin clients did not. > My understanding is that when a topic is first created, there is sometimes a delay to the metadata propagation. It seems like connect was encountering this issue and rather than adding code to retry they wanted it to be done automatically. This is correct, although in https://github.com/apache/kafka/pull/11797, we reverted the automatic retry logic for `Admin::listOffsets` and added it to `TopicAdmin::retryEndOffsets` (a Connect-specific class and method) instead. > I can see the issue with the compatibility break, but I also see the issue with the metadata propagation. Is there anyway to configure whether we prefer to retry on such an error? There's no user-facing way to handle this scenario in isolation. Users can obviously tweak generic retry settings for their Kafka clients, but that has other, much-wider implications. The `tolerateUnknownTopics` field is limited to the internal API we use for the admin client and is just a bit of a hack to allow us to fail fast in a specific scenario (topic is unknown to the broker) in some contexts (trying to list topic offsets using an admin client) but retry in others (everything else using the `PartitionLeaderStrategy`). > Or is the argument this should always be left to the client to handle the manual retries? Yes, that's the idea--at least for `Admin::listOffsets`. Every other API remains unaffected, so, e.g., `Producer::send` will still retry on attempts to produce to an unknown topic partition (which is its own source of headaches, but that's out of scope for this PR). To summarize: if a topic has just been created, it's possible that `Admin::listOffsets` will throw a spurious `UnknownTopicOrPartitionException` with this change, which is less desirable than an automatic retry. However, if the topic genuinely doesn't exist, `Admin::listOffsets` throwing that exception immediately instead of retrying until the retry timeout expires is better behavior. And, if the topic has just been created, it's debatable whether it's fair to retry for the full retry timeout anyways, instead of a shorter period. This was one of the points in the [final rationale](https://issues.apache.org/jira/browse/KAFKA-12879?focusedCommentId=17503891&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17503891) for reverting this behavioral change originally: > For example, a likely root cause of [KAFKA-12339](https://issues.apache.org/jira/browse/KAFKA-12339) was a Connect worker instantiates its KafkaConfigBackingStore (and other internal topic stores), which creates a KafkaBasedLog that as part of start() creates the topic if it doesn't exist and then immediately tries to read the offsets. That reading of offsets can fail if the metadata for the newly created topic hasn't been propagated to all of the brokers. We can solve this particular root cause easily by retrying the reading of offsets within the KafkaBasedLog's start() method, and since topic metadata should be propagated relatively quickly, we don't need to retry for that long – and most of the time we'd probably successfully retry within a few retries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
