[
https://issues.apache.org/jira/browse/KAFKA-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hrishi Baskaran reassigned KAFKA-21033:
---------------------------------------
Assignee: (was: Hrishi Baskaran)
> Regression in backoff algorithm during streams internal topic create
> --------------------------------------------------------------------
>
> Key: KAFKA-21033
> URL: https://issues.apache.org/jira/browse/KAFKA-21033
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0, 4.3.1
> Environment: Linux amd64
> Reporter: Steven Schlansker
> Priority: Minor
>
> The KAFKA-10357 refactoring ([PR
> #20326|https://github.com/apache/kafka/pull/20326]) extracted the
> internal-topic creation loop of {{InternalTopicManager.makeReady}} into a
> {{createTopics}} helper and carried the end-of-round backoff block into the
> per-topic result loop. This changed the retry behavior of {{makeReady}} in
> three ways since 4.3.0:
> # After every successfully created topic, the manager sleeps
> {{retry.backoff.ms}} when other topics of the same batch are not yet
> iterated, and logs a spurious "Topics ... could not be made ready" INFO line
> for them. The first initialization of a topology with N internal topics pays
> approximately (N-1) x {{retry.backoff.ms}} of sleep even when every create
> succeeds immediately. With the 100 ms default, a topology of ours with 168
> internal topics loses about 17 seconds before its first rebalance completes
> (about 6 seconds on 4.2.1, about 23 seconds on 4.3.1, with 166 "could not be
> made ready" retries). Broker-side creation is not the cause: a plain
> {{AdminClient}} create-then-ready of 60 topics completes in about 1 second on
> both versions.
> # The misplaced block checks the deadline mid-round, so {{makeReady}} can
> throw {{TimeoutException}} while the remaining creates of the round already
> succeeded.
> # Rounds that create nothing lost their backoff entirely: when a topic is
> marked for deletion or a describe transiently fails, {{tempUnknownTopics}} is
> non-empty while {{topicsToCreate}} is empty, and {{makeReady}} re-describes
> in a hot loop until the deadline (by default {{max.poll.interval.ms}} / 2).
> Before PR #20326 the end-of-round backoff covered this path.
> Production deployments pay the startup sleep once per topology creation. Test
> suites and development environments that create fresh topologies pay it on
> every run. Setting {{retry.backoff.ms=5}} on the streams client is a
> workaround for item 1.
> Suggested fix: move the backoff to the end of {{makeReady}}'s round, next to
> its existing deadline check, restoring the pre-4.3 semantics of one backoff
> per round. A PR with a regression test follows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
