[
https://issues.apache.org/jira/browse/KAFKA-21033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Schlansker updated KAFKA-21033:
--------------------------------------
Description:
The KAFKA-10357 refactoring ([PR
#20326|https://github.com/apache/kafka/pull/20326]) extracted the
internal-topic creation loop of {{InternalTopicManager.makeReady}} into a
{{createTopics}} helper and carried the end-of-round backoff block into the
per-topic result loop. This changed the retry behavior of {{makeReady}} in
three ways since 4.3.0:
# After every successfully created topic, the manager sleeps
{{retry.backoff.ms}} when other topics of the same batch are not yet iterated,
and logs a spurious "Topics ... could not be made ready" INFO line for them.
The first initialization of a topology with N internal topics pays
approximately (N-1) x {{retry.backoff.ms}} of sleep even when every create
succeeds immediately. With the 100 ms default, a topology of ours with 168
internal topics loses about 17 seconds before its first rebalance completes
(about 6 seconds on 4.2.1, about 23 seconds on 4.3.1, with 166 "could not be
made ready" retries). Broker-side creation is not the cause: a plain
{{AdminClient}} create-then-ready of 60 topics completes in about 1 second on
both versions.
# The misplaced block checks the deadline mid-round, so {{makeReady}} can throw
{{TimeoutException}} while the remaining creates of the round already succeeded.
# Rounds that create nothing lost their backoff entirely: when a topic is
marked for deletion or a describe transiently fails, {{tempUnknownTopics}} is
non-empty while {{topicsToCreate}} is empty, and {{makeReady}} re-describes in
a hot loop until the deadline (by default {{max.poll.interval.ms}} / 2). Before
PR #20326 the end-of-round backoff covered this path.
Production deployments pay the startup sleep once per topology creation. Test
suites and development environments that create fresh topologies pay it on
every run. Setting {{retry.backoff.ms=5}} on the streams client is a workaround
for item 1.
Suggested fix: move the backoff to the end of {{makeReady}}'s round, next to
its existing deadline check, restoring the pre-4.3 semantics of one backoff per
round. A PR with a regression test follows.
was:
KAFKA-10357 refactoring ([#20326|https://github.com/apache/kafka/pull/20326])
extracted the internal-topic creation
loop of InternalTopicManager.makeReady into a createTopics helper and
carried the end-of-round backoff block into the per-topic result loop.
Since 4.3.0, after every successfully created topic, the manager logs
"Topics ... could not be made ready" and sleeps retry.backoff.ms when
other topics of the same batch are not yet iterated. The first
initialization of a topology with N internal topics therefore pays
~(N-1) * retry.backoff.ms of pure sleep even when every create succeeds
immediately: with the 100 ms default, a topology with 168 internal
topics loses ~17 seconds before its first rebalance completes, plus one
spurious INFO line per topic. Before
[#20326|https://github.com/apache/kafka/pull/20326] the backoff ran once per
validation round.
> Regression in backoff algorithm during streams internal topic create
> --------------------------------------------------------------------
>
> Key: KAFKA-21033
> URL: https://issues.apache.org/jira/browse/KAFKA-21033
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0, 4.3.1
> Environment: Linux amd64
> Reporter: Steven Schlansker
> Assignee: Hrishi Baskaran
> Priority: Minor
>
> The KAFKA-10357 refactoring ([PR
> #20326|https://github.com/apache/kafka/pull/20326]) extracted the
> internal-topic creation loop of {{InternalTopicManager.makeReady}} into a
> {{createTopics}} helper and carried the end-of-round backoff block into the
> per-topic result loop. This changed the retry behavior of {{makeReady}} in
> three ways since 4.3.0:
> # After every successfully created topic, the manager sleeps
> {{retry.backoff.ms}} when other topics of the same batch are not yet
> iterated, and logs a spurious "Topics ... could not be made ready" INFO line
> for them. The first initialization of a topology with N internal topics pays
> approximately (N-1) x {{retry.backoff.ms}} of sleep even when every create
> succeeds immediately. With the 100 ms default, a topology of ours with 168
> internal topics loses about 17 seconds before its first rebalance completes
> (about 6 seconds on 4.2.1, about 23 seconds on 4.3.1, with 166 "could not be
> made ready" retries). Broker-side creation is not the cause: a plain
> {{AdminClient}} create-then-ready of 60 topics completes in about 1 second on
> both versions.
> # The misplaced block checks the deadline mid-round, so {{makeReady}} can
> throw {{TimeoutException}} while the remaining creates of the round already
> succeeded.
> # Rounds that create nothing lost their backoff entirely: when a topic is
> marked for deletion or a describe transiently fails, {{tempUnknownTopics}} is
> non-empty while {{topicsToCreate}} is empty, and {{makeReady}} re-describes
> in a hot loop until the deadline (by default {{max.poll.interval.ms}} / 2).
> Before PR #20326 the end-of-round backoff covered this path.
> Production deployments pay the startup sleep once per topology creation. Test
> suites and development environments that create fresh topologies pay it on
> every run. Setting {{retry.backoff.ms=5}} on the streams client is a
> workaround for item 1.
> Suggested fix: move the backoff to the end of {{makeReady}}'s round, next to
> its existing deadline check, restoring the pre-4.3 semantics of one backoff
> per round. A PR with a regression test follows.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
