Dmitry Vsekhvalnov created KAFKA-6047:
-----------------------------------------
Summary: Allow retries configuration for InternalTopicManager
Key: KAFKA-6047
URL: https://issues.apache.org/jira/browse/KAFKA-6047
Project: Kafka
Issue Type: Improvement
Components: streams
Affects Versions: 0.11.0.0
Reporter: Dmitry Vsekhvalnov
Priority: Minor
There is hardcoded number of retries when kafka-streams attempts to create
internal topics.
*InternalTopicManager.MAX_TOPIC_READY_TRY*
Which is not resilient in some scenarios. Consider setup where replication
factor for internal streams topics == number of brokers. (RF=3 and x3 kafka
brokers). When any of brokers dies kafka-streams can shutdown before broker is
resurrected with approximate log:
{code}
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via StreamsConfig
parameter "replication.factor" or add more brokers to your cluster. Retry #2]
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via StreamsConfig
parameter "replication.factor" or add more brokers to your cluster. Retry #3]
[WARN ] [org.apache.kafka.streams.processor.internals.InternalTopicManager]
[Could not create internal topics: Found only 2 brokers, but replication
factor is 3. Decrease replication factor for internal topics via StreamsConfig
parameter "replication.factor" or add more brokers to your cluster. Retry #4]
[INFO ] [org.apache.kafka.streams.processor.internals.StreamThread]
[stream-thread [Shutting down]
{code}
Would be nice if kafka-streams provides configuration for InternalTopicManager
retries and ideally for retry backoff strategy. To have possibility for
resilience tuning. Possibly can re-use corresponding producer configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
