[
https://issues.apache.org/jira/browse/KAFKA-19165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Schlansker resolved KAFKA-19165.
---------------------------------------
Resolution: Invalid
> PartitionLeaderStrategy has very high error rate during topic initialization
> ----------------------------------------------------------------------------
>
> Key: KAFKA-19165
> URL: https://issues.apache.org/jira/browse/KAFKA-19165
> Project: Kafka
> Issue Type: Improvement
> Components: clients, streams
> Affects Versions: 3.9.0
> Reporter: Steven Schlansker
> Priority: Major
>
> We implemented a Kafka Streams app.
> Some integration tests run a Kafka broker and then connect the Streams app to
> it, to ensure our application functions as desired.
> When initializing each test case, all the Streams topics must be created.
> This is expected as each integration test expects to run its own "copy" of
> the app (different `application.id`)
> The application is *very* chatty about this process.
> We see hundreds of thousands of errors like:
> {code:java}
> 2025-04-16T17:17:33.841Z [kafka-admin-client-thread |
> search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> ERROR o.a.k.c.a.i.PartitionLeaderStrategy - [AdminClient
> clientId=search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> Received unknown topic error for topic
> search-indexing-2025-04-16-r67a-notification-group-eoc-merge-changelog
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> 2025-04-16T17:17:33.841Z [kafka-admin-client-thread |
> search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> ERROR o.a.k.c.a.i.PartitionLeaderStrategy - [AdminClient
> clientId=search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> Received unknown topic error for topic
> search-indexing-2025-04-16-r67a-current-time-store-changelog
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> 2025-04-16T17:17:33.841Z [kafka-admin-client-thread |
> search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> ERROR o.a.k.c.a.i.PartitionLeaderStrategy - [AdminClient
> clientId=search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> Received unknown topic error for topic
> search-indexing-2025-04-16-r67a-notification-group-eoc-merge-changelog
> org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server
> does not host this topic-partition.
> 2025-04-16T17:17:33.841Z [kafka-admin-client-thread |
> search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> ERROR o.a.k.c.a.i.PartitionLeaderStrategy - [AdminClient
> clientId=search-indexing-2025-04-16-r67a-a4975cf6-622a-4f36-93bf-990df7b34a4e-admin]
> Received unknown topic error for topic
> search-indexing-2025-04-16-r67a-current-time-store-changelog {code}
> For a single topic, we get > 6000 errors in just a few seconds. The log file
> ends up being many megabytes of this, to the point where some less-powerful
> text editors struggle to even render the file.
> Having so many errors that are in fact expected and non-actionable harms
> observability of the Kafka Streams platform. Would it be sensible to suppress
> "expected" exceptions of this type, as topics are being created? Or at least
> rate-limit it, for example printing ever 10 seconds "Waiting for topics [...,
> ...] to be created for 30s..."
> I also wonder if the admin client should rate limit how often it pings the
> broker, to reduce broker load in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
