[
https://issues.apache.org/jira/browse/KAFKA-19327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111016#comment-18111016
]
Doyeon Kim commented on KAFKA-19327:
------------------------------------
KAFKA-20976 now allows operators to raise the controller batch limit, but doing
so weakens the bound intended to protect the controller from excessively large
operations. I’d like to work on batching MM2 topic creation so that large
multi-topic replication flows can operate without raising that limit. Would
this direction still be desirable?
> MM2 Source Connector Fails at Scale
> -----------------------------------
>
> Key: KAFKA-19327
> URL: https://issues.apache.org/jira/browse/KAFKA-19327
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.2
> Reporter: Ed Berezitsky
> Priority: Major
>
> We are running MM2 to replicate about 1000+ topics with about 25K partitions
> or something like that.
> {{The MM2 Source Connector fails to start with topics=.*}}
> The Exception:
>
> {code:java}
> [2025-05-23 16:51:06,501] WARN Could not create topic
> <just_one_of_hundreds_failed_topics>.
> (org.apache.kafka.connect.mirror.MirrorSourceConnector)
> java.util.concurrent.CompletionException:
> org.apache.kafka.common.errors.PolicyViolationException: Unable to perform
> excessively large batch operation.
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown
> Source)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown
> Source)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown
> Source)
> at
> org.apache.kafka.common.internals.KafkaCompletableFuture.kafkaCompleteExceptionally(KafkaCompletableFuture.java:49)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.completeExceptionally(KafkaFutureImpl.java:130)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient$1.handleResponse(KafkaAdminClient.java:1845)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.handleResponses(KafkaAdminClient.java:1397)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1550)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1473)
> at java.base/java.lang.Thread.run(Unknown Source) {code}
>
>
> MM2 Source Connector tries to create all the topics in a[ single
> request|https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L518].
> And fails...
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
