[ https://issues.apache.org/jira/browse/KAFKA-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nandini Anagondi resolved KAFKA-10440. -------------------------------------- Resolution: Invalid > MM2 using Kconnect Framework Creates topics recursively on source cluster > ------------------------------------------------------------------------- > > Key: KAFKA-10440 > URL: https://issues.apache.org/jira/browse/KAFKA-10440 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect > Affects Versions: 2.5.0 > Reporter: Nandini Anagondi > Priority: Major > Fix For: 2.5.0 > > Attachments: MirrorSourceConnector-config.json, > MirrorSourceConnector-override-config.json, list_of_topics.txt > > > I found an issue while running MM2 using Kconnect Framework. Based on the > configuration provided in this > [https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorConnectorConfig.java] > *source : localhost:9092, Alias : A* > *target : localhost:9093, Alias : B* > *Issue Faced:* > Topics from the source cluster are created at target cluster but instead of > producing data of source topics to target it is producing to source cluster > topics and those topics are getting created at source because the > *allowAutoTopicCreation=true.* > *How the issue arrises:* > Kconnect is running on source cluster and connectors are configured from > A->B. Connect framework uses connect-distributed.properties to initialise > producerConfig, consumerConfig. So, producer and consumer are initialised > with source bootstrap servers i.e., with > _localhost:9092("source.cluster.producer.bootstrap.servers" : > "localhost:9092”, "source.cluster.consumer.bootstrap.servers" : > "localhost:9092”)_. > This configuration works perfectly fine with all traditional connectors > because they produce and consume from that cluster itself. But MM2 connectors > are contrary to this. Let's see this example > Ideally MM2 should work like this > A B > test-mm2 A.test-mm2 > Indeed it is working like this and creating topics recursively. Please refer > to the attachments. > A B > A.test-mm2 A.A.test-mm2 > test-mm2 A.test-mm2 > This applies to all the topics where data need to be copied. Suppose if it > doesn't have any data it won't cause this issue. > *Fixing the issue:* > MirrorSourceConnector should produce to destination cluster topics while > copying data. This issue can be fixed by overriding bootstrap.servers. To do > this connector.client.config.override.policy should be set to "All" in > connect-distributed.properties & _(producer.override.bootstrap.servers, > consumer.override.bootstrap.servers)_ should match with > _target.cluster.bootstrap.servers_ and this need to be provided in connectors > payload.So, connectors now produce the data to target cluster. > A B > test-mm2 A.test-mm2 > List of the topics in both cases: [^list_of_topics.txt] > You can find the config in both cases here: > [^MirrorSourceConnector-config.json] - Config that matches with > MirrorConnectorConfig > [^MirrorSourceConnector-override-config.json]Config that works. > To imitate the scenario: Start the kconnect at source cluster, Configure > connector with MirrorSourceConnector-config.json, create a topic and produce > data to it. Check the list of topics in 5-10seconds. -- This message was sent by Atlassian Jira (v8.20.10#820010)