Marcel Renders created KAFKA-9746:
-------------------------------------
Summary: MM2 produces messages in the wrong cluster
Key: KAFKA-9746
URL: https://issues.apache.org/jira/browse/KAFKA-9746
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.4.1, 2.4.0
Reporter: Marcel Renders
When the MirrorSourceConnector (MM2) is used to get messages from a remote
Kafka cluster to the Kafka cluster that is connected to Kafka Connect, MM2 is
working fine. However, when the connected cluster is being copied to the remote
Kafka cluster, the messages are sent to the connected cluster, causing an
infinite loop.
Note that the topics are in fact created as expected in the remote Kafka
cluster and no errors are thrown except flush failures due to the infinite loop.
In short:
* Connected target -> unconnected source: OK
* Connected source -> unconnected target: NOK
*Desired situation*
The connected Kafka instance should only be used by Kafka Connect to store
configs, offsets and status. It should be possible to mirror one Kafka cluster
to another while neither are connected to the Kafka Connect instance.
Typically the Kafka connectors are either sink or source, but the MM2 is
different because the connector settings require both target and source
configurations.
*Reproducing the issue*
This issue occurs with the following configuration, where localhost:9092 is
used for Kafka Connect:
{code:java}
{
"name": "MM_TEST",
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"errors.log.enable": "true",
"enable.auto.commit": "true",
"auto.commit.interval.ms": "20000",
"refresh.topics.interval.seconds": "60",
"replication.factor": "1",
"sync.topic.acls.enabled": "false",
"checkpoints.topic.replication.factor": "1",
"heartbeats.topic.replication.factor": "1",
"offset-syncs.topic.replication.factor": "1",
"replication.policy.separator": "",
"topics": "'test1'",
"tasks.max": "1",
"key.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter",
"source.cluster.bootstrap.servers": "localhost:9092",
"source.cluster.alias": "",
"source.cluster.security.protocol": "PLAINTEXT",
"source.cluster.group.id": "consumer_mm_test",
"target.cluster.bootstrap.servers": "localhost:9093",
"target.cluster.alias": "",
"target.cluster.security.protocol": "PLAINTEXT"
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
