Hi All, I'm trying to run MM2 in a Kafka connect cluster but I'm having some issues. I had successfully run MM2 in a dedicated MM cluster but now haven a requirement that means I have to run in a connect cluster in order to be able to support additional connectors in the same cluster. I'm just trying to replicate a topic from the source cluster to the target cluster at the moment. Firstly I'm looking to understand which configuration goes into the connect.distributed.properties file and which would go into the connector configuration. The documentation here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RunningMirrorMakerinaConnectcluster has the following example: { "name": "us-west-source", "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "source.cluster.alias": "us-west", "target.cluster.alias": "us-east", "source.cluster.bootstrap.servers": "us-west-host1:9091", "topics": ".*"} Is this all that's required? How will the MirrorSourceConnector know where to send the records i.e. do we not have to specify a target.cluster.bootstrap.servers list? It seems like if you don't specify a target.cluster.bootstrap.servers property the connector seems to fail on initialization of the target admin client since this property is required. Currenlty I have the following in a connect.distributed.properites file which I use to start one Kafka connect worker: bootstrap.servers=localhost9092group.id=mygroupid config.storage.replication.factor=1config.storage.topic=myconfigtopicoffset.storage.replication.factor=1offset.storage.topic=myoffsetstopicstatus.storage.replication.factor=1status.storage.topic=mystatustopic replication.factor=1rest.port=8082 key.converter=org.apache.kafka.connect.storage.StringConvertervalue.converter=org.apache.kafka.connect.storage.StringConverterinternal.key.converter=org.apache.kafka.connect.json.JsonConverterinternal.value.converter=org.apache.kafka.connect.json.JsonConverter
Then I create a connector with the following config: curl localhost:8082/connectors -X POST { "name": "replicate_testing1" "config": { "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector", "clusters": "source, target", "source.cluster.alias": "A", "target.cluster.alias": "B", "source.cluster.bootstrap.servers": "localhost:9092", "target.cluster.bootstrap.servers": "myserver1:9092,myserver2:9092,myserver3:9092", "A->B.enabled": "true", "A->B.topics": "testing1", "config.storage.replication.factor": "1", "config.storage.topic": "myconfigtopic", "offset.storage.replication.factor": "1", "offset.storage.topic": "myoffsetstopic", "status.storage.replication.factor": "1", "status.storage.topic": "mystatustopic", "checkpoints.topic.replication.factor": "1", "heartbeats.topic.replication.factor" "1", "offset-syncs.topic.replication.factor: "1", "target.security.protocol": "SASL_SSL", "target.sasl.kerberos.service.name": "kafka", "target.sasl.jaas.config": "com.sun.security.auth.module.lrb5LoginModule required useTicketCache=true" }} At the moment I'm just trying to replicate the "testing1" topic from the source to the target cluster. The connector starts successfully but I can't replicate from source to target. Firstly, if someone could confirm this is the correct way in terms of configuration to run MM in a connect cluster that would be great then I can describe the issues I'm seeing. Many Thanks, Jamie