Hello Gérald,
I have the exact same problem. Mirrormaker 2.0 Javadocs documentation is only
slated for release 2.5.0 (see https://issues.apache.org/jira/browse/KAFKA-8930).
I am also prototyping Mirrormaker 2.0 and I have successfully run the
Mirrormaker 2.0 scripts (connect-mirror-maker.sh) using this config,
replicating between two Kubernetes Kafka broker instances:
Clusters = MC,DC
MC.bootstrap.servers=172.17.5.249:32010
DC.bootstrap.servers=172.17.5.254:32000
topics=test-topic
MC->DC.topics=test-topic
MC->DC.enabled=true
replication.factor=1
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1
But instantiating the Java MirrorMaker 2.0 classes, the Mirrormaker 2.0 can't
create its own topics (it fails with the same replication factor 3 vs 1 issue),
even though the properties are set in the MirrorMakerConfig class:
HashMap<String,String> mm2Props = new HashMap<>();
mm2Props.put("clusters","MC,DC");
mm2Props.put("MC.bootstrap.servers","172.17.5.249:32010");
mm2Props.put("DC.bootstrap.servers","172.17.5.254:32000");
mm2Props.put("MC->DC.topics","test-topic");
mm2Props.put("MC->DC.enabled","true");
mm2Props.put("sync.topic.acls.enabled", "false");
mm2Props.put("emit.checkpoints.interval.seconds", "1");
mm2Props.put("emit.heartbeats.interval.seconds", "1");
mm2Props.put("refresh.topics.interval.seconds", "60");
mm2Props.put("refresh.groups.interval.seconds", "60");
mm2Props.put("checkpoints.topic.replication.factor", "1");
mm2Props.put("heartbeats.topic.replication.factor", "1");
mm2Props.put("offset-syncs.topic.replication.factor", "1");
mm2Props.put("config.storage.topic.replication.factor", "1");
mm2Props.put("offset.storage.topic.replication.factor", "1");
mm2Props.put("status.storage.topic.replication.factor", "1");
mm2Props.put("replication.factor", "1");
//instantiate our new Mirrormaker 2.0 config object
MirrorMakerConfig mm2Config = new MirrorMakerConfig(mm2Props);
The internal topics created for this single topic (test-topic) is:
On MC:
mm2-configs.DC.internal
mm2-offset-syncs.DC.internal
mm2-offsets.DC.internal
mm3-status.DC.internal
On DC:
MC.checkpoints.internal
MC.heartbeats
mm2-configs.MC.internal
mm2-offsets.MC.internal
mm2-status.MC.internal
Thanks
Pirow
-----Original Message-----
From: Gérald Quintana <[email protected]>
Sent: Thursday, 20 February 2020 23:40
To: [email protected]
Subject: Mirror Maker 2 internal topics
Hello,
I am prototyping Kafka replication with Mirror Maker 2. At the beginning, I had
hard times with
org.apache.kafka.connect.errors.ConnectException: Error while attempting to
create/find topic(s) 'mm2-offsets.dst.internal'...
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 1.
...
Caused by:
org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 1.
I managed to find the settings to put in mm2.properties
(offset.storage.replication.factor=1), and discovered 6 internal topics.
But it raises broader questions: What internal topics should I expect with MM2?
What are their purpose? On which side of the replicator (source, target or
both) are they? How can I tune them (topic name, replication factor...)? And
more generally where can I find a mm2.properties settings documentation?
Thanks for your help,
Gérald
