Hello,
We're currently testing Mirrormaker 2.0 functionality for replication between
clusters. I have successfully run the Mirrormaker 2.0 script
(connect-mirror-maker.sh) using this config, replicating between two Kubernetes
Kafka broker instances:
Clusters = MC,DC
MC.bootstrap.servers=172.17.5.249:32010
DC.bootstrap.servers=172.17.5.254:32000
topics=test-topic
MC->DC.topics=test-topic
MC->DC.enabled=true
replication.factor=1
checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1
So the next step is to roll MirrorMaker 2.0 into our Java service architecture.
To do this I follow the test examples (by Ryanne Dolan) and create an instance
of the MirrorMakerConfig class as a container for the config and then
instantiate a MirrorMaker class itself:
//create the map for the mirrormaker 2.0 properties
HashMap<String,String> mm2Props = new HashMap<>();
//TODO: remove this manual props
mm2Props.put("clusters","MC,DC");
mm2Props.put("MC.bootstrap.servers","172.17.5.249:32010");
mm2Props.put("DC.bootstrap.servers","172.17.5.254:32000");
mm2Props.put("MC->DC.topics","test-topic");
mm2Props.put("MC->DC.enabled","true");
mm2Props.put("sync.topic.acls.enabled", "false");
mm2Props.put("emit.checkpoints.interval.seconds", "1");
mm2Props.put("emit.heartbeats.interval.seconds", "1");
mm2Props.put("refresh.topics.interval.seconds", "1");
mm2Props.put("refresh.groups.interval.seconds", "1");
mm2Props.put("checkpoints.topic.replication.factor", "1");
mm2Props.put("heartbeats.topic.replication.factor", "1");
mm2Props.put("offset-syncs.topic.replication.factor", "1");
mm2Props.put("config.storage.topic.replication.factor", "1");
mm2Props.put("offset.storage.topic.replication.factor", "1");
mm2Props.put("status.storage.topic.replication.factor", "1");
mm2Props.put("replication.factor", "1");
//instantiate our new Mirrormaker 2.0 config object
MirrorMakerConfig mm2Config = new MirrorMakerConfig(mm2Props);
//instantiate Mirrormaker 2.0 and run
this.mm2 = new MirrorMaker(mm2Config,null,Time.SYSTEM);
this.mm2.start();
Looking at the logs, this works up to the point where the internal MM2 topics
are to be created (they demand replication factor of 3 even though the config
specifies 1):
2020-02-20 13:40:42.518 INFO 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : AdminClientConfig values:
bootstrap.servers = [172.17.5.254:32000]
client.dns.lookup = default
client.id =
connections.max.idle.ms = 300000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 120000
retries = 5
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
2020-02-20 13:40:42.517 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'producer.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration 'group.id' was
supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'status.storage.topic' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'offset.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration 'header.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'consumer.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'config.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'offset.storage.topic' was supplied but isn't a known config.
2020-02-20 13:40:42.521 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration 'value.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'status.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration 'key.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.522 WARN 7 --- [erder-connect-1]
o.a.k.clients.admin.AdminClientConfig : The configuration
'admin.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1]
o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.3.0
2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1]
o.a.kafka.common.utils.AppInfoParser : Kafka commitId: fc1aaa116b661c8a
2020-02-20 13:40:42.522 INFO 7 --- [erder-connect-1]
o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1582206042522
2020-02-20 13:40:42.534 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'config.storage.topic' was supplied but isn't a known config.
2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'producer.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration 'group.id' was
supplied but isn't a known config.
2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'status.storage.topic' was supplied but isn't a known config.
2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'offset.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.539 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration 'header.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'consumer.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'config.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'offset.storage.topic' was supplied but isn't a known config.
2020-02-20 13:40:42.540 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration 'value.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'status.storage.topic.replication.factor' was supplied but isn't a known config.
2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration 'key.converter'
was supplied but isn't a known config.
2020-02-20 13:40:42.541 WARN 7 --- [erder-connect-2]
o.a.k.clients.admin.AdminClientConfig : The configuration
'admin.bootstrap.servers' was supplied but isn't a known config.
2020-02-20 13:40:42.541 INFO 7 --- [erder-connect-2]
o.a.kafka.common.utils.AppInfoParser : Kafka version: 2.3.0
2020-02-20 13:40:42.543 INFO 7 --- [erder-connect-2]
o.a.kafka.common.utils.AppInfoParser : Kafka commitId: fc1aaa116b661c8a
2020-02-20 13:40:42.543 INFO 7 --- [ scheduling-1]
o.a.kafka.connect.mirror.MirrorMaker : Kafka MirrorMaker started
2020-02-20 13:40:42.543 INFO 7 --- [erder-connect-2]
o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1582206042541
2020-02-20 13:40:42.572 ERROR 7 --- [erder-connect-1]
o.a.k.c.r.distributed.DistributedHerder : [Worker clientId=connect-1,
groupId=DC-mm2] Uncaught exception in herder work thread, exiting:
org.apache.kafka.connect.errors.ConnectException: Error while attempting to
create/find topic(s) 'mm2-offsets.DC.internal'
at
org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:262)
~[connect-runtime-2.3.0.jar!/:na]
at
org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99)
~[connect-runtime-2.3.0.jar!/:na]
at
org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:127)
~[connect-runtime-2.3.0.jar!/:na]
at
org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109)
~[connect-runtime-2.3.0.jar!/:na]
at org.apache.kafka.connect.runtime.Worker.start(Worker.java:184)
~[connect-runtime-2.3.0.jar!/:na]
at
org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:121)
~[connect-runtime-2.3.0.jar!/:na]
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:240)
~[connect-runtime-2.3.0.jar!/:na]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
~[na:na]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
~[na:na]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication
factor: 3 larger than available brokers: 1.
at
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
~[kafka-clients-2.3.0.jar!/:na]
at
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
~[kafka-clients-2.3.0.jar!/:na]
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
~[kafka-clients-2.3.0.jar!/:na]
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
~[kafka-clients-2.3.0.jar!/:na]
at
org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:229)
~[connect-runtime-2.3.0.jar!/:na]
... 11 common frames omitted
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 1.
Any help would be appreciated.
Thanks
Pirow Engelbrecht | Senior Design Engineer
Tel +27 12 678 9740 (ext. 9879) | Cell +27 63 148 3376
76 Regency Drive | Irene | Centurion | 0157<https://goo.gl/maps/v9ZbwjqpPyL2>
[cid:[email protected]]<https://etion.co.za>
Facebook<https://www.facebook.com/Etion-Limited-2194612947433812?_rdc=1&_rdr> |
YouTube<https://www.youtube.com/channel/UCUY-5oeACtLk2uTsEjZCU6A> |
LinkedIn<https://www.linkedin.com/company/etionltd> |
Twitter<https://twitter.com/Etionlimited> |
Instagram<https://www.instagram.com/Etionlimited/>
