Mithun Kumar created KAFKA-9884:
-----------------------------------
Summary: Unable to override some client properties in Mirror maker
2.0
Key: KAFKA-9884
URL: https://issues.apache.org/jira/browse/KAFKA-9884
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.4.1, 2.5.0, 2.4.0
Reporter: Mithun Kumar
Attachments: mm2.log
I have a two 3 node kafka clusters. MirrorMaker 2.0 is being run as a cluster
with bin/connect-mirror-maker.sh mm2.properties
I am trying to disable message duplication on replication by enabling
idempotence. I understand that EOS is marked as a future work in
[KIP-382|https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
however it should be possible by setting enable.idempotence = true and retries
> 0.
The <cluster_alias>.enable.idempotence = true takes effect, however overriding
the retries fails. I tried all 3 versions that provide MM2 2.4.0 , 2.4.1 and
2.5.0.
My mm2.properties config :
{noformat}
name = pri_to_bkp
connector.class = org.apache.kafka.connect.mirror.MirrorSourceConnector
topics = test-mm-topic-3
groups = .*
clusters = pri, bkp
source.cluster.alias = pri
target.cluster.alias = bkp
sasl.mechanism = GSSAPI
sasl.kerberos.service.name = kafka
security.protocol = SASL_PLAINTEXT
sasl.jaas.config = com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
keyTab="/etc/security/keytabs/user.keytab" \
principal="[email protected]";
pri.enable.idempotence = true
bkp.enable.idempotence = true
pri.retries = 2147483647
bkp.retries = 2147483647
pri.bootstrap.servers = SASL_PLAINTEXT://kafka1:9092,
SASL_PLAINTEXT://kafka2:9092, SASL_PLAINTEXT://kafka3:9092
bkp.bootstrap.servers = SASL_PLAINTEXT://bkp-kafka1:9092,
SASL_PLAINTEXT://bkp-kafka2:9092, SASL_PLAINTEXT://bkp-kafka3:9092
pri->bkp.enabled = true
pri->bkp.topics = "test-mm-topic-3"
{noformat}
The error leading to failure is:
{noformat}
[2020-04-17 15:46:26,525] ERROR [Worker clientId=connect-1, groupId=pri-mm2]
Uncaught exception in herder work thread, exiting:
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:297)
org.apache.kafka.common.config.ConfigException: Must set retries to non-zero
when using the idempotent producer.
at
org.apache.kafka.clients.producer.ProducerConfig.maybeOverrideAcksAndRetries(ProducerConfig.java:432)
at
org.apache.kafka.clients.producer.ProducerConfig.postProcessParsedConfig(ProducerConfig.java:400)
at
org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:110)
at
org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
at
org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:481)
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:326)
at
org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:270)
at
org.apache.kafka.connect.util.KafkaBasedLog.createProducer(KafkaBasedLog.java:248)
at
org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:129)
at
org.apache.kafka.connect.storage.KafkaStatusBackingStore.start(KafkaStatusBackingStore.java:199)
at
org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:124)
at
org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:284)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-04-17 15:46:29,089] INFO [Worker clientId=connect-1, groupId=pri-mm2]
Herder stopped
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:636)
[2020-04-17 15:46:29,089] INFO [Worker clientId=connect-2, groupId=bkp-mm2]
Herder stopping
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:616)
[2020-04-17 15:46:34,090] INFO [Worker clientId=connect-2, groupId=bkp-mm2]
Herder stopped
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:636)
[2020-04-17 15:46:34,090] INFO Kafka MirrorMaker stopped.
(org.apache.kafka.connect.mirror.MirrorMaker:191)
{noformat}
The complete log file is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
