Daan Bosch created KAFKA-12937:
----------------------------------
Summary: Mirrormaker2 can only start from the beginning of a topic
Key: KAFKA-12937
URL: https://issues.apache.org/jira/browse/KAFKA-12937
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 2.8.0
Environment: Dockerized environment
Reporter: Daan Bosch
*Goal*:
I want to replace Mirrormaker version 1 with Mirrormaker2.
To do this I want to:
start Mirrormaker2 from the latest offset of every topic
stop Mirrormaker1
There should only be a couple of double messages.
What happened:
Mirrormaker2 starts replicating from the start of all topics
*How to reproduce:*
Start two Kafka clusters, A and B
I produce 3000 messages to cluster A on a topic (TOPIC1)
Kafka Connect is running and connected to cluster B
Start a Mirrormaker2 task in connect to replicate messages from cluster A. Wit
the option:
consumer auto.offset.reset to latest
Produce another 3000 messages to cluster A on the same topic (TOPIC1)
*Expected result:*
Cluster B will only contain the messages produced the second time (3000 in
total) on TOPIC1
Actual result:
The mirror picks up all messages from the start (6000 in total) and replicates
them to cluster B
*Additional logs:*
Logs from the consumer of the Mirrormaker task:
mirrormaker.log:7581:mirrormaker_1 | [2021-06-11
09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-8
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7583:mirrormaker_1 | [2021-06-11
09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-3
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7585:mirrormaker_1 | [2021-06-11
09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-2
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7587:mirrormaker_1 | [2021-06-11
09:31:40,403] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-1
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7589:mirrormaker_1 | [2021-06-11
09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-0
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7591:mirrormaker_1 | [2021-06-11
09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-7
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7593:mirrormaker_1 | [2021-06-11
09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-6
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7595:mirrormaker_1 | [2021-06-11
09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-5
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)
mirrormaker.log:7597:mirrormaker_1 | [2021-06-11
09:31:40,404] INFO [Consumer clientId=consumer-null-4, groupId=null] Seeking to
offset 0 for partition perf-test-4
(org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are
trying to seek to a position and thus overriding the latest offset
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
