Edoardo Comar created KAFKA-16622:
-------------------------------------
Summary: Mirromaker2 first Checkpoint not emitted until consumer
group fully catches up once
Key: KAFKA-16622
URL: https://issues.apache.org/jira/browse/KAFKA-16622
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.6.2, 3.7.0, 3.8.0
Reporter: Edoardo Comar
Attachments: edo-connect-mirror-maker-sourcetarget.properties
We observed an excessively delayed emission of the MM2 Checkpoint record.
It only gets created when the source consumer reaches the end of a topic. This
does not seem reasonable.
In a very simple setup :
Tested with a standalone single process MirrorMaker2 mirroring between two
single-node kafka clusters(mirromaker config attached) with quick refresh
intervals (eg 5 sec) and a small offset.lag.max (eg 10)
create a single topic in the source cluster
produce data to it (e.g. 10000 records)
start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec between
polls which commits after each poll
watch the Checkpoint topic in the target cluster
bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
--topic source.checkpoints.internal \
--formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
--from-beginning
-> no record appears in the checkpoint topic until the consumer reaches the end
of the topic (ie its consumer group lag gets down to 0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
