[
https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840966#comment-17840966
]
Edoardo Comar commented on KAFKA-16622:
---------------------------------------
Activating DEBUG logging
```
[2024-04-25 21:20:10,856] DEBUG [MirrorCheckpointConnector|task-0]
translateDownstream(mygroup1,mytopic-0,13805): Skipped
(OffsetSync{topicPartition=mytopic-0, upstreamOffset=19999,
downstreamOffset=19999} is ahead of upstream consumer group 13805)
(org.apache.kafka.connect.mirror.OffsetSyncStore:125)
```
The checkpoint is not emitted because the topic-partition has been mirrorred
further than where the consumer group is,
so until the group catches up no checkpoints will be emitted.
Question for [~gregharris73]
this behavior would mean that any consumers in groups that are behind the log
end
that are switched from consuming from source cluster to the target cluster
to reprocess the entire partition ? They would have access to no translated
offsets.
> Mirromaker2 first Checkpoint not emitted until consumer group fully catches
> up once
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-16622
> URL: https://issues.apache.org/jira/browse/KAFKA-16622
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.0, 3.6.2, 3.8.0
> Reporter: Edoardo Comar
> Priority: Major
> Attachments: edo-connect-mirror-maker-sourcetarget.properties
>
>
> We observed an excessively delayed emission of the MM2 Checkpoint record.
> It only gets created when the source consumer reaches the end of a topic.
> This does not seem reasonable.
> In a very simple setup :
> Tested with a standalone single process MirrorMaker2 mirroring between two
> single-node kafka clusters(mirromaker config attached) with quick refresh
> intervals (eg 5 sec) and a small offset.lag.max (eg 10)
> create a single topic in the source cluster
> produce data to it (e.g. 10000 records)
> start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec
> between polls which commits after each poll
> watch the Checkpoint topic in the target cluster
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
> --topic source.checkpoints.internal \
> --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
> --from-beginning
> -> no record appears in the checkpoint topic until the consumer reaches the
> end of the topic (ie its consumer group lag gets down to 0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
