Hi,
I am using MM2 version 3.1.0 for a uni-directional mirroring of
source-cluster -> target-cluster.
I am confused with the values found in the
source-cluster.checkpoints.internal topic at the target cluster when
the source's earliest offset is not 0.
In my test setup, I prepared a topic samples.greetings with 10 records
at the source cluster and using consumer-group greeting10 to consume
those 10 records at the source cluster first. After that, I deleted 5
records from this topic to move the earliest offset to 5 and
subsequently produced 3 additional records to this topic.
Next, I verified consumer group greetings's state at the source
cluster using kafka-consumer-group.sh, which showed the offset 10 and
the lag 3, as expected.
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
greetings samples.greetings 0 10 13
3 - - -
Next, I started the MM2 to mirror this topic to the target cluster.
After the mirroring is complete, I used
RemoteClusterUtils.translateOffsets to get the translated offset of
this consumer group. This returned 10 instead of 5. Then, I also
looked into topic source-cluster.checkpoints.internal at the target
cluster. I was expecting to find records with {upstreamOffset:10,
downstreamOffset:5}, but I found the records with {upstreamOffset:10,
downstreamOffset:10}.
And consumer group greetings's state at the target cluster was shown
by kafka-consumer-group.sh as
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
greetings samples.greetings 0 10 8
-2 - - -
I thought the checkpoint topic would have the translated
downstreamOffset of 5 and the consumer offset at the target cluster
was translated to 5. This would have allowed the consumer to simply
switch to the target cluster. Therefore, I am wondering if this is the
expected behavior or something went wrong.
To be able to simply switch the consumer to the target cluster, I
could reset the current offset at the target cluster to the correct
offset by comparing the LAG values at the source and target clusters
and setting the current offset by subtracting the difference (assuming
the mirroring is stopped so that the updated offset won't be
overwritten) from the current offset value. But this is a bit of pain.
Thank you for any help.
Regards, aki
