[
https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703339#comment-17703339
]
Greg Harris edited comment on KAFKA-14666 at 3/21/23 7:08 PM:
--------------------------------------------------------------
I proposed a tactical fix for this in
[https://github.com/apache/kafka/pull/13429] which provides variable-accuracy
translation between the most recent restart of MM2 and the end of the
replicated topic, similar to the existing behavior pre KAFKA-12468.
This uses strategy (2) from above, but limited to syncs read during a single
task lifetime to get monotonicity without re-reading the checkpoint topic.
A separate improvement can be considered to allow for translation of offsets
prior to the latest restart of MM2, or increasing the accuracy of the
translation with new configurations.
was (Author: gharris1727):
I proposed a tactical fix for this in
[https://github.com/apache/kafka/pull/13429] which provides variable-accuracy
translation between the most recent restart of MM2 and the end of the
replicated topic, similar to the existing behavior pre KAFKA-12468.
A separate improvement can be considered to allow for translation of offsets
prior to the latest restart of MM2, or increasing the accuracy of the
translation with new configurations.
> MM2 should translate consumer group offsets behind replication flow
> -------------------------------------------------------------------
>
> Key: KAFKA-14666
> URL: https://issues.apache.org/jira/browse/KAFKA-14666
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Affects Versions: 3.5.0
> Reporter: Greg Harris
> Assignee: Greg Harris
> Priority: Major
>
> MirrorMaker2 includes an offset translation feature which can translate the
> offsets for an upstream consumer group to a corresponding downstream consumer
> group. It does this by keeping a topic of offset-syncs to correlate upstream
> and downstream offsets, and translates any source offsets which are ahead of
> the replication flow.
> However, if a replication flow is closer to the end of a topic than the
> consumer group, then the offset translation feature will refuse to translate
> the offset for correctness reasons. This is because the MirrorCheckpointTask
> only keeps the latest offset correlation between source and target, it does
> not have sufficient information to translate older offsets.
> The workarounds for this issue are to:
> 1. Pause the replication flow occasionally to allow the source to get ahead
> of MM2
> 2. Increase the offset.lag.max to delay offset syncs, increasing the window
> for translation to happen. With the fix for KAFKA-12468, this will also
> increase the lag of applications that are ahead of the replication flow, so
> this is a tradeoff.
> Instead, the MirrorCheckpointTask should provide correct and best-effort
> translation for consumer groups behind the replication flow by keeping
> additional state, or re-reading the offset-syncs topic. This should be a
> substantial improvement for use-cases where applications have a higher
> latency to commit than the replication flow, or where applications are
> reading from the earliest offset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
