[
https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Harris reassigned KAFKA-14666:
-----------------------------------
Assignee: Greg Harris
> MM2 should translate consumer group offsets behind replication flow
> -------------------------------------------------------------------
>
> Key: KAFKA-14666
> URL: https://issues.apache.org/jira/browse/KAFKA-14666
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Affects Versions: 3.5.0
> Reporter: Greg Harris
> Assignee: Greg Harris
> Priority: Major
>
> MirrorMaker2 includes an offset translation feature which can translate the
> offsets for an upstream consumer group to a corresponding downstream consumer
> group. It does this by keeping a topic of offset-syncs to correlate upstream
> and downstream offsets, and translates any source offsets which are ahead of
> the replication flow.
> However, if a replication flow is closer to the end of a topic than the
> consumer group, then the offset translation feature will refuse to translate
> the offset for correctness reasons. This is because the MirrorCheckpointTask
> only keeps the latest offset correlation between source and target, it does
> not have sufficient information to translate older offsets.
> The workarounds for this issue are to:
> 1. Pause the replication flow occasionally to allow the source to get ahead
> of MM2
> 2. Increase the offset.lag.max to delay offset syncs, increasing the window
> for translation to happen. With the fix for KAFKA-12468, this will also
> increase the lag of applications that are ahead of the replication flow, so
> this is a tradeoff.
> Instead, the MirrorCheckpointTask should provide correct and best-effort
> translation for consumer groups behind the replication flow by keeping
> additional state, or re-reading the offset-syncs topic. This should be a
> substantial improvement for use-cases where applications have a higher
> latency to commit than the replication flow, or where applications are
> reading from the earliest offset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
