[
https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-13659:
----------------------------------
Description: MirrorCheckpointTask uses OffsetSyncStore, and does not check
whether OffsetSyncStore managed to read to the "end" of the offset-syncs topic.
OffsetSyncStore should fetch the endoffset of the topic at startup, and set a
flag when it finally reaches the endoffset in consumption.
MirrorCheckpointTask.poll should wait for this flag to be true before doing any
in-memory updates and group offset management. (was: - MirrorCheckpointTask
uses OffsetSyncStore, and does not check whether OffsetSyncStore managed to
read to the "end" of the offset-syncs topic. OffsetSyncStore should fetch the
endoffset of the topic at startup, and set a flag when it finally reaches the
endoffset in consumption. MirrorCheckpointTask.poll should wait for this flag
to be true before doing any in-memory updates and group offset management.
- MirrorCheckpointTask can create checkpoints which point into the "future" -
meaning it sometimes translates consumer offsets in a way that the target
offset is greater than the endoffset of the replica topic partition.
MirrorCheckpointTask should fetch the endoffsets of the affected topics, and
make sure that it does not try to set the consumer offset to anything higher
than the endoffset.)
> MM2 should read all offset syncs at start up and should not set consumer
> offset higher than the end offset
> ----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Kanalas Vidor
> Assignee: Kanalas Vidor
> Priority: Major
>
> MirrorCheckpointTask uses OffsetSyncStore, and does not check whether
> OffsetSyncStore managed to read to the "end" of the offset-syncs topic.
> OffsetSyncStore should fetch the endoffset of the topic at startup, and set a
> flag when it finally reaches the endoffset in consumption.
> MirrorCheckpointTask.poll should wait for this flag to be true before doing
> any in-memory updates and group offset management.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
