[
https://issues.apache.org/jira/browse/KAFKA-15168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kamal Chandraprakash updated KAFKA-15168:
-----------------------------------------
Description:
For a partition p0 and a given leader epoch, the remote log manager can upload
duplicate segments due to leader change. RemoteLogMetadata cache should handle
the duplicate segments which may affect the follower and consumer.
(eg)
L0 uploaded the segment PZyYVdsJQWeBAdBDPqkcVA at t0 for offset range 10 - 90
L1 uploads the segment L5Ufv71IToiZYKgsluzcyA at t1 for offset range 5 - 100
In the
[RemoteLogLeaderEpochState|https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogLeaderEpochState.java]
class, the {{offsetToId}} is a navigable map. It sorts the entries by
start-offset which keeps the state as:
{code:java}
(5 - 100) -> L5Ufv71IToiZYKgsluzcyA (T1)
(10 - 90) -> PZyYVdsJQWeBAdBDPqkcVA (T0){code}
For a fetch request with fetch-offset as 92, the RemoteLogLeaderEpochState will
return the segment PZyYVdsJQWeBAdBDPqkcVA instead of L5Ufv71IToiZYKgsluzcyA,
which doesn't have the respective offset and throws error back to the caller.
was:
For a partition p0 and a given leader epoch, the remote log manager can upload
duplicate segments due to leader change. RemoteLogMetadata cache should handle
the duplicate segments which may affect the follower and consumer.
(eg)
L0 uploaded the segment PZyYVdsJQWeBAdBDPqkcVA at t0 for offset range 10 - 90
L1 uploads the segment L5Ufv71IToiZYKgsluzcyA at t1 for offset range 5 - 100
In the RemoteLogLeaderEpochState class, the {{offsetToId}} is a navigable map.
It sorts the entries by start-offset which keeps the state as:
{code:java}
(5 - 100) -> L5Ufv71IToiZYKgsluzcyA (T1)
(10 - 90) -> PZyYVdsJQWeBAdBDPqkcVA (T0){code}
For a fetch request with fetch-offset as 92, the RemoteLogLeaderEpochState will
return the segment PZyYVdsJQWeBAdBDPqkcVA instead of L5Ufv71IToiZYKgsluzcyA,
which doesn't have the respective offset and throws error back to the caller.
> Handle overlapping remote log segments in RemoteLogMetadata cache
> -----------------------------------------------------------------
>
> Key: KAFKA-15168
> URL: https://issues.apache.org/jira/browse/KAFKA-15168
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Kamal Chandraprakash
> Assignee: Kamal Chandraprakash
> Priority: Major
>
> For a partition p0 and a given leader epoch, the remote log manager can
> upload duplicate segments due to leader change. RemoteLogMetadata cache
> should handle the duplicate segments which may affect the follower and
> consumer.
> (eg)
> L0 uploaded the segment PZyYVdsJQWeBAdBDPqkcVA at t0 for offset range 10 - 90
> L1 uploads the segment L5Ufv71IToiZYKgsluzcyA at t1 for offset range 5 - 100
> In the
> [RemoteLogLeaderEpochState|https://github.com/apache/kafka/blob/trunk/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogLeaderEpochState.java]
> class, the {{offsetToId}} is a navigable map. It sorts the entries by
> start-offset which keeps the state as:
> {code:java}
> (5 - 100) -> L5Ufv71IToiZYKgsluzcyA (T1)
> (10 - 90) -> PZyYVdsJQWeBAdBDPqkcVA (T0){code}
> For a fetch request with fetch-offset as 92, the RemoteLogLeaderEpochState
> will return the segment PZyYVdsJQWeBAdBDPqkcVA instead of
> L5Ufv71IToiZYKgsluzcyA, which doesn't have the respective offset and throws
> error back to the caller.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
