[jira] [Commented] (KAFKA-8001) Fetch from future replica stalls when local replica becomes a leader
[ https://issues.apache.org/jira/browse/KAFKA-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16787263#comment-16787263 ] ASF GitHub Bot commented on KAFKA-8001: --- colinhicks commented on pull request #6395: KAFKA-8001: Reset future replica fetcher when local log becomes leader. URL: https://github.com/apache/kafka/pull/6395 Upon becoming leader, the local replica can fail with `FENCED_LEADER_EPOCH` on its next fetch from the the future replica. In this condition, fetching is stalled until the next leader change. This patch avoids this scenario by removing then re-adding from `replicaAlterLogDirsManager` partitions for which both the local log is leader and a future replica exists. The test case asserts that such a partition is reset for fetching with the new leader epoch. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fetch from future replica stalls when local replica becomes a leader > > > Key: KAFKA-8001 > URL: https://issues.apache.org/jira/browse/KAFKA-8001 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Colin Hicks >Priority: Critical > > With KIP-320, fetch from follower / future replica returns > FENCED_LEADER_EPOCH if current leader epoch in the request is lower than the > leader epoch known to the leader (or local replica in case of future replica > fetching). In case of future replica fetching from the local replica, if > local replica becomes the leader of the partition, the next fetch from future > replica fails with FENCED_LEADER_EPOCH and fetching from future replica is > stopped until the next leader change. > Proposed solution: on local replica leader change, future replica should > "become a follower" again, and go through the truncation phase. Or we could > optimize it, and just update partition state of the future replica to reflect > the updated current leader epoch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8001) Fetch from future replica stalls when local replica becomes a leader
[ https://issues.apache.org/jira/browse/KAFKA-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851436#comment-16851436 ] ASF GitHub Bot commented on KAFKA-8001: --- soondenana commented on pull request #6839: KAFKA-8001: Move log from replica into partition URL: https://github.com/apache/kafka/pull/6839 A partition object contain one or many replica objects. These replica objects in turn can have the "log" if the replica corresponds to the local node. All the code in Partition or ReplicaManager peek into replica object to fetch the log if they need to operate on that. As replica object can represent a local replica or a remote one, this lead to a bunch of "if-else" code in log fetch and offset update code. NOTE: In addition to a "log" that is in use during normal operation, if an alter log directory command is issued, we also create a future log object. This object catches up with local log and then we switch the log directory. So temporarily a Partition can have two local logs. Before this change both logs are inside replica objects. This change is an attempt to untangle this relationship. In particular it moves "log" from a replica object to Partition. So a partition contains a local log to which all writes go. And it maintains a list of replica for offset and "caught up time" data that it uses for replication protocol. The replica correspoding to Local node contains a log object, but the object is now read only and no code except Replica and test code use it. Every other part of code in Partion and ReplicaManger use the log object stored in Partition. This uncouples the replica-log relation and all the "if-else" code went away. Couple of more structural changes are made in this change: 1. Two subclasses of Replica are introduced: LocalReplica and RemoteReplica. This makes it clear what each replica stores and is capable of. 2. The "log" in Partition is also wrapped in a LogInfo wrapper, which encapuslates all the code that either operated on "log" or maintained state of it. Unit tests have been updated to take care of change in heirarchy. Tested by running multiple brokers and produced and consumed data. Also changed log directory back and forth to make sure that alter log directory use case works. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fetch from future replica stalls when local replica becomes a leader > > > Key: KAFKA-8001 > URL: https://issues.apache.org/jira/browse/KAFKA-8001 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Vikas Singh >Priority: Critical > > With KIP-320, fetch from follower / future replica returns > FENCED_LEADER_EPOCH if current leader epoch in the request is lower than the > leader epoch known to the leader (or local replica in case of future replica > fetching). In case of future replica fetching from the local replica, if > local replica becomes the leader of the partition, the next fetch from future > replica fails with FENCED_LEADER_EPOCH and fetching from future replica is > stopped until the next leader change. > Proposed solution: on local replica leader change, future replica should > "become a follower" again, and go through the truncation phase. Or we could > optimize it, and just update partition state of the future replica to reflect > the updated current leader epoch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8001) Fetch from future replica stalls when local replica becomes a leader
[ https://issues.apache.org/jira/browse/KAFKA-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851438#comment-16851438 ] ASF GitHub Bot commented on KAFKA-8001: --- soondenana commented on pull request #6839: KAFKA-8001: Move log from replica into partition URL: https://github.com/apache/kafka/pull/6839 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fetch from future replica stalls when local replica becomes a leader > > > Key: KAFKA-8001 > URL: https://issues.apache.org/jira/browse/KAFKA-8001 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Vikas Singh >Priority: Critical > > With KIP-320, fetch from follower / future replica returns > FENCED_LEADER_EPOCH if current leader epoch in the request is lower than the > leader epoch known to the leader (or local replica in case of future replica > fetching). In case of future replica fetching from the local replica, if > local replica becomes the leader of the partition, the next fetch from future > replica fails with FENCED_LEADER_EPOCH and fetching from future replica is > stopped until the next leader change. > Proposed solution: on local replica leader change, future replica should > "become a follower" again, and go through the truncation phase. Or we could > optimize it, and just update partition state of the future replica to reflect > the updated current leader epoch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8001) Fetch from future replica stalls when local replica becomes a leader
[ https://issues.apache.org/jira/browse/KAFKA-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851443#comment-16851443 ] ASF GitHub Bot commented on KAFKA-8001: --- soondenana commented on pull request #6841: KAFKA-8001: Move log from replica into partition URL: https://github.com/apache/kafka/pull/6841 A partition object contain one or many replica objects. These replica objects in turn can have the "log" if the replica corresponds to the local node. All the code in Partition or ReplicaManager peek into replica object to fetch the log if they need to operate on that. As replica object can represent a local replica or a remote one, this lead to a bunch of "if-else" code in log fetch and offset update code. NOTE: In addition to a "log" that is in use during normal operation, if an alter log directory command is issued, we also create a future log object. This object catches up with local log and then we switch the log directory. So temporarily a Partition can have two local logs. Before this change both logs are inside replica objects. This change is an attempt to untangle this relationship. In particular it moves "log" from a replica object to Partition. So a partition contains a local log to which all writes go. And it maintains a list of replica for offset and "caught up time" data that it uses for replication protocol. The replica correspoding to Local node contains a log object, but the object is now read only and no code except Replica and test code use it. Every other part of code in Partion and ReplicaManger use the log object stored in Partition. This uncouples the replica-log relation and all the "if-else" code went away. Couple of more structural changes are made in this change: 1. Two subclasses of Replica are introduced: LocalReplica and RemoteReplica. This makes it clear what each replica stores and is capable of. 2. The "log" in Partition is also wrapped in a LogInfo wrapper, which encapuslates all the code that either operated on "log" or maintained state of it. Unit tests have been updated to take care of change in heirarchy. Tested by running multiple brokers and produced and consumed data. Also changed log directory back and forth to make sure that alter log directory use case works. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fetch from future replica stalls when local replica becomes a leader > > > Key: KAFKA-8001 > URL: https://issues.apache.org/jira/browse/KAFKA-8001 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Vikas Singh >Priority: Critical > > With KIP-320, fetch from follower / future replica returns > FENCED_LEADER_EPOCH if current leader epoch in the request is lower than the > leader epoch known to the leader (or local replica in case of future replica > fetching). In case of future replica fetching from the local replica, if > local replica becomes the leader of the partition, the next fetch from future > replica fails with FENCED_LEADER_EPOCH and fetching from future replica is > stopped until the next leader change. > Proposed solution: on local replica leader change, future replica should > "become a follower" again, and go through the truncation phase. Or we could > optimize it, and just update partition state of the future replica to reflect > the updated current leader epoch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8001) Fetch from future replica stalls when local replica becomes a leader
[
https://issues.apache.org/jira/browse/KAFKA-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16868153#comment-16868153
]
Vikas Singh commented on KAFKA-8001:
Comment from Jason on PR that need to be taken care of:
{noformat}
I was discussing with @apovzner and we realized that there may be a few more
issues here. For a normal replica, whenever we observe an epoch bump, we go
through a reconciliation protocol to find a truncation point at which we know
it is safe to resume fetching. Basically it works like this:
Follower observes epoch bump and enters Truncating state.
Follower sends OffsetsForLeaderEpoch query to leader with the latest epoch from
its log.
Leader looks in its local log to find the largest epoch less than or equal to
the requested epoch and returns its end offset.
The follower will truncate to this offset and then possibly go back to 2. if
additional truncation is needed.
Once truncation is complete, the follower enters the Fetching state.
For a future replica, the protocol is basically the same, but rather than
sending OffsetsForLeaderEpoch to the leader, we use the state from the local
replica which may or may not be the leader of the bumped epoch. The basic
problem we realized is that this log reconciliation is not safe while the local
replica is in the Truncating state and we have nothing at the moment to
guarantee that it is not in that state. Basically we have to wait until it has
reached step 5 before the future replica can do its own truncation.
I suspect probably what we have to do to fix this problem is move the state of
the fetcher (i.e. Truncating|Fetching and the current epoch) out of
AbstractFetcherThread and into something which can be accessed by alter log dir
fetcher. The most obvious candidate seems like kafka.cluster.Replica.
Unfortunately, this may not be a small work, but I cannot think how to make
this protocol work unless we do it.{noformat}
> Fetch from future replica stalls when local replica becomes a leader
>
>
> Key: KAFKA-8001
> URL: https://issues.apache.org/jira/browse/KAFKA-8001
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.0, 2.1.1
>Reporter: Anna Povzner
>Assignee: Vikas Singh
>Priority: Critical
>
> With KIP-320, fetch from follower / future replica returns
> FENCED_LEADER_EPOCH if current leader epoch in the request is lower than the
> leader epoch known to the leader (or local replica in case of future replica
> fetching). In case of future replica fetching from the local replica, if
> local replica becomes the leader of the partition, the next fetch from future
> replica fails with FENCED_LEADER_EPOCH and fetching from future replica is
> stopped until the next leader change.
> Proposed solution: on local replica leader change, future replica should
> "become a follower" again, and go through the truncation phase. Or we could
> optimize it, and just update partition state of the future replica to reflect
> the updated current leader epoch.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
