[jira] [Commented] (KAFKA-8002) Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
[ https://issues.apache.org/jira/browse/KAFKA-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16780793#comment-16780793 ] ASF GitHub Bot commented on KAFKA-8002: --- bob-barrett commented on pull request #6346: KAFKA-8002: Replica reassignment to new log dir may not complete if f… URL: https://github.com/apache/kafka/pull/6346 …uture and current replicas segment files have different base offsets This patch fixes a bug in log dir reassignment where Partition.maybeReplaceCurrentWithFutureReplica would compare the entire LogEndOffsetMetadata of each replica to determine whether the reassignment has completed. If the active segments of both replicas have different base segments (for example, if the current replica had previously been cleaned and the future replica rolled segments at different points), the reassignment will never complete. The fix is to compare only the LogEndOffsetMetadata.messageOffset for each replica. Tested with a unit test that simulates the compacted current replica case. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Replica reassignment to new log dir may not complete if future and current > replicas segment files have different base offsets > - > > Key: KAFKA-8002 > URL: https://issues.apache.org/jira/browse/KAFKA-8002 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.1 >Reporter: Anna Povzner >Assignee: Bob Barrett >Priority: Critical > > Once future replica fetches log end offset, the intended logic is to finish > the move (and rename the future dir to current replica dir, etc). However, > the check in Partition.maybeReplaceCurrentWithFutureReplica compares the > whole LogOffsetMetadata vs. log end offset. The resulting behavior is that > the re-assignment will not finish for topic partitions that were cleaned/ > compacted such that base offset of the last segment is different for the > current and future replica. > The proposed fix is to compare only log end offsets of the current and future > replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8002) Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
[ https://issues.apache.org/jira/browse/KAFKA-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783081#comment-16783081 ] ASF GitHub Bot commented on KAFKA-8002: --- bob-barrett commented on pull request #6364: KAFKA-8002: Log dir reassignment stalls if future replica has differe… URL: https://github.com/apache/kafka/pull/6364 …nt segment base offset This patch fixes a bug in log dir reassignment where Partition.maybeReplaceCurrentWithFutureReplica would compare the entire LogEndOffsetMetadata of each replica to determine whether the reassignment has completed. If the active segments of both replicas have different base segments (for example, if the current replica had previously been cleaned and the future replica rolled segments at different points), the reassignment will never complete. The fix is to compare only the LogEndOffsetMetadata.messageOffset for each replica. Tested with a unit test that simulates the compacted current replica case. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Replica reassignment to new log dir may not complete if future and current > replicas segment files have different base offsets > - > > Key: KAFKA-8002 > URL: https://issues.apache.org/jira/browse/KAFKA-8002 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Bob Barrett >Priority: Critical > > Once future replica fetches log end offset, the intended logic is to finish > the move (and rename the future dir to current replica dir, etc). However, > the check in Partition.maybeReplaceCurrentWithFutureReplica compares the > whole LogOffsetMetadata vs. log end offset. The resulting behavior is that > the re-assignment will not finish for topic partitions that were cleaned/ > compacted such that base offset of the last segment is different for the > current and future replica. > The proposed fix is to compare only log end offsets of the current and future > replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8002) Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
[ https://issues.apache.org/jira/browse/KAFKA-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783790#comment-16783790 ] ASF GitHub Bot commented on KAFKA-8002: --- hachikuji commented on pull request #6364: KAFKA-8002: Log dir reassignment stalls if future replica has differe… URL: https://github.com/apache/kafka/pull/6364 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Replica reassignment to new log dir may not complete if future and current > replicas segment files have different base offsets > - > > Key: KAFKA-8002 > URL: https://issues.apache.org/jira/browse/KAFKA-8002 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0, 2.1.1 >Reporter: Anna Povzner >Assignee: Bob Barrett >Priority: Critical > > Once future replica fetches log end offset, the intended logic is to finish > the move (and rename the future dir to current replica dir, etc). However, > the check in Partition.maybeReplaceCurrentWithFutureReplica compares the > whole LogOffsetMetadata vs. log end offset. The resulting behavior is that > the re-assignment will not finish for topic partitions that were cleaned/ > compacted such that base offset of the last segment is different for the > current and future replica. > The proposed fix is to compare only log end offsets of the current and future > replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
