[
https://issues.apache.org/jira/browse/KAFKA-9089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16959373#comment-16959373
]
ASF GitHub Bot commented on KAFKA-9089:
---------------------------------------
hachikuji commented on pull request #7562: KAFKA-9089; Reassignment should be
resilient to unexpected errors
URL: https://github.com/apache/kafka/pull/7562
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reassignment should be resilient to unexpected errors
> -----------------------------------------------------
>
> Key: KAFKA-9089
> URL: https://issues.apache.org/jira/browse/KAFKA-9089
> Project: Kafka
> Issue Type: Bug
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
> Priority: Major
> Fix For: 2.4.0
>
>
> Reassignment changes typically involve both an update to the assignment state
> in zookeeper and an update to the in-memory representation of that state (in
> the ControllerContext). We can run into trouble when these states get
> inconsistent with each other, so the reassignment logic attempts to follow
> some rules to reduce the impact from this:
> * When creating a new reassignment, we update the state in zookeeper first
> before updating memory. Until the reassignment is known to be persisted, we
> do not begin executing any reassignment logic.
> * When completing a reassignment, all of the completion steps are executed
> before the state is updated in zookeeper. In the event of a failure, the new
> controller can retry reassignment completion.
> However, the current logic does not follow these rules strictly which can
> lead to state inconsistencies in the case of an unexpected error.
> # When we override or cancel an existing assignment, we currently use an
> intermediate assignment state which is only reflected in memory. It is
> basically a mix of the previous assignment state and the overlapping parts of
> the new reassignment. The purpose of this is to shutdown unneeded replicas
> from the existing reassignment. Since the intermediate state is not
> persisted, a controller failure will revert to the old reassignment. Any
> exception which does not cause a controller failure will result in state
> divergence.
> # The target replicas of a reassignment are represented both in the existing
> assignment (PartitionReplicaAssignment) and in a separate context object
> (ReassignedPartitionContext). The reassignment context is updated before a
> reassignment has been accepted and persisted. The intent is to remove this
> context object in the event of a submission failure, but an unexpected error
> will leave it around.
> We can make reassignment more resilient to unexpected errors by using
> consistent update invariants. Specifically we can remove the intermediate
> assignment state and enforce the invariant that any active reassignment must
> be persisted before being reflected in memory. Additionally, we can make the
> assignment state the source of truth for the target replicas and eliminate
> the possibility of inconsistency. Doing so simplifies the reassignment logic
> and makes it more resilient.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
