GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/3122
KAFKA-5310: reset ControllerContext during resignation
This ticket is all about ControllerContext initialization and teardown. The
key points are:
1. we should teardown ControllerContext during resignation instead of
waiting on election to fix it up. A heapdump shows that the former controller
keeps pretty much all of its ControllerContext state laying around.
2. we don't properly teardown/reset
ControllerContext.partitionsBeingReassigned. This can cause problems when the
former controller becomes re-elected as controller at a later point in time.
Suppose a partition assignment is initially R0. Now suppose a reassignment
R1 gets stuck during controller C0 and an admin tries to "undo" R1 (by deleting
/admin/partitions_reassigned, deleting /controller, and submitting another
reassignment specifying R0). The new controller C1 may succeed with R0. If the
controller moves back to C0, it will then reattempt R1 even though that
partition reassignment has been cleared from zookeeper prior to shifting the
controller back to C0. This results in the actual partition reassignment in
zookeeper being unexpectedly changed back to R1.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-5310
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3122.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3122
----
commit 256ff84b8263295f393b1c9eb0fdcdc35e5d67e8
Author: Onur Karaman <[email protected]>
Date: 2017-05-23T06:28:47Z
KAFKA-5310: reset ControllerContext during resignation
This ticket is all about ControllerContext initialization and teardown. The
key points are:
1. we should teardown ControllerContext during resignation instead of
waiting on election to fix it up. A heapdump shows that the former controller
keeps pretty much all of its ControllerContext state laying around.
2. we don't properly teardown/reset
ControllerContext.partitionsBeingReassigned. This can cause problems when the
former controller becomes re-elected as controller at a later point in time.
Suppose a partition assignment is initially R0. Now suppose a reassignment
R1 gets stuck during controller C0 and an admin tries to "undo" R1 (by deleting
/admin/partitions_reassigned, deleting /controller, and submitting another
reassignment specifying R0). The new controller C1 may succeed with R0. If the
controller moves back to C0, it will then reattempt R1 even though that
partition reassignment has been cleared from zookeeper prior to shifting the
controller back to C0. This results in the actual partition reassignment in
zookeeper being unexpectedly changed back to R1.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
