[
https://issues.apache.org/jira/browse/KAFKA-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13730243#comment-13730243
]
Swapnil Ghike commented on KAFKA-999:
-------------------------------------
I think the right fix would be to undo any changes made to controllerContext
during shutdownBroker() if the shutdown attempt failed on the controller. If
the shutdown attempt failed, the controller's state should be equivalent to the
state it would be in if the shutdown attempt was never made.
In this case, it would mean removing broker 1 which is being shut down from
controllerContext.shuttingDownBrokerIds set if a shutdown attempt failed. Thus,
when the controller is sending out leaderAndIsr requests after broker 2 comes
up, it will include broker 1 in the alive brokers.
As far as dumbing down the broker is concerned to make it accept whatever
controller says, I don't have a strong feeling. As Neha described, it will also
resolve this ticket. However, doing only that and not fixing
KafkaController.shutdownBroker() will hide an underlying bug - the controller
had sent prematurely optimized information to broker 2 (the broker 1 was not
dead, but the controller thought it will soon be dead, so why include it in
alive leaders anyways).
I also think it's ok for the broker to be a bit intelligent. As such,
become-follower operation starts a thread that fetches from the leader. This
fetching is a peer-to-peer action, so I don't see anything wrong with a broker
being able to decide whether it wants to fetch or not.
What do you think?
> Controlled shutdown never succeeds until the broker is killed
> -------------------------------------------------------------
>
> Key: KAFKA-999
> URL: https://issues.apache.org/jira/browse/KAFKA-999
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Critical
>
> A race condition in the way leader and isr request is handled by the broker
> and controlled shutdown can lead to a situation where controlled shutdown can
> never succeed and the only way to bounce the broker is to kill it.
> The root cause is that broker uses a smart to avoid fetching from a leader
> that is not alive according to the controller. This leads to the broker
> aborting a become follower request. And in cases where replication factor is
> 2, the leader can never be transferred to a follower since it keeps rejecting
> the become follower request and stays out of the ISR. This causes controlled
> shutdown to fail forever
> One sequence of events that led to this bug is as follows -
> - Broker 2 is leader and controller
> - Broker 2 is bounced (uncontrolled shutdown)
> - Controller fails over
> - Controlled shutdown is invoked on broker 1
> - Controller starts leader election for partitions that broker 2 led
> - Controller sends become follower request with leader as broker 1 to broker
> 2. At the same time, it does not include broker 1 in alive broker list sent
> as part of leader and isr request
> - Broker 2 rejects leaderAndIsr request since leader is not in the list of
> alive brokers
> - Broker 1 fails to transfer leadership to broker 2 since broker 2 is not in
> ISR
> - Controlled shutdown can never succeed on broker 1
> Since controlled shutdown is a config option, if there are bugs in controlled
> shutdown, there is no option but to kill the broker
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
