[
https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15102589#comment-15102589
]
Jun Rao commented on KAFKA-3083:
--------------------------------
[~fpj], thanks for the clarification. If broker A shrinks the partition ISR in
ZK before step 2, then broker A's ZK session expires, then, broker sends the
shrunk ISR to broker C, two things can happen. (1) C has already received
requests from the new controller B. In this case, A's request will be rejected.
However, since the new controller B is re-elected after broker A shrinks the
ISR in ZK and the new controller read the latest ISR from ZK on initialization,
B will send the latest ISR to broker C. (2) C hasn't received any request from
the new controller B. In this case, A's request will be accepted. The new
controller B will later send the same ISR to broker B, but that's fine. So, in
either case, we are covered.
The problem in the description is really caused by broker A changing ZK after
its session expires. So, it seems the fix would be the following. If the
controller (say A) hits a ZK ConnectionLoss event while reading/writing to ZK,
it will pause the operation. Two possibilities can follow. In the case when the
controller A's ZK session expires, it will just ignore all the outstanding ZK
events. This guarantees that controller A can't touch ZK any more after a new
controller is elected (which has to happen after controller A's
SessionExpiration event). So, the new controller is guaranteed to read the
latest ZK data, act on this, and send the latest info to the broker. This would
avoid the issue in the description.
In the second case, controller A will get a SyncConnected event. In this case,
does controller A just resume from where it's left off? Or does it ignore all
outstanding events and re-read all subscribed ZK paths (since there could be
missing events between the connection loss event and the SyncConnected event)?
Finally, ZkClient actually hides the ZK ConnectionLoss event and only informs
the application when the ZK session expires. To pursue this, we will have to
access ZK directly.
> a soft failure in controller may leader a topic partition in an inconsistent
> state
> ----------------------------------------------------------------------------------
>
> Key: KAFKA-3083
> URL: https://issues.apache.org/jira/browse/KAFKA-3083
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.9.0.0
> Reporter: Jun Rao
> Assignee: Mayuresh Gharat
>
> The following sequence can happen.
> 1. Broker A is the controller and is in the middle of processing a broker
> change event. As part of this process, let's say it's about to shrink the isr
> of a partition.
> 2. Then broker A's session expires and broker B takes over as the new
> controller. Broker B sends the initial leaderAndIsr request to all brokers.
> 3. Broker A continues by shrinking the isr of the partition in ZK and sends
> the new leaderAndIsr request to the broker (say C) that leads the partition.
> Broker C will reject this leaderAndIsr since the request comes from a
> controller with an older epoch. Now we could be in a situation that Broker C
> thinks the isr has all replicas, but the isr stored in ZK is different.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
