[
https://issues.apache.org/jira/browse/KAFKA-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-532:
--------------------------------
Attachment: kafka-532-v3.patch
>> So we don't need to rewrite leaderAndIsr during controller failover. When
>> sending leaderAndIsr requests, we just need to send the controller epoc
>> stored in the leaderAndIsr path.
That still doesn't solve the problem. Here is the problem with re-publishing
the previous controller's decision with an older epoch-
Let's say that controller with epoch 1 elected leaders for partitions p1 and
p2. Over time, controller moves and epoch increments to 2. Let's say this
controller reassigns partition p2 and re-elects leader for p2 as part of that.
Now, controller with epoch 2 goes into a GC pause and controller moves to epoch
3. This new controller re-publishes leader and isr decision made by controller
epoch 1 as well as epoch 2. Now, the same broker will receive leader and isr
requests for 2 different epochs. So, it will reject the requests sent by epoch
1 since it already received requests with a higher epoch (2). Ignoring state
change requests is dangerous and can lead to a situation where some partitions
are offline.
So, the controller should use its own epoch while sending state change requests
to the brokers. Precisely, here is how it will work -
1. The leader and isr path has the epoch of the controller that made the new
leader/isr decision. The leader and isr path will be conditionally updated. If
the conditional update fails, the controller re-reads the controller epoch
value. If the epoch has changed, it knows that another controller has taken
over and it aborts the state change operation.
2. Whenever a controller sends a state change request to a broker
(leaderAndIsr/stopReplica), it tags the request with its own epoch. In other
words, it certifies that decision to be current and correct.
3. Each broker maintains the last known highest controller epoch. The broker
will reject any state change request that is tagged with a controller epoch
value lower than what it knows.
Since multiple controllers during leader election is tricky, lets dive into
some details -
When the controller changes leader/isr state, it first
1. conditionally updates the zookeeper path for that partition
2. sends the leader/isr request to the brokers
If the controller goes into soft failure before step #1, a new controller will
get elected and it will notice that partition is offline, elect the leader and
send the leader/isr request to the broker with the new epoch. When the failed
controller falls out of the soft failure, it will try to update the zk path,
but will fail and abort the operation.
If the controller goes into soft failure between steps 1 & 2, a new controller
will get elected and will just resend failed controller's leader/isr decision
to the broker using its own controller epoch. When the failed controller wakes
up, it might try to send the leader/isr decision to the broker, but the broker
will reject that request since it already knows a higher controller epoch
These changes were covered in patch v2, uploading v3 after rebasing.
> Multiple controllers can co-exist during soft failures
> ------------------------------------------------------
>
> Key: KAFKA-532
> URL: https://issues.apache.org/jira/browse/KAFKA-532
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: bugs
> Attachments: kafka-532-v1.patch, kafka-532-v2.patch,
> kafka-532-v3.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> If the current controller experiences an intermittent soft failure (GC pause)
> in the middle of leader election or partition reassignment, a new controller
> might get elected and start communicating new state change decisions to the
> brokers. After recovering from the soft failure, the old controller might
> continue sending some stale state change decisions to the brokers, resulting
> in unexpected failures. We need to introduce a controller generation id that
> increments with controller election. The brokers should reject any state
> change requests by a controller with an older generation id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
