I've confirmed that the same thing happens even if it's not the controller
that's killed hard. Also, in several trials, it took between 10-30 seconds
to recover.
Jason
On Wed, Apr 8, 2015 at 1:31 PM, Jason Rosenberg j...@squareup.com wrote:
Hello,
I'm still trying to get to the bottom of an issue we had previously, with
an unclean shutdown during an upgrade to 0.8.2.1 (from 0.8.1.1). In that
case, the controlled shutdown was interrupted, and the node was shutdown
abruptly. This resulted in about 5 minutes of unavailability for most
partitions. (I think that issue is related to the one reported by Thunder
in the thread titled: Problem with node after restart no partitions?).
Anyway, while investigating that, I've gotten side-tracked, trying
understand what the expected behavior should be, if the controller node
dies abruptly.
To test this, I have a small test cluster (2 nodes, 100 partitions, each
with replication factor 2, using 0.8.2.1). There are also a few test
producer clients, some of them high volume
I intentionally killed the controller node hard. I noticed that for 10
seconds, the second node spammed the logs for 10 seconds trying to fetch
data for the partitions it was following on the node that was killed.
Finally, after about 10 seconds, the second node elected itself the new
controller, and things slowly recovered.
Clients could not successfully produce to the affected partitions until
the new controller was elected (and got failed meta-data requests trying to
discover the new leader partition).
I would have expected the cluster to recover more quickly if a node fails,
if we have available replicas that can become leader and start receiving
data. With just 100 partitions, I would have expected this recovery to
happen very quickly. (Whereas in our previous issue, where it seemed to
take 5 minutes, the longer duration there was probably related to a much
larger number of partitions).
Anyway, before I start filing Jira's and attaching log snippets, I'd like
to understand what the expected behavior should be?
If a controller (or really any node in the cluster) undergoes unclean
shutdown, how should the cluster respond, in keeping replicas available
(assuming all replicas were in ISR before the shutdown). How fast should
controller and partition leader election happen in this case?
Thanks,
Jason
