Hello, I'm still trying to get to the bottom of an issue we had previously, with an unclean shutdown during an upgrade to 0.8.2.1 (from 0.8.1.1). In that case, the controlled shutdown was interrupted, and the node was shutdown abruptly. This resulted in about 5 minutes of unavailability for most partitions. (I think that issue is related to the one reported by Thunder in the thread titled: "Problem with node after restart no partitions?").
Anyway, while investigating that, I've gotten side-tracked, trying understand what the expected behavior should be, if the controller node dies abruptly. To test this, I have a small test cluster (2 nodes, 100 partitions, each with replication factor 2, using 0.8.2.1). There are also a few test producer clients, some of them high volume.... I intentionally killed the controller node hard. I noticed that for 10 seconds, the second node spammed the logs for 10 seconds trying to fetch data for the partitions it was following on the node that was killed. Finally, after about 10 seconds, the second node elected itself the new controller, and things slowly recovered. Clients could not successfully produce to the affected partitions until the new controller was elected (and got failed meta-data requests trying to discover the new leader partition). I would have expected the cluster to recover more quickly if a node fails, if we have available replicas that can become leader and start receiving data. With just 100 partitions, I would have expected this recovery to happen very quickly. (Whereas in our previous issue, where it seemed to take 5 minutes, the longer duration there was probably related to a much larger number of partitions). Anyway, before I start filing Jira's and attaching log snippets, I'd like to understand what the expected behavior should be? If a controller (or really any node in the cluster) undergoes unclean shutdown, how should the cluster respond, in keeping replicas available (assuming all replicas were in ISR before the shutdown). How fast should controller and partition leader election happen in this case? Thanks, Jason
