We haven't been testing the ShutdownBroker command in 0.8.1 rigorously since in 0.8.1, one can do the controlled shutdown through the new config "controlled.shutdown.enable". Instead of running the ShutdownBroker command during the upgrade, you can also wait until under replicated partition count drops to 0 after each restart before moving to the next one.
Thanks, Jun On Thu, Mar 20, 2014 at 3:14 PM, Ryan Berdeen <rberd...@hubspot.com> wrote: > While upgrading from 0.8.0 to 0.8.1 in place, I observed some surprising > behavior using kafka.admin.ShutdownBroker. At the start, there were no > underreplicated partitions. After running > > bin/kafka-run-class.sh kafka.admin.ShutdownBroker --broker 10 ... > > Partitions that had replicas on broker 10 were under-replicated: > > bin/kafka-topics.sh --describe --under-replicated-partitions ... > Topic: analytics-activity Partition: 2 Leader: 12 Replicas: 12,10 Isr: > 12 > Topic: analytics-activity Partition: 6 Leader: 11 Replicas: 11,10 Isr: > 11 > Topic: analytics-activity Partition: 14 Leader: 14 Replicas: 14,10 Isr: > 14 > ... > > While restarting the broker process, many produce requests failed with > kafka.common.UnknownTopicOrPartitionException. > > After each broker restart, I used the preferred leader election tool for > all topics. Now, after finishing all of the broker restarts, the cluster > seems to be stuck in leader election. Running the tool fails with > "kafka.admin.AdminOperationException: Preferred replica leader election > currently in progress..." > > Are any of these known issues? Is there a safer way to shutdown and restart > brokers that does not cause producer failures and under-replicated > partitions? >
