We have a situation on one of our clusters where when we run a partition reassignment using /usr/local/kafka/bin/kafka-reassign-partitions.sh we see the reassignment data published to the zookeeper in /admin/reassign_partitions and all partitions requests show in progress but the brokers do not appear to be seeing the "request" and thus the requested reassignments all remain stuck "in progress" and nothing is moving. This is a process we have used before on this cluster and use often across our clusters but this is the first time we have seen this. The cluster seems healthy in all other respects. We have validated that the json input we are passing to the reassign script is correct.
OS CentOS 6.7 Kafka v2.0.0-1 Zookeeper v3.4.6-3 Any idea on what could cause this or ways to troubleshoot further would be appreciated. Devin Kramer