Hi list, I've added a kafka node on our Hortonworks cluster and while executing the reassign partition procedure something went wrong and I'm kind of stuck.
ZK nodes: RHTPINEC001, RHTPINEC004, RHTPINEC005 Kafka nodes: RHTPINEC008 (broker id = 1001), RHTPINEC007 (broker id = 1002, this is the new broker) Relevant Topics: poc_3b_syslog, poc_3a_syslog, uma_syslog_topic Number of partitions: 6 Topics to move file: {"topics": [{"topic": "poc_3a_syslog"},{"topic": "poc_3b_syslog"},{"topic": "uma_syslog_topic"}], "version":1 } Generated reassign output: {"version":1,"partitions":[{"topic":"uma_syslog_topic","partition":2,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":0,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":2,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":1,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":0,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":1,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":0,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":4,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":3,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":2,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":3,"replicas":[1002]},{"topic":"poc_3a_syslog","partition":5,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":3,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":1,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":4,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":6,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":5,"replicas":[1001]},{"topic":"uma_syslog_topic","partition":5,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":4,"replicas":[1001]}] Reassignment request: $ ./kafka-reassign-partitions.sh --zookeeper 10.135.96.207:2181 --reassignment-json-file /tmp/move_me --broker-list "1001,1002" --execute Verify: $ ./kafka-reassign-partitions.sh --zookeeper 10.135.96.207:2181 --reassignment-json-file /tmp/move_me --broker-list "1001,1002" --verify Status of partition reassignment: Reassignment of partition [uma_syslog_topic,0] completed successfully Reassignment of partition [poc_3b_syslog,4] completed successfully Reassignment of partition [uma_syslog_topic,6] completed successfully Reassignment of partition [poc_3a_syslog,0] completed successfully Reassignment of partition [poc_3a_syslog,2] completed successfully Reassignment of partition [poc_3b_syslog,0] completed successfully Reassignment of partition [uma_syslog_topic,2] completed successfully Reassignment of partition [uma_syslog_topic,1] is still in progress Reassignment of partition [poc_3b_syslog,3] is still in progress Reassignment of partition [uma_syslog_topic,4] completed successfully Reassignment of partition [poc_3a_syslog,4] completed successfully Reassignment of partition [poc_3b_syslog,5] is still in progress Reassignment of partition [poc_3a_syslog,1] is still in progress Reassignment of partition [poc_3a_syslog,3] is still in progress Reassignment of partition [poc_3b_syslog,1] is still in progress Reassignment of partition [uma_syslog_topic,3] is still in progress Reassignment of partition [poc_3a_syslog,5] is still in progress Reassignment of partition [poc_3b_syslog,2] completed successfully Reassignment of partition [uma_syslog_topic,5] is still in progress And I got stuck getting this to move.. Can't see anything moving on the logs on the broker 1002. After this I've cleared the /admin/reassign_partitions znode and I'm trying to abort the procedure but can't go anywhere. Right now the current status of the topics/partitions is: Topic:poc_3a_syslog PartitionCount:6 ReplicationFactor:1 Configs:retention.ms=86400000 Topic: poc_3a_syslog Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3a_syslog Partition: 1 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: poc_3a_syslog Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3a_syslog Partition: 3 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: poc_3a_syslog Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3a_syslog Partition: 5 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic:poc_3b_syslog PartitionCount:6 ReplicationFactor:1 Configs:retention.ms=86400000 Topic: poc_3b_syslog Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3b_syslog Partition: 1 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: poc_3b_syslog Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3b_syslog Partition: 3 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: poc_3b_syslog Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: poc_3b_syslog Partition: 5 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic:uma_syslog_topic PartitionCount:7 ReplicationFactor:1 Configs:retention.ms=259200000 Topic: uma_syslog_topic Partition: 0 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: uma_syslog_topic Partition: 1 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: uma_syslog_topic Partition: 2 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: uma_syslog_topic Partition: 3 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: uma_syslog_topic Partition: 4 Leader: 1001 Replicas: 1001 Isr: 1001 Topic: uma_syslog_topic Partition: 5 Leader: 1001 Replicas: 1002,1001 Isr: 1001 Topic: uma_syslog_topic Partition: 6 Leader: 1001 Replicas: 1001 Isr: 1001 Any ideas to sort this out? Thanks BR Alex