Hi list,
I've added a kafka node on our Hortonworks cluster and while executing the
reassign partition procedure something went wrong and I'm kind of stuck.
ZK nodes: RHTPINEC001, RHTPINEC004, RHTPINEC005
Kafka nodes: RHTPINEC008 (broker id = 1001), RHTPINEC007 (broker id = 1002,
this is the new broker)
Relevant Topics: poc_3b_syslog, poc_3a_syslog, uma_syslog_topic
Number of partitions: 6
Topics to move file:
{"topics":
[{"topic": "poc_3a_syslog"},{"topic": "poc_3b_syslog"},{"topic":
"uma_syslog_topic"}],
"version":1
}
Generated reassign output:
{"version":1,"partitions":[{"topic":"uma_syslog_topic","partition":2,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":0,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":2,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":1,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":0,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":1,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":0,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":4,"replicas":[1002]},{"topic":"poc_3b_syslog","partition":3,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":2,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":3,"replicas":[1002]},{"topic":"poc_3a_syslog","partition":5,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":3,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":1,"replicas":[1001]},{"topic":"poc_3a_syslog","partition":4,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":6,"replicas":[1001]},{"topic":"poc_3b_syslog","partition":5,"replicas":[1001]},{"topic":"uma_syslog_topic","partition":5,"replicas":[1002]},{"topic":"uma_syslog_topic","partition":4,"replicas":[1001]}]
Reassignment request:
$ ./kafka-reassign-partitions.sh --zookeeper 10.135.96.207:2181
--reassignment-json-file /tmp/move_me --broker-list "1001,1002" --execute
Verify:
$ ./kafka-reassign-partitions.sh --zookeeper 10.135.96.207:2181
--reassignment-json-file /tmp/move_me --broker-list "1001,1002" --verify
Status of partition reassignment:
Reassignment of partition [uma_syslog_topic,0] completed successfully
Reassignment of partition [poc_3b_syslog,4] completed successfully
Reassignment of partition [uma_syslog_topic,6] completed successfully
Reassignment of partition [poc_3a_syslog,0] completed successfully
Reassignment of partition [poc_3a_syslog,2] completed successfully
Reassignment of partition [poc_3b_syslog,0] completed successfully
Reassignment of partition [uma_syslog_topic,2] completed successfully
Reassignment of partition [uma_syslog_topic,1] is still in progress
Reassignment of partition [poc_3b_syslog,3] is still in progress
Reassignment of partition [uma_syslog_topic,4] completed successfully
Reassignment of partition [poc_3a_syslog,4] completed successfully
Reassignment of partition [poc_3b_syslog,5] is still in progress
Reassignment of partition [poc_3a_syslog,1] is still in progress
Reassignment of partition [poc_3a_syslog,3] is still in progress
Reassignment of partition [poc_3b_syslog,1] is still in progress
Reassignment of partition [uma_syslog_topic,3] is still in progress
Reassignment of partition [poc_3a_syslog,5] is still in progress
Reassignment of partition [poc_3b_syslog,2] completed successfully
Reassignment of partition [uma_syslog_topic,5] is still in progress
And I got stuck getting this to move.. Can't see anything moving on the
logs on the broker 1002.
After this I've cleared the /admin/reassign_partitions znode and I'm trying
to abort the procedure but can't go anywhere.
Right now the current status of the topics/partitions is:
Topic:poc_3a_syslog PartitionCount:6 ReplicationFactor:1
Configs:retention.ms=86400000
Topic: poc_3a_syslog Partition: 0 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3a_syslog Partition: 1 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: poc_3a_syslog Partition: 2 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3a_syslog Partition: 3 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: poc_3a_syslog Partition: 4 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3a_syslog Partition: 5 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic:poc_3b_syslog PartitionCount:6 ReplicationFactor:1
Configs:retention.ms=86400000
Topic: poc_3b_syslog Partition: 0 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3b_syslog Partition: 1 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: poc_3b_syslog Partition: 2 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3b_syslog Partition: 3 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: poc_3b_syslog Partition: 4 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: poc_3b_syslog Partition: 5 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic:uma_syslog_topic PartitionCount:7 ReplicationFactor:1
Configs:retention.ms=259200000
Topic: uma_syslog_topic Partition: 0 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: uma_syslog_topic Partition: 1 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: uma_syslog_topic Partition: 2 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: uma_syslog_topic Partition: 3 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: uma_syslog_topic Partition: 4 Leader: 1001 Replicas:
1001 Isr: 1001
Topic: uma_syslog_topic Partition: 5 Leader: 1001 Replicas:
1002,1001 Isr: 1001
Topic: uma_syslog_topic Partition: 6 Leader: 1001 Replicas:
1001 Isr: 1001
Any ideas to sort this out?
Thanks
BR
Alex
