Re: One Partition missing a node in ISR

Sebastian Schmitz Sun, 29 Sep 2019 18:56:16 -0700

Hello,

I just ran the kafka-reassign-partitions with --generate to create thejson and then with --execute to run it.Now when checking with --verify I can see that the 4 partitions (it nowchanged from only one partitions not having all in ISR to 12 not beingall in ISR) are successful, but the others are still in progress....That status remains:

root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./kafka-topics.sh--bootstrap-server localhost:9092 --topic my_topic --describeTopic:my_topic PartitionCount:16 ReplicationFactor:3Configs:segment.bytes=1073741824,message.format.version=2.3-IV1,retention.bytes=1073741824 Topic: my_topic Partition: 0 Leader: 1 Replicas: 2,3,1Isr: 1 Topic: my_topic Partition: 1 Leader: 1 Replicas: 3,1,2Isr: 1 Topic: my_topic Partition: 2 Leader: 1 Replicas: 1,2,3Isr: 1 Topic: my_topic Partition: 3 Leader: 1 Replicas: 2,1,3Isr: 1 Topic: my_topic Partition: 4 Leader: 1 Replicas: 3,2,1Isr: 1 Topic: my_topic Partition: 5 Leader: 1 Replicas: 1,3,2Isr: 1 Topic: my_topic Partition: 6 Leader: 1 Replicas: 2,3,1Isr: 1 Topic: my_topic Partition: 7 Leader: 1 Replicas: 3,1,2Isr: 1 Topic: my_topic Partition: 8 Leader: 1 Replicas: 1,2,3Isr: 1 Topic: my_topic Partition: 9 Leader: 1 Replicas: 2,1,3Isr: 1 Topic: my_topic Partition: 10 Leader: 1 Replicas: 3,2,1Isr: 1 Topic: my_topic Partition: 11 Leader: 1 Replicas: 1,3,2Isr: 1 Topic: my_topic Partition: 12 Leader: 1 Replicas: 2,3,1Isr: 1,3,2 Topic: my_topic Partition: 13 Leader: 2 Replicas: 3,1,2Isr: 1,3,2 Topic: my_topic Partition: 14 Leader: 3 Replicas: 1,2,3Isr: 1,3,2 Topic: my_topic Partition: 15 Leader: 1 Replicas: 2,1,3Isr: 1,3,2

root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin#./kafka-reassign-partitions.sh --bootstrap-server localhost:9092--zookeeper atazkafkp01.aucklandtransport.govt.nz:2181--reassignment-json-file move2.json --execute

Current partition replica assignment

{"version":1,"partitions":[{"topic":"my_topic","partition":7,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":14,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":8,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":4,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":13,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":15,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":6,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":11,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":12,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":10,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":9,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"my_topic","partition":5,"replicas":[3,2,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

Status of partition reassignment:
Reassignment of partition my_topic-7 is still in progress
Reassignment of partition my_topic-14 completed successfully
Reassignment of partition my_topic-8 is still in progress
Reassignment of partition my_topic-4 is still in progress
Reassignment of partition my_topic-3 is still in progress
Reassignment of partition my_topic-13 completed successfully
Reassignment of partition my_topic-1 is still in progress
Reassignment of partition my_topic-15 completed successfully
Reassignment of partition my_topic-6 is still in progress
Reassignment of partition my_topic-11 is still in progress
Reassignment of partition my_topic-0 is still in progress
Reassignment of partition my_topic-12 completed successfully
Reassignment of partition my_topic-10 is still in progress
Reassignment of partition my_topic-2 is still in progress
Reassignment of partition my_topic-9 is still in progress
Reassignment of partition my_topic-5 is still in progress

Status of partition reassignment:
Reassignment of partition my_topic-7 is still in progress
Reassignment of partition my_topic-14 completed successfully
Reassignment of partition my_topic-8 is still in progress
Reassignment of partition my_topic-4 is still in progress
Reassignment of partition my_topic-3 is still in progress
Reassignment of partition my_topic-13 completed successfully
Reassignment of partition my_topic-1 is still in progress
Reassignment of partition my_topic-15 completed successfully
Reassignment of partition my_topic-6 is still in progress
Reassignment of partition my_topic-11 is still in progress
Reassignment of partition my_topic-0 is still in progress
Reassignment of partition my_topic-12 completed successfully
Reassignment of partition my_topic-10 is still in progress
Reassignment of partition my_topic-2 is still in progress
Reassignment of partition my_topic-9 is still in progress
Reassignment of partition my_topic-5 is still in progress

I also checked Zookeeper for active brokers:

root@kafka_node_1:/opt/kafka_2.12-2.3.0/bin# ./zookeeper-shell.shnode1:2181 ls /brokers/ids

Connecting to node1:2181

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[1, 2, 3]

What's next?

Thanks

Sebastian


On 26-Sep-19 10:04 PM, M. Manna wrote:

hello,

Could you please try to run kafka-reassign-partitions with your topic
reassignment JSON? That doesn't require any restart, and should tell you if
any issues with reassignment. The examples are provided in Confluence wiki.

I would recommend that you do a "Describe" on your topic to ensure that all
partitions and ISR metadata is up-to-date.

Thanks,



On Thu, 26 Sep 2019 at 03:28, Sebastian Schmitz <
sebastian.schm...@propellerhead.co.nz> wrote:

Hello,

I have one topic with 12 partitions where the partition 0 is missing one
node from ISR... Is there a way I get get it back to work again without
having to do some weird stuff like restarting the cluster?
Because this missing node in ISR is causing some problems for the
consumers...

Thx

Sebastian


--
DISCLAIMER
This email contains information that is confidential and which
may be
legally privileged. If you have received this email in error please

notify the sender immediately and delete the email.
This email is intended
solely for the use of the intended recipient and you may not use or
disclose this email in any way.


--
DISCLAIMER

This email contains information that is confidential and whichmay belegally privileged. If you have received this email in error please

notify the sender immediately and delete the email.

This email is intendedsolely for the use of the intended recipient and you may not use ordisclose this email in any way.

Re: One Partition missing a node in ISR

Reply via email to