[ https://issues.apache.org/jira/browse/KAFKA-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16185754#comment-16185754 ]
Manikumar commented on KAFKA-4712: ---------------------------------- This can be related to KAFKA-5758 > kafka-reassign-partitions causes ISR to shrink > ---------------------------------------------- > > Key: KAFKA-4712 > URL: https://issues.apache.org/jira/browse/KAFKA-4712 > Project: Kafka > Issue Type: Bug > Components: admin > Affects Versions: 0.10.1.1 > Reporter: Andrew Holford > > Hi, > We have been doing some work on our kafka clusters recently which required us > to shrink the number of brokers and then increase the number again once the > work on the hosts was complete. > During the reassign partitions work to move data away from and back to the > servers we have worked on, the ISR has been seen to drop below our > min.insync.replicas value causing an outage to our data flow until the ISR > increases again. > From the documentation our understanding was that during the reassignment the > number of replicas should increase to include the new ones and then decrease > back to the replication factor, is this not the case? > The process we have followed is below: > 1. generate a json which contains all our topics.. > for example something like... > {"version":1,"topics":[{"topic":"topic1"},{"topic":"topic2"},{"topic":"topic3"}]} > 2. use the topic list json with the kafka-partition-assignment tool with the > generate option, eg: > bin/kafka-reassign-partitions --zookeeper > zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --topics-to-move-json-file > /tmp/topic-list.json --broker-list "1,2,3,4,5,6,7" --generate > 3. we then take the proposed configuration, place in another file, and > execute as below: > bin/kafka-reassign-partitions --zookeeper > zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --reassignment-json-file > /tmp/topic-to-reassign.json --execute > During the execution, we see lots of times different topics and different > partitions reduce to 1 ISR, one example below: > [2017-01-30 12:22:38,748] INFO Partition > [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for > partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1 to 5,1,7 > (kafka.cluster.Partition) > [2017-01-30 12:22:38,712] INFO Partition > [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for > partition [com_ig_trade_v1_position_snapshot--live,9] from 5 to 5,1 > (kafka.cluster.Partition) > [2017-01-30 12:22:38,582] INFO Partition > [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Shrinking ISR for > partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1,7 to 5 > (kafka.cluster.Partition) > [2017-01-30 12:22:38,326] INFO Partition > [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for > partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1,5 to > 5,1,6,2,7 (kafka.cluster.Partition) > [2017-01-30 12:14:49,906] INFO Partition > [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for > partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1 to 2,6,1,5 > (kafka.cluster.Partition) > Is this what is expected? > Do you have any recommendations / procedure / best practice which we could > use when decreasing / increasing the cluster size, without the ISRs shrinking? > Thanks, > Andy. -- This message was sent by Atlassian JIRA (v6.4.14#64029)