Hi, We are using kafka 0.7.
2 brokers, each broker has 10 partitions for one topic 3 consumers in one consumer group, each consumer create 10 streams. Today, when we want to rollout new service. After we restart one consumer we find exceptions and warning. kafka.common.ConsumerRebalanceFailedException: RecommendEvent_sd-sns-relation01.bj-1399630465426-53d3aefc can't rebalance after 4 retries [INFO 2014-05-12 15:17:47.364] kafka.utils.Logging$class.info(Logging.scala:61) [conflict in /consumers/RecommendEvent/owners/sensei/1-2 data: RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-2 stored data: RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1] [INFO 2014-05-12 15:17:47.366] kafka.utils.Logging$class.info(Logging.scala:61) [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the partition ownership to be deleted: 1-2] [INFO 2014-05-12 15:17:47.375] kafka.utils.Logging$class.info(Logging.scala:61) [conflict in /consumers/RecommendEvent/owners/sensei/1-3 data: RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-3 stored data: RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1] [INFO 2014-05-12 15:17:47.375] kafka.utils.Logging$class.info(Logging.scala:61) [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the partition ownership to be deleted: 1-3] [INFO 2014-05-12 15:17:47.385] kafka.utils.Logging$class.info(Logging.scala:61) [conflict in /consumers/RecommendEvent/owners/sensei/1-5 data: RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e-5 stored data: RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2] [INFO 2014-05-12 15:17:47.386] kafka.utils.Logging$class.info(Logging.scala:61) [RecommendEvent_sd-sns-relation01.bj-1399879066480-5426fb5e waiting for the partition ownership to be deleted: 1-5] And I opened zk viewer. In zk, we found 2 consumers in ConsumerGroup/ids: RecommendEvent_sd-sns-relation02.bj-1399635256619-5d8123c6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3 And in owners/topic/ we found all partitions are assigned to sd-sns-relation03.bj: Here is the owner info: 1:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 1:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 1:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 1:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 1:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 1:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 1:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 1:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 1:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 1:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 2:0 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-0 2:1 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-1 2:2 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-2 2:3 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-3 2:4 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-4 2:5 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-5 2:6 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-6 2:7 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-7 2:8 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-8 2:9 RecommendEvent_sd-sns-relation03.bj-1399635121250-487bdbb3-9 So all partitions are assigned to sd-sns-relation03.bj, but from logs and counter, we are sure sd-sns-relation02.bj has input too. My question is: 1. why rebalance failed? 2. why owner info is wrong? btw: zkclient is 0.2
