No, that is not the reason. Initially all the partitions were assigned the messages and those were processed very fast and sit idle even other partitions are having a lot of messages to be processed. So I was under impression that rebalance should be triggered and messages will be re-distributed equally again.
Thanks Achintya -----Original Message----- From: Sharninder [mailto:sharnin...@gmail.com] Sent: Wednesday, November 23, 2016 12:33 AM To: us...@kafka.apache.org Cc: dev@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed Could it be because of the partition key ? On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Hi there, > > We are doing the load test in Kafka with 25tps and first 9 hours it > went fine almost 80K/hr messages were processed after that we see a > lot of lags and we stopped the incoming load. > > Currently we see 15K/hr messages are processing. We have 40 consumer > instances with concurrency 4 and 2 topics and both is having 160 > partitions so each consumer with each partition. > > What we found that some of the partitions are sitting idle and some of > are overloaded and its really slowing down the consumer message processing. > > Why rebalancing is not happening and existing messages are not > distributed equally among the instances? We tried to restart the app > still the same pace. Any idea what could be the reason? > > Thanks > Achintya > > -- -- Sharninder
