Thank you Guozhang. Let me clarify : "some of the partitions are sitting idle and some of are overloaded", I mean we stopped the load after 9 hours as see the messages were processing very slow. That time we observed that some partitions had lot of messages and some were sitting idle. So my question why messages were not shared if we see some are overloaded and some are having 0 messages. Even we started the kafka servers and application servers too but nothing happened, still it was processing very slow and messages were not distributed. So we are concerned what should do this kind of situation and make the consumers more speedy.
Thanks Achintya -----Original Message----- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Thursday, November 24, 2016 11:21 PM To: users@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed The default partition assignment strategy is the RangePartitioner. Note it is per-topic, so if you use the default partitioner then in your case 160 partitions of each of the topic will be assigned to the first 160 consumer instances, each getting two partitions, one partition from each. So the consumer should be balanced on the consumer-instance basis. I'm not sure what you meant by "some of the partitions are sitting idle and some of are overloaded", do you mean that some partitions does not have new data coming in and others keep getting high traffic producing to it that the consumer cannot keep up? In this case it is no the consumer's issue, but the producer not producing in a balanced manner. Guozhang On Thu, Nov 24, 2016 at 7:45 PM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Java consumer. 0.9.1 > > Thanks > Achintya > > -----Original Message----- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Thursday, November 24, 2016 8:28 PM > To: users@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Which version of Kafka are you using with your consumer? Is it Scala > or Java consumers? > > > Guozhang > > > On Wed, Nov 23, 2016 at 6:38 AM, Ghosh, Achintya (Contractor) < > achintya_gh...@comcast.com> wrote: > > > No, that is not the reason. Initially all the partitions were > > assigned the messages and those were processed very fast and sit > > idle even other partitions are having a lot of messages to be processed. > > So I was under impression that rebalance should be triggered and > > messages will be re-distributed equally again. > > > > Thanks > > Achintya > > > > -----Original Message----- > > From: Sharninder [mailto:sharnin...@gmail.com] > > Sent: Wednesday, November 23, 2016 12:33 AM > > To: users@kafka.apache.org > > Cc: d...@kafka.apache.org > > Subject: Re: Kafka consumers are not equally distributed > > > > Could it be because of the partition key ? > > > > On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < > > achintya_gh...@comcast.com> wrote: > > > > > Hi there, > > > > > > We are doing the load test in Kafka with 25tps and first 9 hours > > > it went fine almost 80K/hr messages were processed after that we > > > see a lot of lags and we stopped the incoming load. > > > > > > Currently we see 15K/hr messages are processing. We have 40 > > > consumer instances with concurrency 4 and 2 topics and both is > > > having 160 partitions so each consumer with each partition. > > > > > > What we found that some of the partitions are sitting idle and > > > some of are overloaded and its really slowing down the consumer > > > message > > processing. > > > > > > Why rebalancing is not happening and existing messages are not > > > distributed equally among the instances? We tried to restart the > > > app still the same pace. Any idea what could be the reason? > > > > > > Thanks > > > Achintya > > > > > > > > > > > > -- > > -- > > Sharninder > > > > > > -- > -- Guozhang > -- -- Guozhang
