Could it be because of the partition key ? On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote:
> Hi there, > > We are doing the load test in Kafka with 25tps and first 9 hours it went > fine almost 80K/hr messages were processed after that we see a lot of lags > and we stopped the incoming load. > > Currently we see 15K/hr messages are processing. We have 40 consumer > instances with concurrency 4 and 2 topics and both is having 160 partitions > so each consumer with each partition. > > What we found that some of the partitions are sitting idle and some of are > overloaded and its really slowing down the consumer message processing. > > Why rebalancing is not happening and existing messages are not distributed > equally among the instances? We tried to restart the app still the same > pace. Any idea what could be the reason? > > Thanks > Achintya > > -- -- Sharninder
