Re: Kafka consumers are not equally distributed
Could it be because of the partition key ? On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Hi there, > > We are doing the load test in Kafka with 25tps and first 9 hours it went > fine almost 80K/hr messages were processed after that we see a lot of lags > and we stopped the incoming load. > > Currently we see 15K/hr messages are processing. We have 40 consumer > instances with concurrency 4 and 2 topics and both is having 160 partitions > so each consumer with each partition. > > What we found that some of the partitions are sitting idle and some of are > overloaded and its really slowing down the consumer message processing. > > Why rebalancing is not happening and existing messages are not distributed > equally among the instances? We tried to restart the app still the same > pace. Any idea what could be the reason? > > Thanks > Achintya > > -- -- Sharninder
Re: Mapping a consumer in a consumer group to a partition in a topic
As far as I know, with a consumer group implementation you cannot pin consumers to partitions. That logic is taken care of by the high level API on its own. > On 23-Sep-2015, at 6:38 AM, Spandan Harithas Karamchedu > wrote: > > Hi, > > We created a topic with 3 partitions and a replication factor of 3. We are > able to implement a consumer to get the data from a specific partition in a > topic but we are struck in implementing a consumer within a specified > consumer group to be mapped to single partition of a topic and get the data > from the single partition of the topic. Can you let us how we can configure > a consumer to a partition within consumer group? > > > > We wanted to develop a kafka cluster similar to below one as specified in > kafka docs. We are using kafka 0.8.1. > > > > > Thanks > > Spandan
