Hi, Currently we trying to configure Kafka in our system for pulling messages from Queues.
We have multiple consumers( we might want to add consumers if load on one consumer increases) which need to receive and process messages from a Kafka queue. Based on my understanding, under a single consumer group, one partition can be read by only one consumer. So if we want to make the setup such that no consumer gets over loaded in any case, what would be the best way to do it. If we have 6 partitions and 3 consumers which are equally efficient, then load seems to be distributed equally. Suppose one of the consumers say Consumer-3, for some reason processes the data 10 times slower, the we would want to reduce the load on Consumer-3 and equally distribute load on Consumer-1 and Counsumer-2. We wanted a pull based system which would help us in reducing the load on a slow consumer. Please let us know if there is any way to do this? Does kafka have any alternate implementation in such cases? Thanks, Madhavi.
