hi there, I'm not sure whether this is the correct group to post this, but pls suggest if there is any other channel which can guide me on this.
I understand that kafka consumer group concept enables us to come up with parallel processing of partitions by the underlying consumers within the same consumer group, meaning if I spin up multiple consumer objects each belonging to the same consumer group-id would have load balanced across its underlying consumer instances, by using various partition assignment strategy configs which are natively available. We were trying out with RoundRobin assignment strategy config; created an application which subscribes with 2 topics ( say topic-1 and topic-2, each having 10 partitions ) and created 10 consumer objects per topic, so that aspiration is that we have got 20 consumer objects processing from each of the partitions (total 20 ) of 2 topics; so far so good when we run one instance of this application as each consumer is attached to one partition of a topic. When we try to spin up another instance of this application, where it again does the same thing by attaching itself to same consumer-group-id, such that we will now have total 40 consumer objects overall and they get load balanced in such a way that 10 consumers from earlier instance of the application(1) released their partition assignments to this new instance (2) and seeming to be having an equal distribution among both these instances where each is sharing half of the load ( 10 consumers from each instance processing messages from 10 partitions each and 10 consumers in each instance staying idle) Now the fun starts when we’re trying to spin up 3rd instance of the application, we expected the load balance to be something like 7, 7 and 6 for each application instance, however it is turning out to be 7, 8 and 5 and sometimes even random allocations, which I don’t consider as fair equal distribution. We understand that we can not have more consumers than the partitions available out there, were looking to have some fine load balancing across the application instances sharing the same consumer group id to not to overburden one particular instance as such. Are we missing any config here or some fundamental understanding, please guide, many thanks!
