Re: Closing a KafkaConsumer triggers a rebalance in KafkaConsumers subscribed to other topics. Is this the expected behavior?

[Jacob Sheck]

I believe you have answered your own question.  Having the system enforce
which topics each consumer in a consumer group needs to consume messages
from may be restricting, and possibly expensive.  I have seen this brought
up a few times and discussed it with a colleague.  I am not clear as to
what the advantage is to have the members of a consumer group consume from
different topics.  I have seen instances where this was caused by a
misconfiguration, but are there cases where this would be done
intentionally?  If it doesn't exist already, maybe mismatched topics in a
consumer group should be alerted on?

On Tue, Apr 3, 2018 at 11:18 AM Gabriel Giussi 
wrote:

> Kafka brokers version: 0.11.0.0
> Kafka client version: 0.11.0.2
>
> If we have two KafkaConsumer using the same group.id (running in the same
> process or in two different processes) and one of them is closed, it
> triggers a rebalance in the other KafkaConsumer even if they were
> subscribed to different topics.
> I suppose that brokers must be taking into account only the group.id for a
> rebalance and not the subscribed topics corresponding to the pair
> (group_id,member_id) of the LeaveGroupRequest but I'm wondering if this is
> the expected behavior or it's something that should be improved?
>
> I guess that is probably the first option to avoid a more complex rebalance
> in the broker and considering that the solution is very simple, i.e. just
> use different group ids for different KafkaConsumer that subscribe to
> different topics even if they are running in the same process.
>
> Thanks.
>

Closing a KafkaConsumer triggers a rebalance in KafkaConsumers subscribed to other topics. Is this the expected behavior?

[Gabriel Giussi]

Kafka brokers version: 0.11.0.0
Kafka client version: 0.11.0.2

If we have two KafkaConsumer using the same group.id (running in the same
process or in two different processes) and one of them is closed, it
triggers a rebalance in the other KafkaConsumer even if they were
subscribed to different topics.
I suppose that brokers must be taking into account only the group.id for a
rebalance and not the subscribed topics corresponding to the pair
(group_id,member_id) of the LeaveGroupRequest but I'm wondering if this is
the expected behavior or it's something that should be improved?

I guess that is probably the first option to avoid a more complex rebalance
in the broker and considering that the solution is very simple, i.e. just
use different group ids for different KafkaConsumer that subscribe to
different topics even if they are running in the same process.

Thanks.