[
https://issues.apache.org/jira/browse/KAFKA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boyang Chen updated KAFKA-7641:
-------------------------------
Description:
In the JIRA discussion https://issues.apache.org/jira/browse/KAFKA-7610, Jason
concluded an edge case of current consumer protocol which could cause memory
burst on broker side:
```the case we observed in practice was caused by a consumer that was slow to
rejoin the group after a rebalance had begun. At the same time, there were new
members that were trying to join the group for the first time. The request
timeout was significantly lower than the rebalance timeout, so the JoinGroup of
the new members kept timing out. The timeout caused a retry and the group size
eventually become quite large because we could not detect the fact that the new
members were no longer there.```
Since many disorganized join group requests are spamming the group metadata, we
should define a cap on broker side to avoid one consumer group from growing too
large. So far I feel it's appropriate to introduce this as a server config
since most times this value is only dealing with error scenarios, client users
shouldn't worry about this config.
was:
In the JIRA discussion https://issues.apache.org/jira/browse/KAFKA-7610, Jason
concluded an edge case of current consumer protocol which could cause memory
burst on broker side:
```the case we observed in practice was caused by a consumer that was slow to
rejoin the group after a rebalance had begun. At the same time, there were new
members that were trying to join the group for the first time. The request
timeout was significantly lower than the rebalance timeout, so the JoinGroup of
the new members kept timing out. The timeout caused a retry and the group size
eventually become quite large because we could not detect the fact that the new
members were no longer there.```
Since many disorganized join group requests are spamming the group metadata, we
should define a cap on broker side to avoid one consumer group from growing too
large. So far I feel it's appropriate to introduce this as a server config
since most times this value is only dealing with error scenarios.
> Add `consumer.group.max.size` to cap consumer metadata size on broker
> ---------------------------------------------------------------------
>
> Key: KAFKA-7641
> URL: https://issues.apache.org/jira/browse/KAFKA-7641
> Project: Kafka
> Issue Type: Improvement
> Reporter: Boyang Chen
> Priority: Major
>
> In the JIRA discussion https://issues.apache.org/jira/browse/KAFKA-7610,
> Jason concluded an edge case of current consumer protocol which could cause
> memory burst on broker side:
> ```the case we observed in practice was caused by a consumer that was slow to
> rejoin the group after a rebalance had begun. At the same time, there were
> new members that were trying to join the group for the first time. The
> request timeout was significantly lower than the rebalance timeout, so the
> JoinGroup of the new members kept timing out. The timeout caused a retry and
> the group size eventually become quite large because we could not detect the
> fact that the new members were no longer there.```
> Since many disorganized join group requests are spamming the group metadata,
> we should define a cap on broker side to avoid one consumer group from
> growing too large. So far I feel it's appropriate to introduce this as a
> server config since most times this value is only dealing with error
> scenarios, client users shouldn't worry about this config.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
