[
https://issues.apache.org/jira/browse/KAFKA-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250410#comment-17250410
]
Justin Jack commented on KAFKA-7641:
------------------------------------
stanislavkozlovski commented on pull request #6163: KAFKA-7641: Introduce
"consumer.group.max.size" config to limit consumer group sizes
URL: https://github.com/apache/kafka/pull/6163
This patch introduces a new config - "consumer.group.max.size", which caps the
maximum size any consumer group can reach. It has a default value of
Int.MAX_VALUE.
Once a consumer group is of the maximum size, subsequent JoinGroup requests
receive a MAX_SIZE_REACHED error.
In the case where the config is changed and a Coordinator broker with the new
config loads an old group that is over the threshold, members are kicked out of
the group and a rebalance is forced.
I have added two integration tests for both scenarios - a member joining an
already-full group and a rolling restart with a new config
Committer Checklist (excluded from commit message)
[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
for more details please visit our [online manga reading
directory|https://mangapan.com/]
> Add `consumer.group.max.size` to cap consumer metadata size on broker
> ---------------------------------------------------------------------
>
> Key: KAFKA-7641
> URL: https://issues.apache.org/jira/browse/KAFKA-7641
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Boyang Chen
> Assignee: Stanislav Kozlovski
> Priority: Major
> Labels: kip
> Fix For: 2.2.0
>
>
> In the JIRA discussion https://issues.apache.org/jira/browse/KAFKA-7610,
> Jason concluded an edge case of current consumer protocol which could cause
> memory burst on broker side:
> ```the case we observed in practice was caused by a consumer that was slow to
> rejoin the group after a rebalance had begun. At the same time, there were
> new members that were trying to join the group for the first time. The
> request timeout was significantly lower than the rebalance timeout, so the
> JoinGroup of the new members kept timing out. The timeout caused a retry and
> the group size eventually become quite large because we could not detect the
> fact that the new members were no longer there.```
> Since many disorganized join group requests are spamming the group metadata,
> we should define a cap on broker side to avoid one consumer group from
> growing too large. So far I feel it's appropriate to introduce this as a
> server config since most times this value is only dealing with error
> scenarios, client users shouldn't worry about this config.
> KIP-389:
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Introduce+a+configurable+consumer+group+size+limit]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
