Hey everybody, It's been a week since this KIP and not much discussion has been made. I assume that this is a straight forward change and I will open a voting thread in the next couple of days if nobody has anything to suggest.
Best, Stanislav On Thu, Nov 22, 2018 at 12:56 PM Stanislav Kozlovski <stanis...@confluent.io> wrote: > Greetings everybody, > > I have enriched the KIP a bit with a bigger Motivation section and also > renamed it. > KIP: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-389%3A+Introduce+a+configurable+consumer+group+size+limit > > I'm looking forward to discussions around it. > > Best, > Stanislav > > On Tue, Nov 20, 2018 at 1:47 PM Stanislav Kozlovski < > stanis...@confluent.io> wrote: > >> Hey there everybody, >> >> Thanks for the introduction Boyang. I appreciate the effort you are >> putting into improving consumer behavior in Kafka. >> >> @Matt >> I also believe the default value is high. In my opinion, we should aim to >> a default cap around 250. This is because in the current model any consumer >> rebalance is disrupting to every consumer. The bigger the group, the longer >> this period of disruption. >> >> If you have such a large consumer group, chances are that your >> client-side logic could be structured better and that you are not using the >> high number of consumers to achieve high throughput. >> 250 can still be considered of a high upper bound, I believe in practice >> users should aim to not go over 100 consumers per consumer group. >> >> In regards to the cap being global/per-broker, I think that we should >> consider whether we want it to be global or *per-topic*. For the time >> being, I believe that having it per-topic with a global default might be >> the best situation. Having it global only seems a bit restricting to me and >> it never hurts to support more fine-grained configurability (given it's the >> same config, not a new one being introduced). >> >> On Tue, Nov 20, 2018 at 11:32 AM Boyang Chen <bche...@outlook.com> wrote: >> >>> Thanks Matt for the suggestion! I'm still open to any suggestion to >>> change the default value. Meanwhile I just want to point out that this >>> value is a just last line of defense, not a real scenario we would expect. >>> >>> >>> In the meanwhile, I discussed with Stanislav and he would be driving the >>> 389 effort from now on. Stanislav proposed the idea in the first place and >>> had already come up a draft design, while I will keep focusing on KIP-345 >>> effort to ensure solving the edge case described in the JIRA< >>> https://issues.apache.org/jira/browse/KAFKA-7610>. >>> >>> >>> Thank you Stanislav for making this happen! >>> >>> >>> Boyang >>> >>> ________________________________ >>> From: Matt Farmer <m...@frmr.me> >>> Sent: Tuesday, November 20, 2018 10:24 AM >>> To: dev@kafka.apache.org >>> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member >>> metadata growth >>> >>> Thanks for the KIP. >>> >>> Will this cap be a global cap across the entire cluster or per broker? >>> >>> Either way the default value seems a bit high to me, but that could just >>> be >>> from my own usage patterns. I’d have probably started with 500 or 1k but >>> could be easily convinced that’s wrong. >>> >>> Thanks, >>> Matt >>> >>> On Mon, Nov 19, 2018 at 8:51 PM Boyang Chen <bche...@outlook.com> wrote: >>> >>> > Hey folks, >>> > >>> > >>> > I would like to start a discussion on KIP-389: >>> > >>> > >>> > >>> https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BEnforce%2Bgroup.max.size%2Bto%2Bcap%2Bmember%2Bmetadata%2Bgrowth&data=02%7C01%7C%7Cb0ee4fe97ad44cc046eb08d64e8f5d90%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636782774981237462&sdata=Q2T7hIoVq8GiPVhr0HIxVkGNChkiz1Pvk2zyLD5gCu8%3D&reserved=0 >>> > >>> > >>> > This is a pretty simple change to cap the consumer group size for >>> broker >>> > stability. Give me your valuable feedback when you got time. >>> > >>> > >>> > Thank you! >>> > >>> >> >> >> -- >> Best, >> Stanislav >> > > > -- > Best, > Stanislav > -- Best, Stanislav
