Sounds good! I've updated the KIP with another small section under Motivation. If there aren't any objections or further recommendations, I plan on starting a VOTE thread in the following days.
Best, Stanislav On Wed, Jan 9, 2019 at 8:54 PM Gwen Shapira <g...@confluent.io> wrote: > Thanks for the data driven approach, Stanislav. I love it :) > And thank you for sharing your formula, Boyang. I totally agree that > rebalance latency will not grow linearly with the consumer group size. > > My recommendation, considering what we know today: > 1. Add the limit config, and set it to MAX_INT by default (effectively > unlimited, without a magic number like -1) > 2. Document our thoughts - the concern about runaway groups, > Pinterest's 500 limit, Confluent's experience with few thousand > consumers in a group, the conclusions from Stanislav's memory research > (Personally, I wouldn't want what is essentially a linked list that we > iterate to grow beyond 1M). > > Mostly likely, 99% of the users won't need it and those who do will > have the right information to figure things out (or at least, they'll > know everything that we know). > > WDYT? > > On Wed, Jan 9, 2019 at 4:25 AM Stanislav Kozlovski > <stanis...@confluent.io> wrote: > > > > Hey everybody, > > > > I ran a quick benchmark and took some heap dumps to gauge how much memory > > each consumer in a group is using, all done locally. > > The setup was the following: 10 topics with 10 partitions each (100 > > partitions total) and one consumer group with 10 members, then expanded > to > > 20 members. > > Here are some notes of my findings in a public Google doc: > > > https://docs.google.com/document/d/1Z4aY5qg8lU2uNXzdgp_30_oJ9_I9xNelPko6GIQYXYk/edit?usp=sharing > > > > > > On Mon, Jan 7, 2019 at 10:51 PM Boyang Chen <bche...@outlook.com> wrote: > > > > > Hey Stanislav, > > > > > > I think the time taken to rebalance is not linearly correlated with > number > > > of consumers with our application. As for our current and future use > cases, > > > the main concern for Pinterest is still on the broker memory not CPU, > > > because crashing server by one application could have cascading effect > on > > > all jobs. > > > Do you want to drive a more detailed formula on how to compute the > memory > > > consumption against number of consumers within the group? > > > > > > In the meantime, I'm pretty buying in the motivation of this KIP, so I > > > think the follow-up work is just refinement to make the new config > easy to > > > use. We should be good > > > to vote IMO. > > > > > > Best, > > > Boyang > > > ________________________________ > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > Sent: Monday, January 7, 2019 4:21 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member > > > metadata growth > > > > > > Hey there, > > > > > > Per Gwen's comments, I slightly reworked the motivation section. Let me > > > know if it's any better now > > > > > > I completely agree that it would be best if we were to add a > recommended > > > number to a typical consumer group size. There is a problem that > timing the > > > CPU usage and rebalance times of consumer groups is tricky. We can > update > > > the KIP with memory guidelines (e.g 1 consumer in a group uses X > memory, > > > therefore 100 use Y). > > > I fear that the most useful recommendations though would be knowing > the CPU > > > impact of large consumer groups and the rebalance times. That is, > > > unfortunately, tricky to test and measure. > > > > > > @Boyang, you had mentioned some numbers used in Pinterest. If > available to > > > you, would you be comfortable sharing the number of consumers you are > using > > > in a group and maybe the potential time it takes to rebalance it? > > > > > > I'd appreciate any anecdotes regarding consumer group sizes from the > > > community > > > > > > Best, > > > Stanislav > > > > > > On Thu, Jan 3, 2019 at 1:59 AM Boyang Chen <bche...@outlook.com> > wrote: > > > > > > > Thanks Gwen for the suggestion! +1 on the guidance of defining > > > > group.max.size. I guess a sample formula would be: > > > > 2 * (# of brokers * average metadata cache size * 80%) / (# of > consumer > > > > groups * size of a single member metadata) > > > > > > > > if we assumed non-skewed partition assignment and pretty fair > consumer > > > > group consumption. The "2" is the 95 percentile of normal > distribution > > > and > > > > 80% is just to buffer some memory capacity which are both open to > > > > discussion. This config should be useful for Kafka platform team to > make > > > > sure one extreme large consumer group won't bring down the whole > cluster. > > > > > > > > What do you think? > > > > > > > > Best, > > > > Boyang > > > > > > > > ________________________________ > > > > From: Gwen Shapira <g...@confluent.io> > > > > Sent: Thursday, January 3, 2019 2:59 AM > > > > To: dev > > > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap member > > > > metadata growth > > > > > > > > Sorry for joining the fun late, but I think the problem we are > solving > > > > evolved a bit in the thread, and I'd like to have better > understanding > > > > of the problem before voting :) > > > > > > > > Both KIP and discussion assert that large groups are a problem, but > > > > they are kinda inconsistent regarding why they are a problem and > whose > > > > problem they are... > > > > 1. The KIP itself states that the main issue with large groups are > > > > long rebalance times. Per my understanding, this is mostly a problem > > > > for the application that consumes data, but not really a problem for > > > > the brokers themselves, so broker admins probably don't and shouldn't > > > > care about it. Also, my understanding is that this is a problem for > > > > consumer groups, but not necessarily a problem for other group types. > > > > 2. The discussion highlights the issue of "run away" groups that > > > > essentially create tons of members needlessly and use up lots of > > > > broker memory. This is something the broker admins will care about a > > > > lot. And is also a problem for every group that uses coordinators and > > > > not just consumers. And since the memory in question is the metadata > > > > cache, it probably has the largest impact on Kafka Streams > > > > applications, since they have lots of metadata. > > > > > > > > The solution proposed makes the most sense in the context of #2, so > > > > perhaps we should update the motivation section of the KIP to reflect > > > > that. > > > > > > > > The reason I'm probing here is that in my opinion we have to give our > > > > users some guidelines on what a reasonable limit is (otherwise, how > > > > will they know?). Calculating the impact of group-size on rebalance > > > > time in order to make good recommendations will take a significant > > > > effort. On the other hand, informing users regarding the memory > > > > footprint of a consumer in a group and using that to make a > reasonable > > > > suggestion isn't hard. > > > > > > > > Gwen > > > > > > > > > > > > On Sun, Dec 30, 2018 at 12:51 PM Stanislav Kozlovski > > > > <stanis...@confluent.io> wrote: > > > > > > > > > > Thanks Boyang, > > > > > > > > > > If there aren't any more thoughts on the KIP I'll start a vote > thread > > > in > > > > > the new year > > > > > > > > > > On Sat, Dec 29, 2018 at 12:58 AM Boyang Chen <bche...@outlook.com> > > > > wrote: > > > > > > > > > > > Yep Stanislav, that's what I'm proposing, and your explanation > makes > > > > sense. > > > > > > > > > > > > Boyang > > > > > > > > > > > > ________________________________ > > > > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > > > > Sent: Friday, December 28, 2018 7:59 PM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap > member > > > > > > metadata growth > > > > > > > > > > > > Hey there everybody, let's work on wrapping this discussion up. > > > > > > > > > > > > @Boyang, could you clarify what you mean by > > > > > > > One more question is whether you feel we should enforce group > size > > > > cap > > > > > > statically or on runtime? > > > > > > Is that related to the option of enabling this config via the > dynamic > > > > > > broker config feature? > > > > > > > > > > > > Regarding that - I feel it's useful to have and I also think it > might > > > > not > > > > > > introduce additional complexity. Ås long as we handle the config > > > being > > > > > > changed midway through a rebalance (via using the old value) we > > > should > > > > be > > > > > > good to go. > > > > > > > > > > > > On Wed, Dec 12, 2018 at 4:12 PM Stanislav Kozlovski < > > > > > > stanis...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > Hey Jason, > > > > > > > > > > > > > > Yes, that is what I meant by > > > > > > > > Given those constraints, I think that we can simply mark the > > > group > > > > as > > > > > > > `PreparingRebalance` with a rebalanceTimeout of the server > setting > > > ` > > > > > > > group.max.session.timeout.ms`. That's a bit long by default (5 > > > > minutes) > > > > > > > but I can't seem to come up with a better alternative > > > > > > > So either the timeout or all members calling joinGroup, yes > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 11, 2018 at 8:14 PM Boyang Chen < > bche...@outlook.com> > > > > wrote: > > > > > > > > > > > > > >> Hey Jason, > > > > > > >> > > > > > > >> I think this is the correct understanding. One more question > is > > > > whether > > > > > > >> you feel > > > > > > >> we should enforce group size cap statically or on runtime? > > > > > > >> > > > > > > >> Boyang > > > > > > >> ________________________________ > > > > > > >> From: Jason Gustafson <ja...@confluent.io> > > > > > > >> Sent: Tuesday, December 11, 2018 3:24 AM > > > > > > >> To: dev > > > > > > >> Subject: Re: [Discuss] KIP-389: Enforce group.max.size to cap > > > member > > > > > > >> metadata growth > > > > > > >> > > > > > > >> Hey Stanislav, > > > > > > >> > > > > > > >> Just to clarify, I think what you're suggesting is something > like > > > > this > > > > > > in > > > > > > >> order to gracefully shrink the group: > > > > > > >> > > > > > > >> 1. Transition the group to PREPARING_REBALANCE. No members are > > > > kicked > > > > > > out. > > > > > > >> 2. Continue to allow offset commits and heartbeats for all > current > > > > > > >> members. > > > > > > >> 3. Allow the first n members that send JoinGroup to stay in > the > > > > group, > > > > > > but > > > > > > >> wait for the JoinGroup (or session timeout) from all active > > > members > > > > > > before > > > > > > >> finishing the rebalance. > > > > > > >> > > > > > > >> So basically we try to give the current members an > opportunity to > > > > finish > > > > > > >> work, but we prevent some of them from rejoining after the > > > rebalance > > > > > > >> completes. It sounds reasonable if I've understood correctly. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> Jason > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On Fri, Dec 7, 2018 at 6:47 AM Boyang Chen < > bche...@outlook.com> > > > > wrote: > > > > > > >> > > > > > > >> > Yep, LGTM on my side. Thanks Stanislav! > > > > > > >> > ________________________________ > > > > > > >> > From: Stanislav Kozlovski <stanis...@confluent.io> > > > > > > >> > Sent: Friday, December 7, 2018 8:51 PM > > > > > > >> > To: dev@kafka.apache.org > > > > > > >> > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to > cap > > > > member > > > > > > >> > metadata growth > > > > > > >> > > > > > > > >> > Hi, > > > > > > >> > > > > > > > >> > We discussed this offline with Boyang and figured that it's > best > > > > to > > > > > > not > > > > > > >> > wait on the Cooperative Rebalancing proposal. Our thinking > is > > > > that we > > > > > > >> can > > > > > > >> > just force a rebalance from the broker, allowing consumers > to > > > > commit > > > > > > >> > offsets if their rebalanceListener is configured correctly. > > > > > > >> > When rebalancing improvements are implemented, we assume > that > > > they > > > > > > would > > > > > > >> > improve KIP-389's behavior as well as the normal rebalance > > > > scenarios > > > > > > >> > > > > > > > >> > On Wed, Dec 5, 2018 at 12:09 PM Boyang Chen < > > > bche...@outlook.com> > > > > > > >> wrote: > > > > > > >> > > > > > > > >> > > Hey Stanislav, > > > > > > >> > > > > > > > > >> > > thanks for the question! `Trivial rebalance` means "we > don't > > > > start > > > > > > >> > > reassignment right now, but you need to know it's coming > soon > > > > > > >> > > and you should start preparation". > > > > > > >> > > > > > > > > >> > > An example KStream use case is that before actually > starting > > > to > > > > > > shrink > > > > > > >> > the > > > > > > >> > > consumer group, we need to > > > > > > >> > > 1. partition the consumer group into two subgroups, where > one > > > > will > > > > > > be > > > > > > >> > > offline soon and the other will keep serving; > > > > > > >> > > 2. make sure the states associated with near-future > offline > > > > > > consumers > > > > > > >> are > > > > > > >> > > successfully replicated on the serving ones. > > > > > > >> > > > > > > > > >> > > As I have mentioned shrinking the consumer group is pretty > > > much > > > > > > >> > equivalent > > > > > > >> > > to group scaling down, so we could think of this > > > > > > >> > > as an add-on use case for cluster scaling. So my > understanding > > > > is > > > > > > that > > > > > > >> > the > > > > > > >> > > KIP-389 could be sequenced within our cooperative > rebalancing< > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FIncremental%2BCooperative%2BRebalancing%253A%2BSupport%2Band%2BPolicies&data=02%7C01%7C%7Cb603e099d6c744d8fac708d65ed51d03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636800666735874264&sdata=BX4DHEX1OMgfVuBOREwSjiITu5aV83Q7NAz77w4avVc%3D&reserved=0 > > > > > > >> > > > > > > > > > >> > > proposal. > > > > > > >> > > > > > > > > >> > > Let me know if this makes sense. > > > > > > >> > > > > > > > > >> > > Best, > > > > > > >> > > Boyang > > > > > > >> > > ________________________________ > > > > > > >> > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > > > > >> > > Sent: Wednesday, December 5, 2018 5:52 PM > > > > > > >> > > To: dev@kafka.apache.org > > > > > > >> > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size to > cap > > > > member > > > > > > >> > > metadata growth > > > > > > >> > > > > > > > > >> > > Hey Boyang, > > > > > > >> > > > > > > > > >> > > I think we still need to take care of group shrinkage > because > > > > even > > > > > > if > > > > > > >> > users > > > > > > >> > > change the config value we cannot guarantee that all > consumer > > > > groups > > > > > > >> > would > > > > > > >> > > have been manually shrunk. > > > > > > >> > > > > > > > > >> > > Regarding 2., I agree that forcefully triggering a > rebalance > > > > might > > > > > > be > > > > > > >> the > > > > > > >> > > most intuitive way to handle the situation. > > > > > > >> > > What does a "trivial rebalance" mean? Sorry, I'm not > familiar > > > > with > > > > > > the > > > > > > >> > > term. > > > > > > >> > > I was thinking that maybe we could force a rebalance, > which > > > > would > > > > > > >> cause > > > > > > >> > > consumers to commit their offsets (given their > > > > rebalanceListener is > > > > > > >> > > configured correctly) and subsequently reject some of the > > > > incoming > > > > > > >> > > `joinGroup` requests. Does that sound like it would work? > > > > > > >> > > > > > > > > >> > > On Wed, Dec 5, 2018 at 1:13 AM Boyang Chen < > > > bche...@outlook.com > > > > > > > > > > > >> wrote: > > > > > > >> > > > > > > > > >> > > > Hey Stanislav, > > > > > > >> > > > > > > > > > >> > > > I read the latest KIP and saw that we already changed > the > > > > default > > > > > > >> value > > > > > > >> > > to > > > > > > >> > > > -1. Do > > > > > > >> > > > we still need to take care of the consumer group > shrinking > > > > when > > > > > > >> doing > > > > > > >> > the > > > > > > >> > > > upgrade? > > > > > > >> > > > > > > > > > >> > > > However this is an interesting topic that worth > discussing. > > > > > > Although > > > > > > >> > > > rolling > > > > > > >> > > > upgrade is fine, `consumer.group.max.size` could always > have > > > > > > >> conflict > > > > > > >> > > with > > > > > > >> > > > the current > > > > > > >> > > > consumer group size which means we need to adhere to one > > > > source of > > > > > > >> > truth. > > > > > > >> > > > > > > > > > >> > > > 1.Choose the current group size, which means we never > > > > interrupt > > > > > > the > > > > > > >> > > > consumer group until > > > > > > >> > > > it transits to PREPARE_REBALANCE. And we keep track of > how > > > > many > > > > > > join > > > > > > >> > > group > > > > > > >> > > > requests > > > > > > >> > > > we have seen so far during PREPARE_REBALANCE. After > reaching > > > > the > > > > > > >> > consumer > > > > > > >> > > > cap, > > > > > > >> > > > we start to inform over provisioned consumers that you > > > should > > > > send > > > > > > >> > > > LeaveGroupRequest and > > > > > > >> > > > fail yourself. Or with what Mayuresh proposed in > KIP-345, we > > > > could > > > > > > >> mark > > > > > > >> > > > extra members > > > > > > >> > > > as hot backup and rebalance without them. > > > > > > >> > > > > > > > > > >> > > > 2.Choose the `consumer.group.max.size`. I feel > incremental > > > > > > >> rebalancing > > > > > > >> > > > (you proposed) could be of help here. > > > > > > >> > > > When a new cap is enforced, leader should be notified. > If > > > the > > > > > > >> current > > > > > > >> > > > group size is already over limit, leader > > > > > > >> > > > shall trigger a trivial rebalance to shuffle some topic > > > > partitions > > > > > > >> and > > > > > > >> > > let > > > > > > >> > > > a subset of consumers prepare the ownership > > > > > > >> > > > transition. Until they are ready, we trigger a real > > > rebalance > > > > to > > > > > > >> remove > > > > > > >> > > > over-provisioned consumers. It is pretty much > > > > > > >> > > > equivalent to `how do we scale down the consumer group > > > without > > > > > > >> > > > interrupting the current processing`. > > > > > > >> > > > > > > > > > >> > > > I personally feel inclined to 2 because we could kill > two > > > > birds > > > > > > with > > > > > > >> > one > > > > > > >> > > > stone in a generic way. What do you think? > > > > > > >> > > > > > > > > > >> > > > Boyang > > > > > > >> > > > ________________________________ > > > > > > >> > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > > > > >> > > > Sent: Monday, December 3, 2018 8:35 PM > > > > > > >> > > > To: dev@kafka.apache.org > > > > > > >> > > > Subject: Re: [Discuss] KIP-389: Enforce group.max.size > to > > > cap > > > > > > member > > > > > > >> > > > metadata growth > > > > > > >> > > > > > > > > > >> > > > Hi Jason, > > > > > > >> > > > > > > > > > >> > > > > 2. Do you think we should make this a dynamic config? > > > > > > >> > > > I'm not sure. Looking at the config from the > perspective of > > > a > > > > > > >> > > prescriptive > > > > > > >> > > > config, we may get away with not updating it > dynamically. > > > > > > >> > > > But in my opinion, it always makes sense to have a > config be > > > > > > >> > dynamically > > > > > > >> > > > configurable. As long as we limit it to being a > cluster-wide > > > > > > >> config, we > > > > > > >> > > > should be fine. > > > > > > >> > > > > > > > > > >> > > > > 1. I think it would be helpful to clarify the details > on > > > > how the > > > > > > >> > > > coordinator will shrink the group. It will need to > choose > > > > which > > > > > > >> members > > > > > > >> > > to > > > > > > >> > > > remove. Are we going to give current members an > opportunity > > > to > > > > > > >> commit > > > > > > >> > > > offsets before kicking them from the group? > > > > > > >> > > > > > > > > > >> > > > This turns out to be somewhat tricky. I think that we > may > > > not > > > > be > > > > > > >> able > > > > > > >> > to > > > > > > >> > > > guarantee that consumers don't process a message twice. > > > > > > >> > > > My initial approach was to do as much as we could to let > > > > consumers > > > > > > >> > commit > > > > > > >> > > > offsets. > > > > > > >> > > > > > > > > > >> > > > I was thinking that we mark a group to be shrunk, we > could > > > > keep a > > > > > > >> map > > > > > > >> > of > > > > > > >> > > > consumer_id->boolean indicating whether they have > committed > > > > > > >> offsets. I > > > > > > >> > > then > > > > > > >> > > > thought we could delay the rebalance until every > consumer > > > > commits > > > > > > >> (or > > > > > > >> > > some > > > > > > >> > > > time passes). > > > > > > >> > > > In the meantime, we would block all incoming fetch > calls (by > > > > > > either > > > > > > >> > > > returning empty records or a retriable error) and we > would > > > > > > continue > > > > > > >> to > > > > > > >> > > > accept offset commits (even twice for a single consumer) > > > > > > >> > > > > > > > > > >> > > > I see two problems with this approach: > > > > > > >> > > > * We have async offset commits, which implies that we > can > > > > receive > > > > > > >> fetch > > > > > > >> > > > requests before the offset commit req has been handled. > i.e > > > > > > consmer > > > > > > >> > sends > > > > > > >> > > > fetchReq A, offsetCommit B, fetchReq C - we may receive > > > A,C,B > > > > in > > > > > > the > > > > > > >> > > > broker. Meaning we could have saved the offsets for B > but > > > > > > rebalance > > > > > > >> > > before > > > > > > >> > > > the offsetCommit for the offsets processed in C come in. > > > > > > >> > > > * KIP-392 Allow consumers to fetch from closest replica > > > > > > >> > > > < > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-392%253A%2BAllow%2Bconsumers%2Bto%2Bfetch%2Bfrom%2Bclosest%2Breplica&data=02%7C01%7C%7Cb603e099d6c744d8fac708d65ed51d03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636800666735874264&sdata=bekXj%2FVdA6flZWQ70%2BSEyHm31%2F2WyWO1EpbvqyjWFJw%3D&reserved=0 > > > > > > >> > > > > > > > > > > >> > > > would > > > > > > >> > > > make it significantly harder to block poll() calls on > > > > consumers > > > > > > >> whose > > > > > > >> > > > groups are being shrunk. Even if we implemented a > solution, > > > > the > > > > > > same > > > > > > >> > race > > > > > > >> > > > condition noted above seems to apply and probably others > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > Given those constraints, I think that we can simply > mark the > > > > group > > > > > > >> as > > > > > > >> > > > `PreparingRebalance` with a rebalanceTimeout of the > server > > > > > > setting ` > > > > > > >> > > > group.max.session.timeout.ms`. That's a bit long by > default > > > > (5 > > > > > > >> > minutes) > > > > > > >> > > > but > > > > > > >> > > > I can't seem to come up with a better alternative > > > > > > >> > > > > > > > > > >> > > > I'm interested in hearing your thoughts. > > > > > > >> > > > > > > > > > >> > > > Thanks, > > > > > > >> > > > Stanislav > > > > > > >> > > > > > > > > > >> > > > On Fri, Nov 30, 2018 at 8:38 AM Jason Gustafson < > > > > > > ja...@confluent.io > > > > > > >> > > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > Hey Stanislav, > > > > > > >> > > > > > > > > > > >> > > > > What do you think about the use case I mentioned in my > > > > previous > > > > > > >> reply > > > > > > >> > > > about > > > > > > >> > > > > > a more resilient self-service Kafka? I believe the > > > benefit > > > > > > >> there is > > > > > > >> > > > > bigger. > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > I see this config as analogous to the open file limit. > > > > Probably > > > > > > >> this > > > > > > >> > > > limit > > > > > > >> > > > > was intended to be prescriptive at some point about > what > > > was > > > > > > >> deemed a > > > > > > >> > > > > reasonable number of open files for an application. > But > > > > mostly > > > > > > >> people > > > > > > >> > > > treat > > > > > > >> > > > > it as an annoyance which they have to work around. If > it > > > > happens > > > > > > >> to > > > > > > >> > be > > > > > > >> > > > hit, > > > > > > >> > > > > usually you just increase it because it is not tied > to an > > > > actual > > > > > > >> > > resource > > > > > > >> > > > > constraint. However, occasionally hitting the limit > does > > > > > > indicate > > > > > > >> an > > > > > > >> > > > > application bug such as a leak, so I wouldn't say it > is > > > > useless. > > > > > > >> > > > Similarly, > > > > > > >> > > > > the issue in KAFKA-7610 was a consumer leak and having > > > this > > > > > > limit > > > > > > >> > would > > > > > > >> > > > > have allowed the problem to be detected before it > impacted > > > > the > > > > > > >> > cluster. > > > > > > >> > > > To > > > > > > >> > > > > me, that's the main benefit. It's possible that it > could > > > be > > > > used > > > > > > >> > > > > prescriptively to prevent poor usage of groups, but > like > > > the > > > > > > open > > > > > > >> > file > > > > > > >> > > > > limit, I suspect administrators will just set it large > > > > enough > > > > > > that > > > > > > >> > > users > > > > > > >> > > > > are unlikely to complain. > > > > > > >> > > > > > > > > > > >> > > > > Anyway, just a couple additional questions: > > > > > > >> > > > > > > > > > > >> > > > > 1. I think it would be helpful to clarify the details > on > > > > how the > > > > > > >> > > > > coordinator will shrink the group. It will need to > choose > > > > which > > > > > > >> > members > > > > > > >> > > > to > > > > > > >> > > > > remove. Are we going to give current members an > > > opportunity > > > > to > > > > > > >> commit > > > > > > >> > > > > offsets before kicking them from the group? > > > > > > >> > > > > > > > > > > >> > > > > 2. Do you think we should make this a dynamic config? > > > > > > >> > > > > > > > > > > >> > > > > Thanks, > > > > > > >> > > > > Jason > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > On Wed, Nov 28, 2018 at 2:42 AM Stanislav Kozlovski < > > > > > > >> > > > > stanis...@confluent.io> > > > > > > >> > > > > wrote: > > > > > > >> > > > > > > > > > > >> > > > > > Hi Jason, > > > > > > >> > > > > > > > > > > > >> > > > > > You raise some very valid points. > > > > > > >> > > > > > > > > > > > >> > > > > > > The benefit of this KIP is probably limited to > > > > preventing > > > > > > >> > "runaway" > > > > > > >> > > > > > consumer groups due to leaks or some other > application > > > bug > > > > > > >> > > > > > What do you think about the use case I mentioned in > my > > > > > > previous > > > > > > >> > reply > > > > > > >> > > > > about > > > > > > >> > > > > > a more resilient self-service Kafka? I believe the > > > benefit > > > > > > >> there is > > > > > > >> > > > > bigger > > > > > > >> > > > > > > > > > > > >> > > > > > * Default value > > > > > > >> > > > > > You're right, we probably do need to be > conservative. > > > Big > > > > > > >> consumer > > > > > > >> > > > groups > > > > > > >> > > > > > are considered an anti-pattern and my goal was to > also > > > > hint at > > > > > > >> this > > > > > > >> > > > > through > > > > > > >> > > > > > the config's default. Regardless, it is better to > not > > > > have the > > > > > > >> > > > potential > > > > > > >> > > > > to > > > > > > >> > > > > > break applications with an upgrade. > > > > > > >> > > > > > Choosing between the default of something big like > 5000 > > > > or an > > > > > > >> > opt-in > > > > > > >> > > > > > option, I think we should go with the *disabled > default > > > > > > option* > > > > > > >> > > (-1). > > > > > > >> > > > > > The only benefit we would get from a big default of > 5000 > > > > is > > > > > > >> default > > > > > > >> > > > > > protection against buggy/malicious applications > that hit > > > > the > > > > > > >> > > KAFKA-7610 > > > > > > >> > > > > > issue. > > > > > > >> > > > > > While this KIP was spawned from that issue, I > believe > > > its > > > > > > value > > > > > > >> is > > > > > > >> > > > > enabling > > > > > > >> > > > > > the possibility of protection and helping move > towards a > > > > more > > > > > > >> > > > > self-service > > > > > > >> > > > > > Kafka. I also think that a default value of 5000 > might > > > be > > > > > > >> > misleading > > > > > > >> > > to > > > > > > >> > > > > > users and lead them to think that big consumer > groups (> > > > > 250) > > > > > > >> are a > > > > > > >> > > > good > > > > > > >> > > > > > thing. > > > > > > >> > > > > > > > > > > > >> > > > > > The good news is that KAFKA-7610 should be fully > > > resolved > > > > and > > > > > > >> the > > > > > > >> > > > > rebalance > > > > > > >> > > > > > protocol should, in general, be more solid after the > > > > planned > > > > > > >> > > > improvements > > > > > > >> > > > > > in KIP-345 and KIP-394. > > > > > > >> > > > > > > > > > > > >> > > > > > * Handling bigger groups during upgrade > > > > > > >> > > > > > I now see that we store the state of consumer > groups in > > > > the > > > > > > log > > > > > > >> and > > > > > > >> > > > why a > > > > > > >> > > > > > rebalance isn't expected during a rolling upgrade. > > > > > > >> > > > > > Since we're going with the default value of the > max.size > > > > being > > > > > > >> > > > disabled, > > > > > > >> > > > > I > > > > > > >> > > > > > believe we can afford to be more strict here. > > > > > > >> > > > > > During state reloading of a new Coordinator with a > > > defined > > > > > > >> > > > max.group.size > > > > > > >> > > > > > config, I believe we should *force* rebalances for > > > groups > > > > that > > > > > > >> > exceed > > > > > > >> > > > the > > > > > > >> > > > > > configured size. Then, only some consumers will be > able > > > to > > > > > > join > > > > > > >> and > > > > > > >> > > the > > > > > > >> > > > > max > > > > > > >> > > > > > size invariant will be satisfied. > > > > > > >> > > > > > > > > > > > >> > > > > > I updated the KIP with a migration plan, rejected > > > > alternatives > > > > > > >> and > > > > > > >> > > the > > > > > > >> > > > > new > > > > > > >> > > > > > default value. > > > > > > >> > > > > > > > > > > > >> > > > > > Thanks, > > > > > > >> > > > > > Stanislav > > > > > > >> > > > > > > > > > > > >> > > > > > On Tue, Nov 27, 2018 at 5:25 PM Jason Gustafson < > > > > > > >> > ja...@confluent.io> > > > > > > >> > > > > > wrote: > > > > > > >> > > > > > > > > > > > >> > > > > > > Hey Stanislav, > > > > > > >> > > > > > > > > > > > > >> > > > > > > Clients will then find that coordinator > > > > > > >> > > > > > > > and send `joinGroup` on it, effectively > rebuilding > > > the > > > > > > >> group, > > > > > > >> > > since > > > > > > >> > > > > the > > > > > > >> > > > > > > > cache of active consumers is not stored outside > the > > > > > > >> > Coordinator's > > > > > > >> > > > > > memory. > > > > > > >> > > > > > > > (please do say if that is incorrect) > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > Groups do not typically rebalance after a > coordinator > > > > > > change. > > > > > > >> You > > > > > > >> > > > could > > > > > > >> > > > > > > potentially force a rebalance if the group is too > big > > > > and > > > > > > kick > > > > > > >> > out > > > > > > >> > > > the > > > > > > >> > > > > > > slowest members or something. A more graceful > solution > > > > is > > > > > > >> > probably > > > > > > >> > > to > > > > > > >> > > > > > just > > > > > > >> > > > > > > accept the current size and prevent it from > getting > > > > bigger. > > > > > > We > > > > > > >> > > could > > > > > > >> > > > > log > > > > > > >> > > > > > a > > > > > > >> > > > > > > warning potentially. > > > > > > >> > > > > > > > > > > > > >> > > > > > > My thinking is that we should abstract away from > > > > conserving > > > > > > >> > > resources > > > > > > >> > > > > and > > > > > > >> > > > > > > > focus on giving control to the broker. The issue > > > that > > > > > > >> spawned > > > > > > >> > > this > > > > > > >> > > > > KIP > > > > > > >> > > > > > > was > > > > > > >> > > > > > > > a memory problem but I feel this change is > useful > > > in a > > > > > > more > > > > > > >> > > general > > > > > > >> > > > > > way. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > So you probably already know why I'm asking about > > > this. > > > > For > > > > > > >> > > consumer > > > > > > >> > > > > > groups > > > > > > >> > > > > > > anyway, resource usage would typically be > proportional > > > > to > > > > > > the > > > > > > >> > > number > > > > > > >> > > > of > > > > > > >> > > > > > > partitions that a group is reading from and not > the > > > > number > > > > > > of > > > > > > >> > > > members. > > > > > > >> > > > > > For > > > > > > >> > > > > > > example, consider the memory use in the offsets > cache. > > > > The > > > > > > >> > benefit > > > > > > >> > > of > > > > > > >> > > > > > this > > > > > > >> > > > > > > KIP is probably limited to preventing "runaway" > > > consumer > > > > > > >> groups > > > > > > >> > due > > > > > > >> > > > to > > > > > > >> > > > > > > leaks or some other application bug. That still > seems > > > > useful > > > > > > >> > > though. > > > > > > >> > > > > > > > > > > > > >> > > > > > > I completely agree with this and I *ask everybody > to > > > > chime > > > > > > in > > > > > > >> > with > > > > > > >> > > > > > opinions > > > > > > >> > > > > > > > on a sensible default value*. > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > I think we would have to be very conservative. The > > > group > > > > > > >> protocol > > > > > > >> > > is > > > > > > >> > > > > > > generic in some sense, so there may be use cases > we > > > > don't > > > > > > >> know of > > > > > > >> > > > where > > > > > > >> > > > > > > larger groups are reasonable. Probably we should > make > > > > this > > > > > > an > > > > > > >> > > opt-in > > > > > > >> > > > > > > feature so that we do not risk breaking anyone's > > > > application > > > > > > >> > after > > > > > > >> > > an > > > > > > >> > > > > > > upgrade. Either that, or use a very high default > like > > > > 5,000. > > > > > > >> > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > >> > > > > > > Jason > > > > > > >> > > > > > > > > > > > > >> > > > > > > On Tue, Nov 27, 2018 at 3:27 AM Stanislav > Kozlovski < > > > > > > >> > > > > > > stanis...@confluent.io> > > > > > > >> > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > >> > > > > > > > Hey Jason and Boyang, those were important > comments > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > One suggestion I have is that it would be > helpful > > > > to put > > > > > > >> your > > > > > > >> > > > > > reasoning > > > > > > >> > > > > > > > on deciding the current default value. For > example, > > > in > > > > > > >> certain > > > > > > >> > > use > > > > > > >> > > > > > cases > > > > > > >> > > > > > > at > > > > > > >> > > > > > > > Pinterest we are very likely to have more > consumers > > > > than > > > > > > 250 > > > > > > >> > when > > > > > > >> > > > we > > > > > > >> > > > > > > > configure 8 stream instances with 32 threads. > > > > > > >> > > > > > > > > For the effectiveness of this KIP, we should > > > > encourage > > > > > > >> people > > > > > > >> > > to > > > > > > >> > > > > > > discuss > > > > > > >> > > > > > > > their opinions on the default setting and > ideally > > > > reach a > > > > > > >> > > > consensus. > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > I completely agree with this and I *ask > everybody to > > > > chime > > > > > > >> in > > > > > > >> > > with > > > > > > >> > > > > > > opinions > > > > > > >> > > > > > > > on a sensible default value*. > > > > > > >> > > > > > > > My thought process was that in the current model > > > > > > rebalances > > > > > > >> in > > > > > > >> > > > large > > > > > > >> > > > > > > groups > > > > > > >> > > > > > > > are more costly. I imagine most use cases in > most > > > > Kafka > > > > > > >> users > > > > > > >> > do > > > > > > >> > > > not > > > > > > >> > > > > > > > require more than 250 consumers. > > > > > > >> > > > > > > > Boyang, you say that you are "likely to have... > when > > > > > > we..." > > > > > > >> - > > > > > > >> > do > > > > > > >> > > > you > > > > > > >> > > > > > have > > > > > > >> > > > > > > > systems running with so many consumers in a > group or > > > > are > > > > > > you > > > > > > >> > > > planning > > > > > > >> > > > > > > to? I > > > > > > >> > > > > > > > guess what I'm asking is whether this has been > > > tested > > > > in > > > > > > >> > > production > > > > > > >> > > > > > with > > > > > > >> > > > > > > > the current rebalance model (ignoring KIP-345) > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Can you clarify the compatibility impact > here? > > > What > > > > > > >> > > > > > > > > will happen to groups that are already larger > than > > > > the > > > > > > max > > > > > > >> > > size? > > > > > > >> > > > > > > > This is a very important question. > > > > > > >> > > > > > > > From my current understanding, when a > coordinator > > > > broker > > > > > > >> gets > > > > > > >> > > shut > > > > > > >> > > > > > > > down during a cluster rolling upgrade, a replica > > > will > > > > take > > > > > > >> > > > leadership > > > > > > >> > > > > > of > > > > > > >> > > > > > > > the `__offset_commits` partition. Clients will > then > > > > find > > > > > > >> that > > > > > > >> > > > > > coordinator > > > > > > >> > > > > > > > and send `joinGroup` on it, effectively > rebuilding > > > the > > > > > > >> group, > > > > > > >> > > since > > > > > > >> > > > > the > > > > > > >> > > > > > > > cache of active consumers is not stored outside > the > > > > > > >> > Coordinator's > > > > > > >> > > > > > memory. > > > > > > >> > > > > > > > (please do say if that is incorrect) > > > > > > >> > > > > > > > Then, I believe that working as if this is a new > > > > group is > > > > > > a > > > > > > >> > > > > reasonable > > > > > > >> > > > > > > > approach. Namely, fail joinGroups when the > max.size > > > is > > > > > > >> > exceeded. > > > > > > >> > > > > > > > What do you guys think about this? (I'll update > the > > > > KIP > > > > > > >> after > > > > > > >> > we > > > > > > >> > > > > settle > > > > > > >> > > > > > > on > > > > > > >> > > > > > > > a solution) > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Also, just to be clear, the resource we are > > > trying > > > > to > > > > > > >> > conserve > > > > > > >> > > > > here > > > > > > >> > > > > > is > > > > > > >> > > > > > > > what? Memory? > > > > > > >> > > > > > > > My thinking is that we should abstract away from > > > > > > conserving > > > > > > >> > > > resources > > > > > > >> > > > > > and > > > > > > >> > > > > > > > focus on giving control to the broker. The issue > > > that > > > > > > >> spawned > > > > > > >> > > this > > > > > > >> > > > > KIP > > > > > > >> > > > > > > was > > > > > > >> > > > > > > > a memory problem but I feel this change is > useful > > > in a > > > > > > more > > > > > > >> > > general > > > > > > >> > > > > > way. > > > > > > >> > > > > > > It > > > > > > >> > > > > > > > limits the control clients have on the cluster > and > > > > helps > > > > > > >> Kafka > > > > > > >> > > > > become a > > > > > > >> > > > > > > > more self-serving system. Admin/Ops teams can > better > > > > > > control > > > > > > >> > the > > > > > > >> > > > > impact > > > > > > >> > > > > > > > application developers can have on a Kafka > cluster > > > > with > > > > > > this > > > > > > >> > > change > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Best, > > > > > > >> > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > On Mon, Nov 26, 2018 at 8:00 PM Jason Gustafson > < > > > > > > >> > > > ja...@confluent.io> > > > > > > >> > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Hi Stanislav, > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Thanks for the KIP. Can you clarify the > > > > compatibility > > > > > > >> impact > > > > > > >> > > > here? > > > > > > >> > > > > > What > > > > > > >> > > > > > > > > will happen to groups that are already larger > than > > > > the > > > > > > max > > > > > > >> > > size? > > > > > > >> > > > > > Also, > > > > > > >> > > > > > > > just > > > > > > >> > > > > > > > > to be clear, the resource we are trying to > > > conserve > > > > here > > > > > > >> is > > > > > > >> > > what? > > > > > > >> > > > > > > Memory? > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > -Jason > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > On Mon, Nov 26, 2018 at 2:44 AM Boyang Chen < > > > > > > >> > > bche...@outlook.com > > > > > > >> > > > > > > > > > > >> > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > Thanks Stanislav for the update! One > suggestion > > > I > > > > have > > > > > > >> is > > > > > > >> > > that > > > > > > >> > > > it > > > > > > >> > > > > > > would > > > > > > >> > > > > > > > > be > > > > > > >> > > > > > > > > > helpful to put your > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > reasoning on deciding the current default > value. > > > > For > > > > > > >> > example, > > > > > > >> > > > in > > > > > > >> > > > > > > > certain > > > > > > >> > > > > > > > > > use cases at Pinterest we are very likely > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > to have more consumers than 250 when we > > > configure > > > > 8 > > > > > > >> stream > > > > > > >> > > > > > instances > > > > > > >> > > > > > > > with > > > > > > >> > > > > > > > > > 32 threads. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > For the effectiveness of this KIP, we should > > > > encourage > > > > > > >> > people > > > > > > >> > > > to > > > > > > >> > > > > > > > discuss > > > > > > >> > > > > > > > > > their opinions on the default setting and > > > ideally > > > > > > reach > > > > > > >> a > > > > > > >> > > > > > consensus. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Best, > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Boyang > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > ________________________________ > > > > > > >> > > > > > > > > > From: Stanislav Kozlovski < > > > stanis...@confluent.io > > > > > > > > > > > >> > > > > > > > > > Sent: Monday, November 26, 2018 6:14 PM > > > > > > >> > > > > > > > > > To: dev@kafka.apache.org > > > > > > >> > > > > > > > > > Subject: Re: [Discuss] KIP-389: Enforce > > > > group.max.size > > > > > > >> to > > > > > > >> > cap > > > > > > >> > > > > > member > > > > > > >> > > > > > > > > > metadata growth > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Hey everybody, > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > It's been a week since this KIP and not much > > > > > > discussion > > > > > > >> has > > > > > > >> > > > been > > > > > > >> > > > > > > made. > > > > > > >> > > > > > > > > > I assume that this is a straight forward > change > > > > and I > > > > > > >> will > > > > > > >> > > > open a > > > > > > >> > > > > > > > voting > > > > > > >> > > > > > > > > > thread in the next couple of days if nobody > has > > > > > > >> anything to > > > > > > >> > > > > > suggest. > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > Best, > > > > > > >> > > > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > On Thu, Nov 22, 2018 at 12:56 PM Stanislav > > > > Kozlovski < > > > > > > >> > > > > > > > > > stanis...@confluent.io> > > > > > > >> > > > > > > > > > wrote: > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > Greetings everybody, > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > I have enriched the KIP a bit with a > bigger > > > > > > Motivation > > > > > > >> > > > section > > > > > > >> > > > > > and > > > > > > >> > > > > > > > also > > > > > > >> > > > > > > > > > > renamed it. > > > > > > >> > > > > > > > > > > KIP: > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BIntroduce%2Ba%2Bconfigurable%2Bconsumer%2Bgroup%2Bsize%2Blimit&data=02%7C01%7C%7Cb603e099d6c744d8fac708d65ed51d03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636800666735874264&sdata=dLVLofL8NnQatVq6WEDukxfIorh7HeQR9TyyUifcAPo%3D&reserved=0 > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > I'm looking forward to discussions around > it. > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > Best, > > > > > > >> > > > > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Nov 20, 2018 at 1:47 PM Stanislav > > > > Kozlovski > > > > > > < > > > > > > >> > > > > > > > > > > stanis...@confluent.io> wrote: > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > >> Hey there everybody, > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> Thanks for the introduction Boyang. I > > > > appreciate > > > > > > the > > > > > > >> > > effort > > > > > > >> > > > > you > > > > > > >> > > > > > > are > > > > > > >> > > > > > > > > > >> putting into improving consumer behavior > in > > > > Kafka. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> @Matt > > > > > > >> > > > > > > > > > >> I also believe the default value is > high. In > > > my > > > > > > >> opinion, > > > > > > >> > > we > > > > > > >> > > > > > should > > > > > > >> > > > > > > > aim > > > > > > >> > > > > > > > > > to > > > > > > >> > > > > > > > > > >> a default cap around 250. This is > because in > > > > the > > > > > > >> current > > > > > > >> > > > model > > > > > > >> > > > > > any > > > > > > >> > > > > > > > > > consumer > > > > > > >> > > > > > > > > > >> rebalance is disrupting to every > consumer. > > > The > > > > > > bigger > > > > > > >> > the > > > > > > >> > > > > group, > > > > > > >> > > > > > > the > > > > > > >> > > > > > > > > > longer > > > > > > >> > > > > > > > > > >> this period of disruption. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> If you have such a large consumer group, > > > > chances > > > > > > are > > > > > > >> > that > > > > > > >> > > > your > > > > > > >> > > > > > > > > > >> client-side logic could be structured > better > > > > and > > > > > > that > > > > > > >> > you > > > > > > >> > > > are > > > > > > >> > > > > > not > > > > > > >> > > > > > > > > using > > > > > > >> > > > > > > > > > the > > > > > > >> > > > > > > > > > >> high number of consumers to achieve high > > > > > > throughput. > > > > > > >> > > > > > > > > > >> 250 can still be considered of a high > upper > > > > bound, > > > > > > I > > > > > > >> > > believe > > > > > > >> > > > > in > > > > > > >> > > > > > > > > practice > > > > > > >> > > > > > > > > > >> users should aim to not go over 100 > consumers > > > > per > > > > > > >> > consumer > > > > > > >> > > > > > group. > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> In regards to the cap being > > > global/per-broker, > > > > I > > > > > > >> think > > > > > > >> > > that > > > > > > >> > > > we > > > > > > >> > > > > > > > should > > > > > > >> > > > > > > > > > >> consider whether we want it to be global > or > > > > > > >> *per-topic*. > > > > > > >> > > For > > > > > > >> > > > > the > > > > > > >> > > > > > > > time > > > > > > >> > > > > > > > > > >> being, I believe that having it per-topic > > > with > > > > a > > > > > > >> global > > > > > > >> > > > > default > > > > > > >> > > > > > > > might > > > > > > >> > > > > > > > > be > > > > > > >> > > > > > > > > > >> the best situation. Having it global only > > > > seems a > > > > > > bit > > > > > > >> > > > > > restricting > > > > > > >> > > > > > > to > > > > > > >> > > > > > > > > me > > > > > > >> > > > > > > > > > and > > > > > > >> > > > > > > > > > >> it never hurts to support more > fine-grained > > > > > > >> > > configurability > > > > > > >> > > > > > (given > > > > > > >> > > > > > > > > it's > > > > > > >> > > > > > > > > > the > > > > > > >> > > > > > > > > > >> same config, not a new one being > introduced). > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> On Tue, Nov 20, 2018 at 11:32 AM Boyang > Chen > > > < > > > > > > >> > > > > > bche...@outlook.com > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >>> Thanks Matt for the suggestion! I'm > still > > > > open to > > > > > > >> any > > > > > > >> > > > > > suggestion > > > > > > >> > > > > > > to > > > > > > >> > > > > > > > > > >>> change the default value. Meanwhile I > just > > > > want to > > > > > > >> > point > > > > > > >> > > > out > > > > > > >> > > > > > that > > > > > > >> > > > > > > > > this > > > > > > >> > > > > > > > > > >>> value is a just last line of defense, > not a > > > > real > > > > > > >> > scenario > > > > > > >> > > > we > > > > > > >> > > > > > > would > > > > > > >> > > > > > > > > > expect. > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> In the meanwhile, I discussed with > Stanislav > > > > and > > > > > > he > > > > > > >> > would > > > > > > >> > > > be > > > > > > >> > > > > > > > driving > > > > > > >> > > > > > > > > > the > > > > > > >> > > > > > > > > > >>> 389 effort from now on. Stanislav > proposed > > > the > > > > > > idea > > > > > > >> in > > > > > > >> > > the > > > > > > >> > > > > > first > > > > > > >> > > > > > > > > place > > > > > > >> > > > > > > > > > and > > > > > > >> > > > > > > > > > >>> had already come up a draft design, > while I > > > > will > > > > > > >> keep > > > > > > >> > > > > focusing > > > > > > >> > > > > > on > > > > > > >> > > > > > > > > > KIP-345 > > > > > > >> > > > > > > > > > >>> effort to ensure solving the edge case > > > > described > > > > > > in > > > > > > >> the > > > > > > >> > > > JIRA< > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-7610&data=02%7C01%7C%7Cb603e099d6c744d8fac708d65ed51d03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636800666735874264&sdata=F55UaGVkDXaj4q7v7jUvPL50pD74GE90R7OGX%2FV3f%2Fs%3D&reserved=0 > > > > > > >> > > > > > > > > > >. > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Thank you Stanislav for making this > happen! > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Boyang > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> ________________________________ > > > > > > >> > > > > > > > > > >>> From: Matt Farmer <m...@frmr.me> > > > > > > >> > > > > > > > > > >>> Sent: Tuesday, November 20, 2018 10:24 > AM > > > > > > >> > > > > > > > > > >>> To: dev@kafka.apache.org > > > > > > >> > > > > > > > > > >>> Subject: Re: [Discuss] KIP-389: Enforce > > > > > > >> group.max.size > > > > > > >> > to > > > > > > >> > > > cap > > > > > > >> > > > > > > > member > > > > > > >> > > > > > > > > > >>> metadata growth > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Thanks for the KIP. > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Will this cap be a global cap across the > > > > entire > > > > > > >> cluster > > > > > > >> > > or > > > > > > >> > > > > per > > > > > > >> > > > > > > > > broker? > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Either way the default value seems a bit > > > high > > > > to > > > > > > me, > > > > > > >> > but > > > > > > >> > > > that > > > > > > >> > > > > > > could > > > > > > >> > > > > > > > > > just > > > > > > >> > > > > > > > > > >>> be > > > > > > >> > > > > > > > > > >>> from my own usage patterns. I'd have > > > probably > > > > > > >> started > > > > > > >> > > with > > > > > > >> > > > > 500 > > > > > > >> > > > > > or > > > > > > >> > > > > > > > 1k > > > > > > >> > > > > > > > > > but > > > > > > >> > > > > > > > > > >>> could be easily convinced that's wrong. > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> Thanks, > > > > > > >> > > > > > > > > > >>> Matt > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> On Mon, Nov 19, 2018 at 8:51 PM Boyang > Chen > > > < > > > > > > >> > > > > > bche...@outlook.com > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > wrote: > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >>> > Hey folks, > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > I would like to start a discussion on > > > > KIP-389: > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-389%253A%2BEnforce%2Bgroup.max.size%2Bto%2Bcap%2Bmember%2Bmetadata%2Bgrowth&data=02%7C01%7C%7Cb603e099d6c744d8fac708d65ed51d03%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636800666735874264&sdata=n%2FHp2DM4k48Q9hayOlc8q5VlcBKFtVWnLDOAzm%2FZ25Y%3D&reserved=0 > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > This is a pretty simple change to cap > the > > > > > > consumer > > > > > > >> > > group > > > > > > >> > > > > size > > > > > > >> > > > > > > for > > > > > > >> > > > > > > > > > >>> broker > > > > > > >> > > > > > > > > > >>> > stability. Give me your valuable > feedback > > > > when > > > > > > you > > > > > > >> > got > > > > > > >> > > > > time. > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > Thank you! > > > > > > >> > > > > > > > > > >>> > > > > > > > >> > > > > > > > > > >>> > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > >> -- > > > > > > >> > > > > > > > > > >> Best, > > > > > > >> > > > > > > > > > >> Stanislav > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > -- > > > > > > >> > > > > > > > > > > Best, > > > > > > >> > > > > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > -- > > > > > > >> > > > > > > > > > Best, > > > > > > >> > > > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > -- > > > > > > >> > > > > > > > Best, > > > > > > >> > > > > > > > Stanislav > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > -- > > > > > > >> > > > > > Best, > > > > > > >> > > > > > Stanislav > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > -- > > > > > > >> > > > Best, > > > > > > >> > > > Stanislav > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > -- > > > > > > >> > > Best, > > > > > > >> > > Stanislav > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > -- > > > > > > >> > Best, > > > > > > >> > Stanislav > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best, > > > > > > > Stanislav > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best, > > > > > > Stanislav > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best, > > > > > Stanislav > > > > > > > > > > > > > > > > -- > > > > Gwen Shapira > > > > Product Manager | Confluent > > > > 650.450.2760 | @gwenshap > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > -- > > > Best, > > > Stanislav > > > > > > > > > -- > > Best, > > Stanislav > > > > -- > Gwen Shapira > Product Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog > -- Best, Stanislav
