Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Becket, Thanks for the replies. Now I see you want to optimize with the heuristic that "if I see a JoinGroup shortly enough after a rebalance is completed, then likely there are more JoinGroups coming". I agree that it will help with a single-instance console consumer for debugging etc, but still

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Guozhang, Sorry for the confusion. I actually meant always "complete" the rebalance immediately when the first consumer joining the group. i.e. the configurable delta only kicks in after the first rebalance. The concern I have was actually not the frequent rebalance for the users, but the

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Becket: I think the problem is that when we have a single member joining an unknown group for the first time ever, do we want to complete the rebalance immediately or not; it does not matter if we want to "start" the rebalance, since even for now if the group coordinator is in the SyncGroup phase

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

I am a little hesitant to add the configuration to the client. It would be more flexible but this seems not the thing that users should worry about (I imagine many people would simply set backoff to 0 just for fast rebalance). I am wondering if the following variant of the current solution will

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Found this thread after posting an alternative idea after we starting hitting this issue ourselves for a job that has a lot of state stores and topic partitions. My suggestion was to have consumer groups have a configurable minimum member count before consumption begins, but that has its own trade

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Just recapping on client-side v.s. broker-side config: we did discuss about adding this as a client-side config and bump up join-group request (I think both Ismael and Ewen questioned about it) to include this configured value to the broker. I cannot remember if there is any strong motivations

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

I forgot to address the broker versus client-side point that Jun raised. We discussed passing the delay via the request and we were unable to make a strong case for it. It seems like tools that use a single consumer and group management (like the Console Consumer) are such a case though. Changing

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Jun/Ismael, Sounds good to me. Thanks, Damian On Tue, 6 Jun 2017 at 23:08 Ismael Juma wrote: > Hi Jun, > > The console consumer issue also came up in a conversation I was having > recently. Seems like the config/server.properties change is a reasonable > compromise given

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Jun, The console consumer issue also came up in a conversation I was having recently. Seems like the config/server.properties change is a reasonable compromise given that we have other defaults that are for development. Ismael On Tue, Jun 6, 2017 at 10:59 PM, Jun Rao

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Onur, It was in my previous email. But here it is again. 1. Better rebalance timing. We will try to rebalance only when all the consumers in a group have joined. The challenge would be someone has to define what does ALL consumers

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Damian. Can you copy the point Becket made earlier that you say isn't addressed? On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy wrote: > Thanks all, the Vote is now closed and the KIP has been accepted with 9 +1s > > 3 binding:: > Guozhang, > Jason, > Ismael > > 6

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Thanks all, the Vote is now closed and the KIP has been accepted with 9 +1s 3 binding:: Guozhang, Jason, Ismael 6 non-binding: Bill, Eno, Mathieu, Matthias, Dong, Mickael Thanks, Damian On Thu, 6 Apr 2017 at 09:26 Ismael Juma wrote: > Thanks for the KIP, +1 (binding). > >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Thanks for the KIP, +1 (binding). Ismael On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang > wrote: > > > +1 > > > > Sorry about the previous email, Gmail seems be

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Onur, Thanks for the update. I misunderstood what you said before. I believe what you are suggesting sounds ok, though i don't think it addresses the point Becket made earlier in the discussion thread. See below. Thanks, Damian 1.

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Delaying the SyncGroupRequest is not what I had in mind. What I was thinking was essentially a client-side stabilization window where the client does nothing other than participate in the group membership protocol and wait a bit for the group to stabilize. During this window, several rounds of

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Damian. After reading the discussion thread again, it still doesn't seem like the thread discussed the option I mentioned earlier. >From what I had understood from the broker-side vs. client-side config debate was that the client-side config from the discussion would cause a wire format

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non-binding) On Mon, Apr 3, 2017 at 9:53 AM, Mathieu Fenniak < mathieu.fenn...@replicon.com> wrote: > +1 (non-binding) > > This will be very helpful for me, looking forward to it! :-) > > On Thu, Mar 30, 2017 at 4:46 AM, Damian Guy wrote: > > > Hi All, > > > > I'd like

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non-binding) This will be very helpful for me, looking forward to it! :-) On Thu, Mar 30, 2017 at 4:46 AM, Damian Guy wrote: > Hi All, > > I'd like to start the voting thread on KIP-134: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non binding) Thanks On Fri, Mar 31, 2017 at 9:01 AM, Damian Guy wrote: > Hi Onur, > > We already discussed similar on the Discussion thread and decided to move > forward with a broker config. > > Thanks, > Damian > > On Fri, 31 Mar 2017 at 02:40 Onur Karaman

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Onur, We already discussed similar on the Discussion thread and decided to move forward with a broker config. Thanks, Damian On Fri, 31 Mar 2017 at 02:40 Onur Karaman wrote: > It seems like there generally has been the assumption that the broker needs > to

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

It seems like there generally has been the assumption that the broker needs to know about this delay either from its own config or provided over the wire from clients. Is this actually true? One alternative I don't think was mentioned was to make this delay concept be completely client-side.

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non-binding) Thanks! On Thu, Mar 30, 2017 at 6:03 PM, Becket Qin wrote: > +1 Thanks for the KIP! > > On Thu, Mar 30, 2017 at 12:55 PM, Jason Gustafson > wrote: > > > +1 Thanks for the KIP! > > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 Thanks for the KIP! On Thu, Mar 30, 2017 at 12:55 PM, Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang > wrote: > > > +1 > > > > Sorry about the previous email, Gmail seems be collapsing them into a >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 Thanks for the KIP! On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang wrote: > +1 > > Sorry about the previous email, Gmail seems be collapsing them into a > single thread on my inbox. > > Guozhang > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 Sorry about the previous email, Gmail seems be collapsing them into a single thread on my inbox. Guozhang On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang wrote: > Damian, could you create a new thread for the voting process? > > Thanks! > > Guozhang > > On Thu, Mar 30,

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Hi Guozhang, This was a new thread! Damian On Thu, 30 Mar 2017 at 19:34, Guozhang Wang wrote: > Damian, could you create a new thread for the voting process? > > Thanks! > > Guozhang > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck wrote: > > >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

Damian, could you create a new thread for the voting process? Thanks! Guozhang On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck wrote: > +1(non-binding) > > On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska > wrote: > > > +1 (non binding) > > > > Thanks >

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1(non-binding) On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska wrote: > +1 (non binding) > > Thanks > Eno > > On 30 Mar 2017, at 18:01, Matthias J. Sax wrote: > > > > +1 > > > > On 3/30/17 3:46 AM, Damian Guy wrote: > >> Hi All, > >> > >> I'd like

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 (non binding) Thanks Eno > On 30 Mar 2017, at 18:01, Matthias J. Sax wrote: > > +1 > > On 3/30/17 3:46 AM, Damian Guy wrote: >> Hi All, >> >> I'd like to start the voting thread on KIP-134: >>

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

+1 On 3/30/17 3:46 AM, Damian Guy wrote: > Hi All, > > I'd like to start the voting thread on KIP-134: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance > > Thanks, > Damian > signature.asc Description: OpenPGP digital signature

[VOTE] KIP-134: Delay initial consumer group rebalance

Hi All, I'd like to start the voting thread on KIP-134: https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance Thanks, Damian