I forgot to address the broker versus client-side point that Jun raised. We discussed passing the delay via the request and we were unable to make a strong case for it. It seems like tools that use a single consumer and group management (like the Console Consumer) are such a case though. Changing the default in server.properties helps with quick start, but doesn't help if the broker is configured elsewhere (say a Cloud-like environment).
It's probably too late for this release, but we should consider it for the next release. Ismael On Tue, Jun 6, 2017 at 10:59 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Everyone, > > Sorry for being late on this thread. I just came across this thread. I have > a couple of concerns on this. (1) It seems the amount of delay will be > application specific. So, it seems that it's better for the delay to be a > client side config instead of a server side one? (2) When running console > consumer in quickstart, a minimum of 3 sec delay seems to be a bad > experience for our users. > > Since we are getting late into the release cycle, it may be a bit too late > to make big changes in the 0.11 release. Perhaps we should at least > consider overriding the delay in config/server.properties to 0 to improve > the quickstart experience? > > Thanks, > > Jun > > > On Tue, Apr 11, 2017 at 12:19 AM, Damian Guy <damian....@gmail.com> wrote: > > > Hi Onur, > > > > It was in my previous email. But here it is again. > > > > ============================================================ > > > > 1. Better rebalance timing. We will try to rebalance only when all the > > consumers in a group have joined. The challenge would be someone has to > > define what does ALL consumers mean, it could either be a time or number > of > > consumers, etc. > > > > 2. Avoid frequent rebalance. For example, if there are 100 consumers in a > > group, today, in the worst case, we may end up with 100 rebalances even > if > > all the consumers joined the group in a reasonably small amount of time. > > Frequent rebalance is also a bad thing for brokers. > > > > Having a client side configuration may solve problem 1 better because > each > > consumer group can potentially configure their own timing. However, it > does > > not really prevent frequent rebalance in general because some of the > > consumers can be misconfigured. (This may have something to do with > KIP-124 > > as well. But if quota is applied on the JoinGroup/SyncGroup request it > may > > cause some unwanted cascading effects.) > > > > Having a broker side configuration may result in less flexibility for > each > > consumer group, but it can prevent frequent rebalance better. I think > with > > some reasonable design, the rebalance timing issue can be resolved on the > > broker side as well. Matthias had a good point on extending the delay > when > > a new consumer joins a group (we actually did something similar to batch > > ISR change propagation). For example, let's say on the broker side, we > will > > always delay 2 seconds each time we see a new consumer joining a consumer > > group. This would probably work for most of the consumer groups and will > > also limit the rebalance frequency to protect the brokers. > > > > I am not sure about the streams use case here, but if something like 2 > > seconds of delay is acceptable for streams, I would prefer adding the > > configuration to the broker so that we can address both problems. > > > > On Thu, 6 Apr 2017 at 17:11 Onur Karaman <onurkaraman.apa...@gmail.com> > > wrote: > > > > > Hi Damian. > > > > > > Can you copy the point Becket made earlier that you say isn't > addressed? > > > > > > On Thu, Apr 6, 2017 at 2:51 AM, Damian Guy <damian....@gmail.com> > wrote: > > > > > > > Thanks all, the Vote is now closed and the KIP has been accepted > with 9 > > > +1s > > > > > > > > 3 binding:: > > > > Guozhang, > > > > Jason, > > > > Ismael > > > > > > > > 6 non-binding: > > > > Bill, > > > > Eno, > > > > Mathieu, > > > > Matthias, > > > > Dong, > > > > Mickael > > > > > > > > Thanks, > > > > Damian > > > > > > > > On Thu, 6 Apr 2017 at 09:26 Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > > > Thanks for the KIP, +1 (binding). > > > > > > > > > > Ismael > > > > > > > > > > On Thu, Mar 30, 2017 at 8:55 PM, Jason Gustafson < > ja...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > +1 Thanks for the KIP! > > > > > > > > > > > > On Thu, Mar 30, 2017 at 12:51 PM, Guozhang Wang < > > wangg...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > Sorry about the previous email, Gmail seems be collapsing them > > > into a > > > > > > > single thread on my inbox. > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 11:34 AM, Guozhang Wang < > > > wangg...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Damian, could you create a new thread for the voting process? > > > > > > > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > On Thu, Mar 30, 2017 at 10:33 AM, Bill Bejeck < > > bbej...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> +1(non-binding) > > > > > > > >> > > > > > > > >> On Thu, Mar 30, 2017 at 1:30 PM, Eno Thereska < > > > > > eno.there...@gmail.com > > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > +1 (non binding) > > > > > > > >> > > > > > > > > >> > Thanks > > > > > > > >> > Eno > > > > > > > >> > > On 30 Mar 2017, at 18:01, Matthias J. Sax < > > > > > matth...@confluent.io> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > > >> > > +1 > > > > > > > >> > > > > > > > > > >> > > On 3/30/17 3:46 AM, Damian Guy wrote: > > > > > > > >> > >> Hi All, > > > > > > > >> > >> > > > > > > > >> > >> I'd like to start the voting thread on KIP-134: > > > > > > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > > > > >> > 134%3A+Delay+initial+consumer+group+rebalance > > > > > > > >> > >> > > > > > > > >> > >> Thanks, > > > > > > > >> > >> Damian > > > > > > > >> > >> > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > >
