Thanks so much Harsha and Jason! Will address the comment and make it a tuple.
Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Jason Gustafson <ja...@confluent.io> Sent: Friday, January 4, 2019 3:50 PM To: dev Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to reduce consumer rebalances Hey Boyang, I had just a follow-up to my comment above. I wanted to suggest an alternative schema for LeaveGroup: LeaveGroupRequest => GroupId [GroupInstanceId MemberId] So we have a single array instead of two arrays. Each element identifies a single member. For dynamic members, we would use GroupInstanceId="" and provide the dynamic MemberId, which is consistent with JoinGroup. For static members, GroupInstanceId must be provided and Member could be considered optional. I think this makes the schema more coherent, but I'll leave it to you if there is a good reason to keep them separate. In any case, my vote is +1. Thanks for the hard work on this KIP! Best, Jason On Fri, Jan 4, 2019 at 1:09 PM Boyang Chen <bche...@outlook.com> wrote: > Thanks Guozhang for the proposal! The update is done. > > ________________________________ > From: Guozhang Wang <wangg...@gmail.com> > Sent: Saturday, January 5, 2019 3:33 AM > To: dev > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > reduce consumer rebalances > > Hello Boyang, > > I've made another pass on the wiki page again. One minor comment is on the > "Server Behavior Changes" section, we should have a paragraph on the > logical changes on handling new versions of LeaveGroupRequest (e.g. how to > handle dynamic member v.s. static member etc). > > Other than that, I do not have further comments. I think we can continue > the voting process after that. > > Guozhang > > On Wed, Jan 2, 2019 at 10:00 AM Boyang Chen <bche...@outlook.com> wrote: > > > Thanks Jason for the comment! I answered it on the discuss thread. > > > > Folks, could we continue the vote for this KIP? This is a very critical > > improvement for our streaming system > > stability and we need to get things rolling right at the start of 2019. > > > > Thank you for your time! > > Boyang > > > > ________________________________ > > From: Jason Gustafson <ja...@confluent.io> > > Sent: Tuesday, December 18, 2018 7:40 AM > > To: dev > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > reduce consumer rebalances > > > > Hi Boyang, > > > > Thanks, the KIP looks good. Just one comment. > > > > The new schema for the LeaveGroup request is slightly odd since it is > > handling both the single consumer use case and the administrative use > case. > > I wonder we could make it consistent from a batching perspective. > > > > In other words, instead of this: > > LeaveGroupRequest => GroupId MemberId [GroupInstanceId] > > > > Maybe we could do this: > > LeaveGroupRequest => GroupId [GroupInstanceId MemberId] > > > > For dynamic members, GroupInstanceId could be empty, which is consistent > > with JoinGroup. What do you think? > > > > Also, just for clarification, what is the expected behavior if the > current > > memberId of a static member is passed to LeaveGroup? Will the static > member > > be removed? I know the consumer will not do this, but we'll still have to > > handle the case on the broker. > > > > Best, > > Jason > > > > > > On Mon, Dec 10, 2018 at 11:54 PM Boyang Chen <bche...@outlook.com> > wrote: > > > > > Thanks Stanislav！ > > > > > > Get Outlook for iOS<https://aka.ms/o0ukef> > > > > > > ________________________________ > > > From: Stanislav Kozlovski <stanis...@confluent.io> > > > Sent: Monday, December 10, 2018 11:28 PM > > > To: dev@kafka.apache.org > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > reduce consumer rebalances > > > > > > This is great work, Boyang. Thank you very much. > > > > > > +1 (non-binding) > > > > > > On Mon, Dec 10, 2018 at 6:09 PM Boyang Chen <bche...@outlook.com> > wrote: > > > > > > > Hey there, could I get more votes on this thread? > > > > > > > > Thanks for the vote from Mayuresh and Mike :) > > > > > > > > Best, > > > > Boyang > > > > ________________________________ > > > > From: Mayuresh Gharat <gharatmayures...@gmail.com> > > > > Sent: Thursday, December 6, 2018 10:53 AM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [VOTE] KIP-345: Introduce static membership protocol to > > > > reduce consumer rebalances > > > > > > > > +1 (non-binding) > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Tue, Dec 4, 2018 at 6:58 AM Mike Freyberger < > > > mike.freyber...@xandr.com> > > > > wrote: > > > > > > > > > +1 (non binding) > > > > > > > > > > On 12/4/18, 9:43 AM, "Patrick Williams" < > > > patrick.willi...@storageos.com > > > > > > > > > > wrote: > > > > > > > > > > Pls take me off this VOTE list > > > > > > > > > > Best, > > > > > > > > > > Patrick Williams > > > > > > > > > > Sales Manager, UK & Ireland, Nordics & Israel > > > > > StorageOS > > > > > +44 (0)7549 676279 > > > > > patrick.willi...@storageos.com > > > > > > > > > > 20 Midtown > > > > > 20 Proctor Street > > > > > Holborn > > > > > London WC1V 6NX > > > > > > > > > > Twitter: @patch37 > > > > > LinkedIn: linkedin.com/in/patrickwilliams4 < > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Flinkedin.com%2Fin%2Fpatrickwilliams4&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=ipDTX%2FGARrFkwZfRuOY0M5m3iJ%2Bnkxovv6u9bBDaTyc%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fslack.storageos.com%2F&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=hxuKU6aZdQU%2FpxpqaaThR6IjpEmwIP5%2F3NhYzMYijkw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > On 03/12/2018, 17:34, "Guozhang Wang" <wangg...@gmail.com> wrote: > > > > > > > > > > Hello Boyang, > > > > > > > > > > I've browsed through the new wiki and there are still a couple of > > > > > minor > > > > > things to notice: > > > > > > > > > > 1. RemoveMemberFromGroupOptions seems not defined anywhere. > > > > > > > > > > 2. LeaveGroupRequest added a list of group instance id, but still > > > > > keep the > > > > > member id as a singleton; is that intentional? I think to make > > > > the > > > > > protocol > > > > > consistent both member id and instance ids could be plural. > > > > > > > > > > 3. About the *kafka-remove-member-from-group.sh *tool, I'm > > > > > wondering if we > > > > > can defer adding this while just add the corresponding calls of > > > > the > > > > > LeaveGroupRequest inside Streams until we have used it in > > > > > production and > > > > > hence have a better understanding on how flexible or extensible > > > > if > > > > > we want > > > > > to add any cmd tools. The rationale is that if we do not > > > > > necessarily need > > > > > it now, we can always add it later with a more think-through API > > > > > design, > > > > > but if we add the tool in a rush, we may need to extend or modify > > > > > it soon > > > > > after we realize its limits in operations. > > > > > > > > > > Otherwise, I'm +1 on the proposal. > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > On Mon, Dec 3, 2018 at 9:14 AM Boyang Chen <bche...@outlook.com> > > > > > wrote: > > > > > > > > > > > Hey community friends, > > > > > > > > > > > > after another month of polishing, KIP-345< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > design is ready for vote. Feel free to add your comment on the > > > > > discussion > > > > > > thread or here. > > > > > > > > > > > > Thanks for your time! > > > > > > > > > > > > Boyang > > > > > > ________________________________ > > > > > > From: Boyang Chen <bche...@outlook.com> > > > > > > Sent: Friday, November 9, 2018 6:35 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: [VOTE] KIP-345: Introduce static membership protocol > > > > to > > > > > reduce > > > > > > consumer rebalances > > > > > > > > > > > > Hey all, > > > > > > > > > > > > > > > > > > thanks so much for all the inputs on KIP-345 so far. The > > > > > original proposal > > > > > > has enhanced a lot with your help. To make sure the > > > > > implementation go > > > > > > smoothly without back and forth, I would like to start a vote > > > > on > > > > > the final > > > > > > design agreement now: > > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=g4%2BMXKpkiQLZXg5HJWfJhw1kc1PbDNwyiX9zkREVqGE%3D&reserved=0 > > > > < > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > > > > > > > KIP-345: Introduce static membership protocol to reduce ...< > > > > > > > > > > > > > > > > > > > > > https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-345%253A%2BIntroduce%2Bstatic%2Bmembership%2Bprotocol%2Bto%2Breduce%2Bconsumer%2Brebalances&data=02%7C01%7C%7C9b12ec4ce9ae4454db8a08d65f3a4862%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636801101252994092&sdata=T4i7L1i0nIeHrrjTeLOOgYKsfzfNEMGDhTazvBEZbXw%3D&reserved=0 > > > > > > > > > > > > > cwiki.apache.org > > > > > > For stateful applications, one of the biggest performance > > > > > bottleneck is > > > > > > the state shuffling. In Kafka consumer, there is a concept > > > > called > > > > > > "rebalance" which means that for given M partitions and N > > > > > consumers in one > > > > > > consumer group, Kafka will try to balance the load between > > > > > consumers and > > > > > > ideally have ... > > > > > > > > > > > > > > > > > > Let me know if you have any questions. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Boyang > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > -- > > > Best, > > > Stanislav > > > > > > > > -- > -- Guozhang >
