Hi David, Thanks for the KIP! The solution makes sense to me.
One thing to clarify. You mentioned in the KIP that: > (After leader rejoin) Then the leader will send an *empty SyncGroup* to collect its own (and existing) assignment. I'd like to confirm that the *empty SyncGroup* is meaning the `assignment` field of the request is empty. Other fields are still filled with correct information. Is that right? I think you should make it clear in the KIP. Thank you. Luke On Fri, Jan 28, 2022 at 8:00 PM David Jacot <dja...@confluent.io.invalid> wrote: > Hi all, > > If there are no further comments. I plan to start a vote thread > on Monday. > > Best, > David > > On Wed, Jan 26, 2022 at 3:16 PM David Jacot <dja...@confluent.io> wrote: > > > > Hi Hector, > > > > Thanks for your comments. > > > > The idea is to add a flag to the onJoinLeader leader method. We > > must do this because the ConsumerCoordinator must be able > > to reconstruct its state while skipping the assignment part. That > > would definitely impact KIP-795. I need to look at your KIP. > > > > That's right. We are thinking about a new consumer protocol but > > we don't have anything concrete to share with the community yet. > > We hope to have something in the near future. > > > > Best, > > David > > > > On Wed, Jan 26, 2022 at 3:10 PM David Jacot <dja...@confluent.io> wrote: > > > > > > Hey Jason, > > > > > > I've updated the KIP based on your comments. Thanks for your > > > feedback! > > > > > > Best, > > > David > > > > > > On Wed, Jan 26, 2022 at 2:52 AM Jason Gustafson > > > <ja...@confluent.io.invalid> wrote: > > > > > > > > Hey David, > > > > > > > > Yeah, there might not be a simple option to address the scenario I > > > > described. Other than a broker-side solution, another way we could > fix it > > > > is by adding additional metadata to the assignment. I do agree that > it > > > > might not be worth the additional complexity. At least we should > probably > > > > update the KIP to describe the limitation. > > > > > > > > @Hector In regard to the new consumer protocol, it's only just beyond > > > > wishful thinking at this point. We are hoping to share some ideas > with the > > > > community in the near future though. > > > > > > > > Best, > > > > Jason > > > > > > > > On Mon, Jan 24, 2022 at 3:06 PM Hector Geraldino (BLOOMBERG/ 919 3RD > A) < > > > > hgerald...@bloomberg.net> wrote: > > > > > > > > > Hi David, > > > > > > > > > > Is the idea here to skip calling performAssignment(...) on the > > > > > AbstractCoordinator.onJoinLeader(...) method, or adding a new > boolean > > > > > parameter to the performAssignment(...) method? The reason I ask > is because > > > > > I raised KIP-795 a few weeks back, which aims to add a public API > for > > > > > AbstractCoordinator, which might change (or not) with this KIP. > > > > > > > > > > I see you also mentioned there's some discussions regarding a new > consumer > > > > > protocol. Is this being discussed somewhere else? I'm curious to > know how > > > > > would it work with other systems (like Kafka Connect or Schema > Registry) > > > > > that rely on the rebalance protocol to handle resource assignments. > > > > > > > > > > Apologies in advance if these questions are off-topic for the > discussion > > > > > at hand. > > > > > > > > > > Regards, > > > > > Hector > > > > > > > > > > From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: > > > > > dev@kafka.apache.org > > > > > Subject: Re: [DISCUSS] KIP-814: Static membership protocol should > let the > > > > > leader skip assignment > > > > > > > > > > Hey Jason, > > > > > > > > > > Thanks for your comments. > > > > > > > > > > Regarding your first point. Yes, you have it right. Let me > complement > > > > > the KIP to be clearer. > > > > > > > > > > Regarding your second point. That is right. New partitions would > not > > > > > be detected while the leader is down. It is definitely not ideal > but that > > > > > seems acceptable to me, at least as a first step. Adding > partitions to > > > > > a topic is an infrequent event so the likelihood of having it > while the > > > > > leader is down is rather low but that could happen. > > > > > > > > > > The only way to not suffer from this would be to monitor the > metadata > > > > > changes on the broker side. This implies that we would parse both > the > > > > > subscriptions and the assignments in order to have the full list > of topics. > > > > > I am not sure that it is worth doing it at the moment given that > we are > > > > > thinking about a new consumer protocol. What do you think? > > > > > > > > > > I suppose that we would need both in the long term as the current > protocol > > > > > is a bit weird at the moment so we need to fix it anyway. We could > > > > > use this KIP to fix the protocol and do a subsequent KIP in the > future for > > > > > the server side monitoring if we need it. > > > > > > > > > > Best, > > > > > David > > > > > > > > > > On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson > > > > > <ja...@confluent.io.invalid> wrote: > > > > > > > > > > > > Hey David, > > > > > > > > > > > > Thanks for the proposal. This was a tricky bug and I think your > approach > > > > > is > > > > > > probably the best way forward. > > > > > > > > > > > > It would be helpful to add a little more detail to the proposal. > When the > > > > > > coordinator detects that the static leader is returning, it will > set > > > > > > `skipAssignment` to true in the `JoinGroup` response. I believe > the > > > > > intent > > > > > > is to return all member subscriptions in this response so that > the leader > > > > > > can monitor all topics subscribed in the group (which might be > different > > > > > > from the consumer's own subscription). The leader will then send > an empty > > > > > > `SyncGroup` request to collect its own assignment. Do I have > that right? > > > > > > > > > > > > I think there might still be an edge case in this proposal > (assuming I've > > > > > > understood it correctly). In between the time that the leader > shuts down > > > > > > and is restarted, it is possible that new partitions are added > to one of > > > > > > the subscribed topics. The returning leader would not know about > it > > > > > > because it has no way to collect the full assignment. Do you > think this > > > > > is > > > > > > a problem? > > > > > > > > > > > > Thanks, > > > > > > Jason > > > > > > > > > > > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot <da...@apache.org> > wrote: > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > I'd like to start a discussion for KIP-814: Static membership > protocol > > > > > > > should let the > > > > > > > leader skip assignment. This is a small extension to the static > > > > > > > membership protocol > > > > > > > to address KAFKA-13435. > > > > > > > > > > > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > > > > > > > > > > > Please let me know what you think. > > > > > > > > > > > > > > Best, > > > > > > > David > > > > > > > > > > > > > > > > > > > > > > >
