Hi Hector, Thanks for your comments.
The idea is to add a flag to the onJoinLeader leader method. We must do this because the ConsumerCoordinator must be able to reconstruct its state while skipping the assignment part. That would definitely impact KIP-795. I need to look at your KIP. That's right. We are thinking about a new consumer protocol but we don't have anything concrete to share with the community yet. We hope to have something in the near future. Best, David On Wed, Jan 26, 2022 at 3:10 PM David Jacot <dja...@confluent.io> wrote: > > Hey Jason, > > I've updated the KIP based on your comments. Thanks for your > feedback! > > Best, > David > > On Wed, Jan 26, 2022 at 2:52 AM Jason Gustafson > <ja...@confluent.io.invalid> wrote: > > > > Hey David, > > > > Yeah, there might not be a simple option to address the scenario I > > described. Other than a broker-side solution, another way we could fix it > > is by adding additional metadata to the assignment. I do agree that it > > might not be worth the additional complexity. At least we should probably > > update the KIP to describe the limitation. > > > > @Hector In regard to the new consumer protocol, it's only just beyond > > wishful thinking at this point. We are hoping to share some ideas with the > > community in the near future though. > > > > Best, > > Jason > > > > On Mon, Jan 24, 2022 at 3:06 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > hgerald...@bloomberg.net> wrote: > > > > > Hi David, > > > > > > Is the idea here to skip calling performAssignment(...) on the > > > AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean > > > parameter to the performAssignment(...) method? The reason I ask is > > > because > > > I raised KIP-795 a few weeks back, which aims to add a public API for > > > AbstractCoordinator, which might change (or not) with this KIP. > > > > > > I see you also mentioned there's some discussions regarding a new consumer > > > protocol. Is this being discussed somewhere else? I'm curious to know how > > > would it work with other systems (like Kafka Connect or Schema Registry) > > > that rely on the rebalance protocol to handle resource assignments. > > > > > > Apologies in advance if these questions are off-topic for the discussion > > > at hand. > > > > > > Regards, > > > Hector > > > > > > From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: > > > dev@kafka.apache.org > > > Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let the > > > leader skip assignment > > > > > > Hey Jason, > > > > > > Thanks for your comments. > > > > > > Regarding your first point. Yes, you have it right. Let me complement > > > the KIP to be clearer. > > > > > > Regarding your second point. That is right. New partitions would not > > > be detected while the leader is down. It is definitely not ideal but that > > > seems acceptable to me, at least as a first step. Adding partitions to > > > a topic is an infrequent event so the likelihood of having it while the > > > leader is down is rather low but that could happen. > > > > > > The only way to not suffer from this would be to monitor the metadata > > > changes on the broker side. This implies that we would parse both the > > > subscriptions and the assignments in order to have the full list of > > > topics. > > > I am not sure that it is worth doing it at the moment given that we are > > > thinking about a new consumer protocol. What do you think? > > > > > > I suppose that we would need both in the long term as the current protocol > > > is a bit weird at the moment so we need to fix it anyway. We could > > > use this KIP to fix the protocol and do a subsequent KIP in the future for > > > the server side monitoring if we need it. > > > > > > Best, > > > David > > > > > > On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson > > > <ja...@confluent.io.invalid> wrote: > > > > > > > > Hey David, > > > > > > > > Thanks for the proposal. This was a tricky bug and I think your approach > > > is > > > > probably the best way forward. > > > > > > > > It would be helpful to add a little more detail to the proposal. When > > > > the > > > > coordinator detects that the static leader is returning, it will set > > > > `skipAssignment` to true in the `JoinGroup` response. I believe the > > > intent > > > > is to return all member subscriptions in this response so that the > > > > leader > > > > can monitor all topics subscribed in the group (which might be different > > > > from the consumer's own subscription). The leader will then send an > > > > empty > > > > `SyncGroup` request to collect its own assignment. Do I have that right? > > > > > > > > I think there might still be an edge case in this proposal (assuming > > > > I've > > > > understood it correctly). In between the time that the leader shuts down > > > > and is restarted, it is possible that new partitions are added to one of > > > > the subscribed topics. The returning leader would not know about it > > > > because it has no way to collect the full assignment. Do you think this > > > is > > > > a problem? > > > > > > > > Thanks, > > > > Jason > > > > > > > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot <da...@apache.org> wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I'd like to start a discussion for KIP-814: Static membership protocol > > > > > should let the > > > > > leader skip assignment. This is a small extension to the static > > > > > membership protocol > > > > > to address KAFKA-13435. > > > > > > > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > > > > > > > Please let me know what you think. > > > > > > > > > > Best, > > > > > David > > > > > > > > > > > > > >
