Hi all, If there are no further comments. I plan to start a vote thread on Monday.
Best, David On Wed, Jan 26, 2022 at 3:16 PM David Jacot <dja...@confluent.io> wrote: > > Hi Hector, > > Thanks for your comments. > > The idea is to add a flag to the onJoinLeader leader method. We > must do this because the ConsumerCoordinator must be able > to reconstruct its state while skipping the assignment part. That > would definitely impact KIP-795. I need to look at your KIP. > > That's right. We are thinking about a new consumer protocol but > we don't have anything concrete to share with the community yet. > We hope to have something in the near future. > > Best, > David > > On Wed, Jan 26, 2022 at 3:10 PM David Jacot <dja...@confluent.io> wrote: > > > > Hey Jason, > > > > I've updated the KIP based on your comments. Thanks for your > > feedback! > > > > Best, > > David > > > > On Wed, Jan 26, 2022 at 2:52 AM Jason Gustafson > > <ja...@confluent.io.invalid> wrote: > > > > > > Hey David, > > > > > > Yeah, there might not be a simple option to address the scenario I > > > described. Other than a broker-side solution, another way we could fix it > > > is by adding additional metadata to the assignment. I do agree that it > > > might not be worth the additional complexity. At least we should probably > > > update the KIP to describe the limitation. > > > > > > @Hector In regard to the new consumer protocol, it's only just beyond > > > wishful thinking at this point. We are hoping to share some ideas with the > > > community in the near future though. > > > > > > Best, > > > Jason > > > > > > On Mon, Jan 24, 2022 at 3:06 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > hgerald...@bloomberg.net> wrote: > > > > > > > Hi David, > > > > > > > > Is the idea here to skip calling performAssignment(...) on the > > > > AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean > > > > parameter to the performAssignment(...) method? The reason I ask is > > > > because > > > > I raised KIP-795 a few weeks back, which aims to add a public API for > > > > AbstractCoordinator, which might change (or not) with this KIP. > > > > > > > > I see you also mentioned there's some discussions regarding a new > > > > consumer > > > > protocol. Is this being discussed somewhere else? I'm curious to know > > > > how > > > > would it work with other systems (like Kafka Connect or Schema Registry) > > > > that rely on the rebalance protocol to handle resource assignments. > > > > > > > > Apologies in advance if these questions are off-topic for the discussion > > > > at hand. > > > > > > > > Regards, > > > > Hector > > > > > > > > From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: > > > > dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let > > > > the > > > > leader skip assignment > > > > > > > > Hey Jason, > > > > > > > > Thanks for your comments. > > > > > > > > Regarding your first point. Yes, you have it right. Let me complement > > > > the KIP to be clearer. > > > > > > > > Regarding your second point. That is right. New partitions would not > > > > be detected while the leader is down. It is definitely not ideal but > > > > that > > > > seems acceptable to me, at least as a first step. Adding partitions to > > > > a topic is an infrequent event so the likelihood of having it while the > > > > leader is down is rather low but that could happen. > > > > > > > > The only way to not suffer from this would be to monitor the metadata > > > > changes on the broker side. This implies that we would parse both the > > > > subscriptions and the assignments in order to have the full list of > > > > topics. > > > > I am not sure that it is worth doing it at the moment given that we are > > > > thinking about a new consumer protocol. What do you think? > > > > > > > > I suppose that we would need both in the long term as the current > > > > protocol > > > > is a bit weird at the moment so we need to fix it anyway. We could > > > > use this KIP to fix the protocol and do a subsequent KIP in the future > > > > for > > > > the server side monitoring if we need it. > > > > > > > > Best, > > > > David > > > > > > > > On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson > > > > <ja...@confluent.io.invalid> wrote: > > > > > > > > > > Hey David, > > > > > > > > > > Thanks for the proposal. This was a tricky bug and I think your > > > > > approach > > > > is > > > > > probably the best way forward. > > > > > > > > > > It would be helpful to add a little more detail to the proposal. When > > > > > the > > > > > coordinator detects that the static leader is returning, it will set > > > > > `skipAssignment` to true in the `JoinGroup` response. I believe the > > > > intent > > > > > is to return all member subscriptions in this response so that the > > > > > leader > > > > > can monitor all topics subscribed in the group (which might be > > > > > different > > > > > from the consumer's own subscription). The leader will then send an > > > > > empty > > > > > `SyncGroup` request to collect its own assignment. Do I have that > > > > > right? > > > > > > > > > > I think there might still be an edge case in this proposal (assuming > > > > > I've > > > > > understood it correctly). In between the time that the leader shuts > > > > > down > > > > > and is restarted, it is possible that new partitions are added to one > > > > > of > > > > > the subscribed topics. The returning leader would not know about it > > > > > because it has no way to collect the full assignment. Do you think > > > > > this > > > > is > > > > > a problem? > > > > > > > > > > Thanks, > > > > > Jason > > > > > > > > > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot <da...@apache.org> wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > I'd like to start a discussion for KIP-814: Static membership > > > > > > protocol > > > > > > should let the > > > > > > leader skip assignment. This is a small extension to the static > > > > > > membership protocol > > > > > > to address KAFKA-13435. > > > > > > > > > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > > > > > > > > > Please let me know what you think. > > > > > > > > > > > > Best, > > > > > > David > > > > > > > > > > > > > > > > > >
