Hey Jason, I've updated the KIP based on your comments. Thanks for your feedback!
Best, David On Wed, Jan 26, 2022 at 2:52 AM Jason Gustafson <ja...@confluent.io.invalid> wrote: > > Hey David, > > Yeah, there might not be a simple option to address the scenario I > described. Other than a broker-side solution, another way we could fix it > is by adding additional metadata to the assignment. I do agree that it > might not be worth the additional complexity. At least we should probably > update the KIP to describe the limitation. > > @Hector In regard to the new consumer protocol, it's only just beyond > wishful thinking at this point. We are hoping to share some ideas with the > community in the near future though. > > Best, > Jason > > On Mon, Jan 24, 2022 at 3:06 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> wrote: > > > Hi David, > > > > Is the idea here to skip calling performAssignment(...) on the > > AbstractCoordinator.onJoinLeader(...) method, or adding a new boolean > > parameter to the performAssignment(...) method? The reason I ask is because > > I raised KIP-795 a few weeks back, which aims to add a public API for > > AbstractCoordinator, which might change (or not) with this KIP. > > > > I see you also mentioned there's some discussions regarding a new consumer > > protocol. Is this being discussed somewhere else? I'm curious to know how > > would it work with other systems (like Kafka Connect or Schema Registry) > > that rely on the rebalance protocol to handle resource assignments. > > > > Apologies in advance if these questions are off-topic for the discussion > > at hand. > > > > Regards, > > Hector > > > > From: dev@kafka.apache.org At: 01/24/22 09:08:58 UTC-5:00To: > > dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-814: Static membership protocol should let the > > leader skip assignment > > > > Hey Jason, > > > > Thanks for your comments. > > > > Regarding your first point. Yes, you have it right. Let me complement > > the KIP to be clearer. > > > > Regarding your second point. That is right. New partitions would not > > be detected while the leader is down. It is definitely not ideal but that > > seems acceptable to me, at least as a first step. Adding partitions to > > a topic is an infrequent event so the likelihood of having it while the > > leader is down is rather low but that could happen. > > > > The only way to not suffer from this would be to monitor the metadata > > changes on the broker side. This implies that we would parse both the > > subscriptions and the assignments in order to have the full list of topics. > > I am not sure that it is worth doing it at the moment given that we are > > thinking about a new consumer protocol. What do you think? > > > > I suppose that we would need both in the long term as the current protocol > > is a bit weird at the moment so we need to fix it anyway. We could > > use this KIP to fix the protocol and do a subsequent KIP in the future for > > the server side monitoring if we need it. > > > > Best, > > David > > > > On Fri, Jan 21, 2022 at 7:51 PM Jason Gustafson > > <ja...@confluent.io.invalid> wrote: > > > > > > Hey David, > > > > > > Thanks for the proposal. This was a tricky bug and I think your approach > > is > > > probably the best way forward. > > > > > > It would be helpful to add a little more detail to the proposal. When the > > > coordinator detects that the static leader is returning, it will set > > > `skipAssignment` to true in the `JoinGroup` response. I believe the > > intent > > > is to return all member subscriptions in this response so that the leader > > > can monitor all topics subscribed in the group (which might be different > > > from the consumer's own subscription). The leader will then send an empty > > > `SyncGroup` request to collect its own assignment. Do I have that right? > > > > > > I think there might still be an edge case in this proposal (assuming I've > > > understood it correctly). In between the time that the leader shuts down > > > and is restarted, it is possible that new partitions are added to one of > > > the subscribed topics. The returning leader would not know about it > > > because it has no way to collect the full assignment. Do you think this > > is > > > a problem? > > > > > > Thanks, > > > Jason > > > > > > On Wed, Jan 19, 2022 at 7:27 AM David Jacot <da...@apache.org> wrote: > > > > > > > Hi folks, > > > > > > > > I'd like to start a discussion for KIP-814: Static membership protocol > > > > should let the > > > > leader skip assignment. This is a small extension to the static > > > > membership protocol > > > > to address KAFKA-13435. > > > > > > > > The KIP is here: https://cwiki.apache.org/confluence/x/C5-kCw. > > > > > > > > Please let me know what you think. > > > > > > > > Best, > > > > David > > > > > > > > > >
