Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi all, I was looking at the OffsetFetchRequest in the context of KIP-848 and I noticed something that I would like to discuss. In the current implementation, when a group is specified multiple times in the request, we ignore all occurrences but the last one. That is because all groups are put in

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi all, Following the PR discussion on this KIP's Admin API PR ( https://github.com/apache/kafka/pull/10964), we have changed the Admin API proposed in this KIP to use a separate ListConsumerGroupOffsetsSpec class to specify per-group spec including partitions for which offsets are fetched. The n

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hello Everyone, I recently opened a PR for KIP-709 and it was pointed out that we still need to come to a consensus on the Admin APIs. Specifically the concern was around the `ListConsumerGroupOffsetsOptions` class. Currently that class contains a List that acts as a filter for the specific top

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Everyone, I’ve begun working on this KIP now and found that another class will be needing public changes. I have updated the KIP to reflect this, so just wanted to update the dev list as well. You can find the updated KIP here:  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Magnus, Thanks for the review, I've added //moved and explanation as requested. Thanks Tom On Wed, Jan 27, 2021 at 12:05 PM Magnus Edenhill wrote: > Hey Thomas, > > I'm late to the game. > > It looks like the "top level" ErrorCode moved from the top-level to the > Group array, which m

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hey Thomas, I'm late to the game. It looks like the "top level" ErrorCode moved from the top-level to the Group array, which makes sense, but it would be good if it was marked as // MOVED in the KIP and also a note that top level errors that are unrelated to the group will be returned as per-grou

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Thanks David I've updated it. On Tue, Jan 26, 2021 at 1:55 PM David Jacot wrote: > Great. That answers my question! > > Thomas, I suggest adding a Related/Future Work section in the > KIP to link KIP-699 more explicitly. > > Thanks, > David > > On Tue, Jan 26, 2021 at 1:30 PM Thomas Scott wrote

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Great. That answers my question! Thomas, I suggest adding a Related/Future Work section in the KIP to link KIP-699 more explicitly. Thanks, David On Tue, Jan 26, 2021 at 1:30 PM Thomas Scott wrote: > Hi Mickael/David, > > I feel like the combination of these 2 KIPs gives the complete solutio

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Mickael/David, I feel like the combination of these 2 KIPs gives the complete solution but they can be implemented independently. I have added a description and links to KIP-699 to KIP-709 to this effect. Thanks Tom On Tue, Jan 26, 2021 at 11:44 AM Mickael Maison wrote: > Hi Thomas, >

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Thomas, Thanks, the KIP looks good. David, I started working on exactly that a few weeks ago: https://cwiki.apache.org/confluence/display/KAFKA/KIP-699%3A+FindCoordinators I hope to complete my draft and start a discussion later on this week. Thanks On Tue, Jan 26, 2021 at 10:06 AM David Jaco

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Thomas, Thanks for the KIP. Overall, the KIP looks good to me. I have only one question: The FindCoordinator API only supports resolving one group id at the time. If we want to get the offsets for say N groups, that means that we have to first issue N FindCoordinator requests, wait for the res

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi Thomas, Thanks for the KIP, this is a useful addition for admin use cases. It may be worth starting the voting thread soon if we want to get this into 2.8.0. Regards, Rajini On Mon, Jan 25, 2021 at 1:52 PM Thomas Scott wrote: > Thanks Ismael, that's a lot better. I've updated the KIP with

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Thanks Ismael, that's a lot better. I've updated the KIP with this behaviour instead. On Mon, Jan 25, 2021 at 11:42 AM Ismael Juma wrote: > Thanks for the KIP, Thomas. One question below: > > Should an Admin client with this new functionality be used against an old > > broker that cannot handle

Re: [DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Thanks for the KIP, Thomas. One question below: Should an Admin client with this new functionality be used against an old > broker that cannot handle these requests then the methods will throw > UnsupportedVersionException as per the usual pattern. Did we consider automatically falling back to t

[DISCUSS] KIP-709: Extend OffsetFetch requests to accept multiple group ids

Hi, I'm starting this thread to discuss KIP-709 to extend OffsetFetch requests to accept multiple group ids. Please check out the KIP here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=173084258 Any comments much appreciated. thanks, Tom