Hi, As we've written the code for KIP-932, we've made some small changes to the KIP to correct mistakes, align with other accepted KIPs and fix some omissions picked up during PR reviews. We also have experience of people writing applications against the ShareConsumer interface and have corrected a couple of problems.
* Using share.version=1 as the switch to enable share groups. This aligns with group.version=1 for KIP-848 and streams.version=1 for KIP-1071. * Aligned with KIP-1043 by using a common GroupState and removing Admin.listShareGroups (because Admin.listGroups handles all group types). * Corrected the topic authorizations on admin requests. * Added SHARE_SESSION_LIMIT_REACHED error code to give proper consumer error handling when the limit is reached. * Adjusted ShareFetch RPC request to remove PartitionMaxBytes and add BatchSize to improve efficiency of fetching and state management. * Changed the rules for explicit acknowledgement because, as originally written, it was simple to write an application that went into an infinite loop. Now, when using explicit acknowledgement, it is mandatory to acknowledge every single record, which allows an exception to be thrown if the application does not do this properly. * Added a method overload with signature ShareConsumer.acknowledge(String, int, long, AcknowledgeType) to go with ShareConsumer.acknowledge(ConsumerRecord, AcknowledgeType). This is because the KIP said it would be possible to choose the acknowledge type in the event of RecordDeserializationException. However, because the application does not have the ConsumerRecord instance when the deserialization fails, but it does have topic name, partition and offset, adding an overloaded method enables the application to choose the acknowledge type. * The SimpleAssignor no longer uses hashing as part of making assignments. In practice, this hindered the assignment computation. SimpleAssignor achieves the balance and stickiness described previously, just without hashing. * Added org.apache.kafka.coordinator.group.api.assignor.GroupSpec.isPartitionAssignable so that partitions which are in the process of being initialized in the share coordinator can be omitted from assignment computation. * Removed the ShareGroupPartitionMetadata record type (KIP-1101). * Added timestamps to ShareSnapshot records so it is possible to manage pruning of records correctly. * Adjusted some of the config defaults and limits such as maximum for lock duration which didn't make sense. KIP-932 mentions some future work and I have started to create new KIPs for this. Thanks, Andrew ________________________________________ From: Jun Rao <j...@confluent.io.INVALID> Sent: 15 August 2024 19:07 To: dev@kafka.apache.org <dev@kafka.apache.org> Subject: Re: [VOTE] KIP-932: Queues for Kafka Hi, Andrew, Thanks for the clarification. Sounds good. Jun On Thu, Aug 15, 2024 at 2:53 AM Andrew Schofield <andrew_schofi...@live.com> wrote: > Hi Jun, > Thanks for the questions. > > 1. The PartitionMetadata is only stored if there is rack ID information. > So, there are situations in which it’s redundant, but almost always the > PartitionMetadata will be empty. > > 2. They are for subtly different purposes. ShareGroupPartitionMetadata > is essentially a persistent cache of the metadata for topic-partitions > which > can be assigned to group members. ShareGroupStatePartitionMetadata > is keeping track of which partitions have share-group state in the share > coordinator. I prefer having them separate, in particular because it means > the ShareGroupPartitionMetadata matches nicely with consumer groups, > and also because share groups are able to work without a share coordinator > (the persister is pluggable). > > I propose that as we develop the code in the group coordinator which uses > the various ShareGroupState RPCs to talk to the share coordinator, > we will confirm whether the record schemas are optimal > and make a further refinement if required. Let me know if that’s OK. > > Thanks, > Andrew > > > > On 14 Aug 2024, at 00:00, Jun Rao <j...@confluent.io.INVALID> wrote: > > > > Hi, Andrew, > > > > Thanks for the update. A couple of followup questions. > > > > 1. ShareGroupPartitionMetadataValue: Is NumPartitions redundant given > > PartitionMetadata? > > > > 2. I assume that ShareGroupPartitionMetadataValue contains initialized > > partitions. Does that > > make ShareGroupStatePartitionMetadataValue.InitializedTopics redundant? > > > > Jun > > > > On Tue, Aug 13, 2024 at 11:56 AM Andrew Schofield < > andrew_schofi...@live.com> > > wrote: > > > >> Hi, > >> During the implementation of the group management code for KIP-932, > >> we learnt that the decision not to persist assignments for share groups > was > >> making it difficult to fit with the coordinator framework, in particular > >> the > >> use of timeline structures. As a result, I decided to change the KIP to > >> follow the consumer group record scheme much more closely. > >> > >> Now, share groups do persist assignments and member epochs which > >> resolves the problems we were experiencing managing soft state in the > >> coordinator and making group membership reliable across coordinator > >> restarts. > >> > >> I have updated the KIP accordingly and the revised code has been merged. > >> If anyone has any comments, please let me know. > >> > >> Thanks, > >> Andrew > >> > >> > >>> On 21 May 2024, at 15:35, Andrew Schofield <andrew_schofi...@live.com> > >> wrote: > >>> > >>> Hi Jun, > >>> All the client metrics are standard. None are required. > >>> > >>> I’ve updated the KIP accordingly. > >>> > >>> Thanks, > >>> Andrew > >>> > >>>> On 15 May 2024, at 21:36, Jun Rao <j...@confluent.io.INVALID> wrote: > >>>> > >>>> Hi, Andrew, > >>>> > >>>> Thanks for the update. Should we mark whether those metrics are > >>>> standard/required for KIP-714? > >>>> > >>>> Jun > >>>> > >>>> On Tue, May 14, 2024 at 7:31 AM Andrew Schofield < > >> andrew_schofi...@live.com> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> I have made a small update to the KIP as a result of testing the new > >>>>> share consumer with client telemetry (KIP-714). > >>>>> > >>>>> I’ve added telemetry metric names to the table of client metrics and > >>>>> also updated the metric group names so that the resulting client > >> metrics > >>>>> sent to the broker have consistent names. > >>>>> > >>>>> Thanks, > >>>>> Andrew > >>>>> > >>>>>> On 8 May 2024, at 12:51, Manikumar <manikumar.re...@gmail.com> > wrote: > >>>>>> > >>>>>> Hi Andrew, > >>>>>> > >>>>>> Thanks for the KIP. Great write-up! > >>>>>> > >>>>>> +1 (binding) > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> On Wed, May 8, 2024 at 12:17 PM Satish Duggana < > >> satish.dugg...@gmail.com> > >>>>> wrote: > >>>>>>> > >>>>>>> Hi Andrew, > >>>>>>> Thanks for the nice KIP, it will allow other messaging use cases to > >> be > >>>>>>> onboarded to Kafka. > >>>>>>> > >>>>>>> +1 from me. > >>>>>>> > >>>>>>> Satish. > >>>>>>> > >>>>>>> On Tue, 7 May 2024 at 03:41, Jun Rao <j...@confluent.io.invalid> > >> wrote: > >>>>>>>> > >>>>>>>> Hi, Andrew, > >>>>>>>> > >>>>>>>> Thanks for the KIP. +1 > >>>>>>>> > >>>>>>>> Jun > >>>>>>>> > >>>>>>>> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar < > >> edoardli...@gmail.com> > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Thanks Andrew, > >>>>>>>>> > >>>>>>>>> +1 (binding) > >>>>>>>>> > >>>>>>>>> Edo > >>>>>>>>> > >>>>>>>>> On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole > >>>>>>>>> <kevers...@cloudflare.com.invalid> wrote: > >>>>>>>>>> > >>>>>>>>>> Hi Andrew > >>>>>>>>>> > >>>>>>>>>> + 1 (Non-Binding) > >>>>>>>>>> > >>>>>>>>>> This will be great addition to Kafka > >>>>>>>>>> > >>>>>>>>>> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal < > >>>>> apoorvmitta...@gmail.com> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Andrew, > >>>>>>>>>>> Thanks for writing the KIP. This is indeed going to be a > valuable > >>>>>>>>> addition > >>>>>>>>>>> to the Kafka, excited to see the KIP. > >>>>>>>>>>> > >>>>>>>>>>> + 1 (Non-Binding) > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Apoorv Mittal > >>>>>>>>>>> +44 7721681581 > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield < > >>>>>>>>>>> andrew_schofield_j...@outlook.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi, > >>>>>>>>>>>> I’ve been working to complete KIP-932 over the past few months > >> and > >>>>>>>>>>>> discussions have quietened down. > >>>>>>>>>>>> > >>>>>>>>>>>> I’d like to open the voting for KIP-932: > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks, > >>>>>>>>>>>> Andrew > > >
