Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-05-22 Thread Andrew Schofield
; Hi, Ivan, >>>>> >>>>> Thanks for the KIP. +1 >>>>> >>>>> Just a minor comment. Should we add metadata.recovery.strategy to the >>>>> Streams and the newly introduced ShareConsumer (KIP-932) too? >>>>> >&g

Re: [VOTE] KIP-932: Queues for Kafka

2024-05-21 Thread Andrew Schofield
KIP-714? > > Jun > > On Tue, May 14, 2024 at 7:31 AM Andrew Schofield > wrote: > >> Hi, >> I have made a small update to the KIP as a result of testing the new >> share consumer with client telemetry (KIP-714). >> >> I’ve added telemetry metric names t

Re: [VOTE] KIP-932: Queues for Kafka

2024-05-19 Thread Andrew Schofield
>> Best, >> David >> >> On Wed, May 15, 2024 at 10:37 PM Jun Rao wrote: >> >>> Hi, Andrew, >>> >>> Thanks for the update. Should we mark whether those metrics are >>> standard/required for KIP-714? >>> >>> Jun >>>

Re: [VOTE] KIP-932: Queues for Kafka

2024-05-14 Thread Andrew Schofield
ar 2024 at 16:32, Kenneth Eversole >>>> wrote: >>>>> >>>>> Hi Andrew >>>>> >>>>> + 1 (Non-Binding) >>>>> >>>>> This will be great addition to Kafka >>>>> >>>>> On Mon, Ma

Re: [VOTE] KIP-1038: Add Custom Error Handler to Producer

2024-05-14 Thread Andrew Schofield
Hi Alieh, Thanks for the KIP. +1 (non-binding) Andrew > On 7 May 2024, at 16:56, Alieh Saeedi wrote: > > Hi all, > > It seems that we have no more comments, discussions, or feedback on > KIP-1038; therefore, I’d like to open voting for the KIP: Add Custom Error > Handler to Producer >

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-14 Thread Andrew Schofield
logs afterward for messages like "Error while >>> fetching >>>>> metadata with correlation id : >>>> {foo-topic=UNKNOWN_TOPIC_OR_PARTITION}". >>>>> >>>>> - I also don't think we need custom XxxResponse enums for ev

[jira] [Created] (KAFKA-16759) Invalid client telemetry transition on consumer close

2024-05-14 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16759: Summary: Invalid client telemetry transition on consumer close Key: KAFKA-16759 URL: https://issues.apache.org/jira/browse/KAFKA-16759 Project: Kafka

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-13 Thread Andrew Schofield
Hi Alieh, Just a few more comments on the KIP. It is looking much less risky now the scope is tighter. [AJS1] It would be nice to have default implementations of the handle methods so an implementor would not need to implement both themselves. [AJS2] Producer configurations which are class names

[jira] [Created] (KAFKA-16734) Add support for formatting records written to share-group state topic

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16734: Summary: Add support for formatting records written to share-group state topic Key: KAFKA-16734 URL: https://issues.apache.org/jira/browse/KAFKA-16734

[jira] [Created] (KAFKA-16730) Initial code for share-group consumer

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16730: Summary: Initial code for share-group consumer Key: KAFKA-16730 URL: https://issues.apache.org/jira/browse/KAFKA-16730 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16733) Add support for formatting new records written to offsets topic

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16733: Summary: Add support for formatting new records written to offsets topic Key: KAFKA-16733 URL: https://issues.apache.org/jira/browse/KAFKA-16733 Project

[jira] [Created] (KAFKA-16732) Support for share-coordinator-metrics in the broker

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16732: Summary: Support for share-coordinator-metrics in the broker Key: KAFKA-16732 URL: https://issues.apache.org/jira/browse/KAFKA-16732 Project: Kafka

[jira] [Created] (KAFKA-16731) Support for share-group-metrics in the broker

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16731: Summary: Support for share-group-metrics in the broker Key: KAFKA-16731 URL: https://issues.apache.org/jira/browse/KAFKA-16731 Project: Kafka Issue

[jira] [Created] (KAFKA-16728) Add dynamic group configuration for heartbeat interval and session timeout

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16728: Summary: Add dynamic group configuration for heartbeat interval and session timeout Key: KAFKA-16728 URL: https://issues.apache.org/jira/browse/KAFKA-16728

[jira] [Created] (KAFKA-16729) Support for read-committed isolation level

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16729: Summary: Support for read-committed isolation level Key: KAFKA-16729 URL: https://issues.apache.org/jira/browse/KAFKA-16729 Project: Kafka Issue

[jira] [Created] (KAFKA-16726) Add dynamic group configuration for offset reset

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16726: Summary: Add dynamic group configuration for offset reset Key: KAFKA-16726 URL: https://issues.apache.org/jira/browse/KAFKA-16726 Project: Kafka

[jira] [Created] (KAFKA-16725) Add broker configurations

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16725: Summary: Add broker configurations Key: KAFKA-16725 URL: https://issues.apache.org/jira/browse/KAFKA-16725 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16727) Add dynamic group configuration for record lock duration

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16727: Summary: Add dynamic group configuration for record lock duration Key: KAFKA-16727 URL: https://issues.apache.org/jira/browse/KAFKA-16727 Project: Kafka

[jira] [Created] (KAFKA-16724) Add new options for kafka-producer-perf-test.sh

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16724: Summary: Add new options for kafka-producer-perf-test.sh Key: KAFKA-16724 URL: https://issues.apache.org/jira/browse/KAFKA-16724 Project: Kafka

[jira] [Created] (KAFKA-16722) Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16722: Summary: Add ConsumerGroupPartitionAssignor and ShareGroupPartitionAssignor Key: KAFKA-16722 URL: https://issues.apache.org/jira/browse/KAFKA-16722 Project

[jira] [Created] (KAFKA-16723) Add kafka-console-share-consumer.sh

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16723: Summary: Add kafka-console-share-consumer.sh Key: KAFKA-16723 URL: https://issues.apache.org/jira/browse/KAFKA-16723 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-16721) Add exceptions for the new error codes

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16721: Summary: Add exceptions for the new error codes Key: KAFKA-16721 URL: https://issues.apache.org/jira/browse/KAFKA-16721 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16718) Add AdminClient.deleteShareGroupOffsets

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16718: Summary: Add AdminClient.deleteShareGroupOffsets Key: KAFKA-16718 URL: https://issues.apache.org/jira/browse/KAFKA-16718 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16719) Add AdminClient.deleteShareGroups

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16719: Summary: Add AdminClient.deleteShareGroups Key: KAFKA-16719 URL: https://issues.apache.org/jira/browse/KAFKA-16719 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-16717) Add AdminClient.alterShareGroupOffsets

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16717: Summary: Add AdminClient.alterShareGroupOffsets Key: KAFKA-16717 URL: https://issues.apache.org/jira/browse/KAFKA-16717 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16720) Add AdminClient.listShareGroupOffsets

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16720: Summary: Add AdminClient.listShareGroupOffsets Key: KAFKA-16720 URL: https://issues.apache.org/jira/browse/KAFKA-16720 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16716) Add AdminClient.describeShareGroups and AdminClient.listShareGroups

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16716: Summary: Add AdminClient.describeShareGroups and AdminClient.listShareGroups Key: KAFKA-16716 URL: https://issues.apache.org/jira/browse/KAFKA-16716 Project

[jira] [Created] (KAFKA-16715) Create KafkaShareConsumer interface

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16715: Summary: Create KafkaShareConsumer interface Key: KAFKA-16715 URL: https://issues.apache.org/jira/browse/KAFKA-16715 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-16714) kafka-share-groups.sh supporting list and describe

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16714: Summary: kafka-share-groups.sh supporting list and describe Key: KAFKA-16714 URL: https://issues.apache.org/jira/browse/KAFKA-16714 Project: Kafka

[jira] [Created] (KAFKA-16713) Add new RPC definitions

2024-05-13 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16713: Summary: Add new RPC definitions Key: KAFKA-16713 URL: https://issues.apache.org/jira/browse/KAFKA-16713 Project: Kafka Issue Type: Sub-task

[jira] [Resolved] (KAFKA-16676) Security docs missing RPCs from KIP-714 and KIP-1000

2024-05-09 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-16676. -- Resolution: Fixed > Security docs missing RPCs from KIP-714 and KIP-1

Re: [VOTE] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

2024-05-08 Thread Andrew Schofield
Hi, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 8 May 2024, at 15:48, David Jacot wrote: > > Hi folks, > > I'd like to start a voting thread for KIP-1041: Drop > `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8). > > KIP: https://cwiki.apache.org/confluence/x/9YobEg

[jira] [Created] (KAFKA-16676) Security docs missing RPCs from KIP-714 and KIP-1000

2024-05-06 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16676: Summary: Security docs missing RPCs from KIP-714 and KIP-1000 Key: KAFKA-16676 URL: https://issues.apache.org/jira/browse/KAFKA-16676 Project: Kafka

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-03 Thread Andrew Schofield
on KRaft clusters? For example, > the feature tool seems to be built only for the KRaft cluster. > > Jun > > On Fri, May 3, 2024 at 10:32 AM Andrew Schofield > wrote: > >> Hi Jun, >> Thanks for your reply. >> >> 147. Yes, I see what you mean. The

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-03 Thread Andrew Schofield
rebalance time always 0? > > 150. I guess in the common case, there will be many more members than > partitions. So the need for (b) will be less common. We can probably leave > the persisting of the assignment out for now. > > 158. The new name sounds good to me. > > Jun > >

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

2024-05-03 Thread Andrew Schofield
Hi Fred, Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a worthwhile improvement. +1 (non-binding) Thanks, Andrew > On 30 Apr 2024, at 14:02, Frédérik Rouleau > wrote: > > Hi all, > > As there is no more activity for a while on the discuss thread, I think we > can

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-02 Thread Andrew Schofield
sState and ReadShareGroupState. > The state in the latter is also offsets. > > Jun > > On Wed, May 1, 2024 at 9:51 PM Andrew Schofield > wrote: > >> Hi Jun, >> Thanks for your reply. >> >> 147. Perhaps the easiest is to take a look at the code in >

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-05-02 Thread Andrew Schofield
Hi, I’ve changed my mind on this one having read through the comments. I don’t think the exception handler should be able to mess with the headers to the detriment of the code that called the handler. While I like the hygiene of having an ImmutableHeaders interface, I feel we can use the

Re: [DISCUSS] KIP-1041: Drop `offsets.commit.required.acks` config in 4.0 (deprecate in 3.8)

2024-05-02 Thread Andrew Schofield
Hi David, I think this KIP is a very good idea. It would be good to get rid of this cruft. Thanks, Andrew > On 2 May 2024, at 18:54, David Jacot wrote: > > Hi folks, > > I have put together a very small KIP to > deprecate offsets.commit.required.acks in 3.8 and remove it in 4.0. See the >

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-05-01 Thread Andrew Schofield
> > 158. ReadShareGroupOffsetsState: The state is the offsets. Should we rename > it to something like ReadShareGroupStartOffset? > > 159. members are assigned members round-robin => members are assigned > round-robin > > 160. "may called": typo > &

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-29 Thread Andrew Schofield
mpacted topic, right? > > 143. So, the client sends DescribeShareGroupOffsets requests to GC, which > then forwards it to SC? > > 147. I guess a client only knows the rebalance triggered by itself, but not > the ones triggered by other members or topic/partition changes? &g

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-26 Thread Andrew Schofield
r do we > assume that clients will do the right thing in this case? By this, I meant > that they will stop fetching immediately. This part is not clear in the KIP > or I missed it. > > Best, > David > > On Thu, Apr 25, 2024 at 1:19 PM Andrew Schofield > wrote: > >> H

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-25 Thread Andrew Schofield
s-per-request-avg: Should we pair it with > records-per-request-max? > > 149. If the shareGroupState is not available, what error code is used in > WriteShareGroupStateResponse? > > Jun > > On Wed, Apr 24, 2024 at 7:12 AM Andrew Schofield > wrote: > >> Hi Jun

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-24 Thread Andrew Schofield
reGroupState as long as the topic still exists. The same > optimization could be applied to DeletingTopics too. > > Jun > > > On Tue, Apr 23, 2024 at 3:57 AM Andrew Schofield > wrote: > >> Hi Jun, >> Thanks for the reply. >> >> 123. Every time

[jira] [Created] (KAFKA-16608) AsyncKafkaConsumer doesn't honour interrupted thread status on KafkaConsumer.poll(Duration)

2024-04-23 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16608: Summary: AsyncKafkaConsumer doesn't honour interrupted thread status on KafkaConsumer.poll(Duration) Key: KAFKA-16608 URL: https://issues.apache.org/jira/browse/KAFKA

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-23 Thread Andrew Schofield
illiseconds to load the share-group state from > the share-group state partitions loaded in the last 30 seconds." > The window depends on metrics.num.samples and metrics.sample.window.ms > and is not always 30 seconds, right? > 137.4 Could you explain write/write-latency a bit more? Do

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

2024-04-23 Thread Andrew Schofield
Hi Fred, Just reviewing the KIP again now that the discussion has quietened down a little. It will soon be ready for a vote I think. I have a few comments about details. A1) The signature of the new constructor for RecordDeserializationException needs to be updated accord to the discussion. I

Re: [VOTE] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

2024-04-22 Thread Andrew Schofield
Hi Nikhil, Thanks for the KIP. Looks good to me. +1 (non-binding) Thanks, Andrew > On 22 Apr 2024, at 09:17, Christo Lolov wrote: > > Heya Nikhil, > > Thanks for the proposal, as mentioned before it makes sense to me! > > +1 (binding) > > Best, > Christo > > On Sat, 20 Apr 2024 at 00:25,

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-22 Thread Andrew Schofield
> group.share.heartbeat.interval.ms. > > 133. group.share.max.size: Would group.share.max.members.per.group be a > more intuitive name? > > 134. group.share.assignors: Why does it need to be a list? > > 135. share.coordinator.threads: Is that per share coordinator or per b

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-04-18 Thread Andrew Schofield
Hi Alieh, Thanks for the KIP. Exception handling in the Kafka producer and consumer is really not ideal. It’s even harder working out what’s going on with the consumer. I’m a bit nervous about this KIP and I agree with Chris that it could do with additional motivation. This would be an

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

2024-04-18 Thread Andrew Schofield
Hi, Thanks for the KIP. I think it’s an interesting idea and it seems to work nicely with how the clients work today. Recently, I’ve been digging in to the record deserialization code in the consumer as part of implementing KIP-932. It’s pretty nasty in there. There are essentially two kinds

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-16 Thread Andrew Schofield
Is there a way for a user to > optimize this? > > 116. For each new RPC, could we list the associated acls? > > 117. Since this KIP changes internal records and RPCs, it would be useful > to document the upgrade process. > > Jun > > On Wed, Apr 10, 2024 at 7:35 AM An

Re: [DISCUSS] KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission

2024-04-16 Thread Andrew Schofield
Hi Nikhil, I agree with Christo. This is a good improvement and I think your choice of Alter permission on the cluster is the best available. Thanks, Andrew > On 15 Apr 2024, at 12:33, Christo Lolov wrote: > > Heya Nikhil, > > Thank you for raising this KIP! > > Your proposal makes sense to me.

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-15 Thread Andrew Schofield
s already > scope to share groups? > > 019: Should we also scope the name of `record-acknowledgement` to follow > `share-acknowledgement`? > > 020: I suppose that the SPEO is always bounded by the HWM. It may be good > to call it out. Is it also bounded by the LSO? > &g

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-04-15 Thread Andrew Schofield
Thanks for the KIP +1 (non-binding) Andrew > On 15 Apr 2024, at 14:16, Chris Egerton wrote: > > Hi Ivan, > > Thanks for the KIP. After the recent changes, this LGTM. +1 (binding) > > Cheers, > > Chris > > On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko > wrote: > >> Hello, >> >> The discussion

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

2024-04-12 Thread Andrew Schofield
Hi Damien, Sebastien and Loic, Thanks for the KIP. The DLQ pattern is well established and bringing this to Kafka Streams is a good improvement. I do plan to add DLQ support to share groups in the future, once KIP-932 is complete. Having broad support in Kafka for DLQs is great. I have a few

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-12 Thread Andrew Schofield
hanged the ambiguous “State” to “DeliveryState” in RPCs and records. And I added a clarification about how the “group.type” configuration should be used. Thanks, Andrew > On 10 Apr 2024, at 15:33, Andrew Schofield > wrote: > > Hi Jun, > Thanks for your questions. > > 41. &

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-10 Thread Andrew Schofield
teShareGroupOffsetsResult exposes > public KafkaFuture partitionResult(final TopicPartition partition) > DeleteShareGroupsResult exposes > public Map> deletedGroups() > Should we make them more consistent? > > 110. Should ShareGroupDescription include fields like GroupEpoch, > Assi

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

2024-04-09 Thread Andrew Schofield
ld remove the mention of this > cluster ID checks from the KIP. > > Best, > Ivan > > [1] > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355 > > On Tue, Apr 9, 2024, at 09:2

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

2024-04-09 Thread Andrew Schofield
ld remove the mention of this > cluster ID checks from the KIP. > > Best, > Ivan > > [1] > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355 > > On Tue, Apr 9, 2024, at 09:2

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

2024-04-09 Thread Andrew Schofield
Hi Ivan, Thanks for the KIP. I can see situations in which this would be helpful. I have one question. The KIP says the client checks the cluster ID when it re-bootstraps and that it will fail if the cluster ID doesn’t match the previously known one. How does it fail? Which exception does it

Re: [VOTE] KIP-1022 Formatting and Updating Features

2024-04-08 Thread Andrew Schofield
Hi Justine, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 8 Apr 2024, at 18:07, Justine Olshan wrote: > > Hello all, > I would like to start a vote for KIP-1022 Formatting and Updating Features >

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-08 Thread Andrew Schofield
Hi David, Thanks for your questions. 70. The Group Coordinator communicates with the Share Coordinator over RPCs. In the general case, it’s an inter-broker call. It is probably possible to optimise for the situation in which the appropriate GC and SC shards are co-located, but the KIP does not

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-05 Thread Andrew Schofield
g > parameter? A similar issue arises wrt topic configs, but this is the first > instance (that I'm aware of) of a config being added during the MM2 era, so > perhaps this is a minor problem worth thinking about. > > Cheers, > > Tom > > On Wed, 3 Apr 2024 at 05:31, Andr

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-04 Thread Andrew Schofield
> >> Hi Andrew (and Omnia), >> >> Thanks for the KIP. I hope to provide some feedback on this KIP soon, but I >> had a thought on the specific subject of group configs and MM2. If brokers >> validate for known groups configs then doesn't this induce an ordering >

Re: [VOTE] 3.6.2 RC2

2024-04-03 Thread Andrew Schofield
Hi, I downloaded the binary release and upgraded a broker from 3.6.0 to 3.6.2. Documentation and protocol links look fine. +1 (non-binding) Thanks, Andrew > On 3 Apr 2024, at 07:17, Manikumar wrote: > > Gentle reminder. Please download, test and vote for the release. > > Thanks, > > On Fri,

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-04-02 Thread Andrew Schofield
uming from large number of topics with large number of partitions what > will be the impact on coordinator memory? > > Thanks > Omnia > > >> On 25 Mar 2024, at 10:23, Andrew Schofield >> wrote: >> >> Hi Justine, >> Thanks for your questions. &

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-28 Thread Andrew Schofield
Hi David, I agree that we should use the same mechanism to gate KIP-932 once that feature reaches production readiness. The precise details of the values will depend upon the current state of all these flags when that release comes. Thanks, Andrew > On 28 Mar 2024, at 07:11, David Jacot wrote:

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-03-25 Thread Andrew Schofield
also mentioned in the KIP how there are a few inter-broker hops to the > share coordinator, etc for a given read operation of a partition. Are we > concerned about performance here? My work in transactions and trying to > optimize performance made me realize how expensive these inter-

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

2024-03-21 Thread Andrew Schofield
:56, Ahmed Sobeh wrote: > > Hi Andrew, > > Apologies for disappearing, I had to undergo knee surgery, all good now! > > I adjusted the KIP as you suggested, do you think it's ready for the voting > stage? > > On Wed, Mar 6, 2024 at 4:02 PM Andrew Schofield < > andr

Re: [DISCUSS] KIP-1008: ParKa - the Marriage of Parquet and Kafka

2024-03-21 Thread Andrew Schofield
Hi Xinli, Thanks for the KIP. I see that the discussion thread has died down, which is often a tricky situation with a KIP. I’ve been thinking about this KIP for a while and it was really good to be able to attend the Kafka Summit London session to get a proper understanding of it. I think it’s

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-03-21 Thread Andrew Schofield
up members or share groups? Are they any ways we try to > handle this or mitigate risks with respect to memory usage and client > connections (wrt to rebalances for example) > > Thanks, > > Justine > > On Fri, Mar 8, 2024 at 12:51 AM Andrew Schofield < > andrew_schof

[VOTE] KIP-932: Queues for Kafka

2024-03-17 Thread Andrew Schofield
Hi, I’ve been working to complete KIP-932 over the past few months and discussions have quietened down. I’d like to open the voting for KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka Thanks, Andrew

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-03-08 Thread Andrew Schofield
share groups, Do we need to clean any in-memory state from > share-partition leaders? > 3. Any metrics for the share-coordinator? > > Thanks > Manikumar > > On Wed, Feb 21, 2024 at 12:11 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >>

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

2024-03-06 Thread Andrew Schofield
eparate flag. >>> >>> Thanks, >>> Andrew >>> >>>> On 28 Feb 2024, at 09:22, Ahmed Sobeh >>> wrote: >>>> >>>> Hi Andrew, >>>> >>>> Thanks for the hint! That sounds reasonable, do you think adding a >>

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-29 Thread Andrew Schofield
Hi David, Thanks for the summary. It seems to me that the Flow-like option is best because it can easily handle cancellations and exceptions, returning the topic partition information and signalling when the last of the results have been returned. I think it’s also equally applicable to any of

Re: [DISCUSS] KIP-939: Support Participation in 2PC

2024-02-28 Thread Andrew Schofield
Hi Artem, I totally agree that a timeout for the 2PC case is a bad idea. It does abandon the 2PC guarantee. Thanks, Andrew > On 28 Feb 2024, at 00:44, Artem Livshits > wrote: > > Hi Jun, > > Thank you for the discussion. > >> For 3b, it would be useful to understand the reason why an admin

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

2024-02-28 Thread Andrew Schofield
et by time? or do you think having a special > argument for this case is better? > > On Tue, Feb 27, 2024 at 2:19 PM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Hi Ahmed, >> Thanks for the KIP. It looks like a useful addition. >> >

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

2024-02-27 Thread Andrew Schofield
Hi Ahmed, Thanks for the KIP. It looks like a useful addition. The use of negative timestamps, and in particular letting the user use `--time -1` or the equivalent `--time latest` is a bit peculiar in this tool already. The negative timestamps come from

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-02-27 Thread Andrew Schofield
Hi Justine, Thanks for the KIP. This area of Kafka is complicated and making it easier is good. When I use the `kafka-features.sh` tool to describe the features on my cluster, I find that there’s a single feature called “metadata.version”. I think this KIP does a handful of things: 1) It

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-23 Thread Andrew Schofield
This is an interesting problem. While it would be possible to use a Consumer, it can’t handle exceptions really. A java.util.Stream has a similar problem. I wonder whether an interface which looks like java.util.concurrent.Flow.Subscriber would be good. Something like: public interface

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-02-20 Thread Andrew Schofield
Hi Manikumar, Thanks for your comments. 1. I believe that in general, there are not situations in which a dynamic config change is prevented because of the existence of a resource. So, if we prevented setting config `group.type=consumer` on resource G of GROUP type if there was a share group G in

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-02-20 Thread Andrew Schofield
sOptions supports filtering by state. Should we > support it here too? > > 31. ConsumerGroupListing includes state. Should we include state in > GroupListing to be consistent? > > Jun > > On Mon, Feb 12, 2024 at 3:55 AM Andrew Schofield < > andrew_schofield_j...@outloo

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

2024-02-20 Thread Andrew Schofield
Hi Apoorv, Thanks for the KIP. +1 (non-binding) Thanks, Andrew > On 19 Feb 2024, at 22:31, Apoorv Mittal wrote: > > Hi, > I’d like to start the voting for KIP-1019: Expose method to determine > Metric Measurability. > >

[jira] [Created] (KAFKA-16261) MembershipManagerImpl.updateSubscription fails if already empty subscription

2024-02-15 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16261: Summary: MembershipManagerImpl.updateSubscription fails if already empty subscription Key: KAFKA-16261 URL: https://issues.apache.org/jira/browse/KAFKA-16261

Re: [VOTE] 3.7.0 RC4

2024-02-15 Thread Andrew Schofield
+1 (non-binding). I used the staged binaries with Scala 2.13. I tried the new group coordinator and consumer group protocol which is included with the Early Access release of KIP-848. Also verified the availability of the new APIs. All working as expected. Thanks, Andrew > On 15 Feb 2024, at

Re: [Discuss] KIP-1019: Expose method to determine Metric Measurability

2024-02-14 Thread Andrew Schofield
Hi Apoorv, Thanks for the KIP. Looks like a useful change to tidy up the metrics code. Thanks, Andrew > On 14 Feb 2024, at 14:55, Apoorv Mittal wrote: > > Hi, > I would like to start discussion of a small KIP which fills a gap in > determining Kafka Metric measurability. > > KIP-1019: Expose

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-02-12 Thread Andrew Schofield
t; > 28. I noticed that the "*--state*" and "*--timeout*" options are not > mentioned for the kafka-share-groups.sh tool. Was this omission > intentional, or is it possibly an oversight in the KIP? > Thanks, > Chirag > > On Mon, Feb 12, 2024 at 5:25 PM Andrew Scho

Re: [DISCUSS] KIP-932: Queues for Kafka

2024-02-12 Thread Andrew Schofield
ost recent SHARE_CHECKPOINT": How does a broker find this efficiently > on restart? > > 25. AcknowledgeCommitCallback: How would an application use it? It doesn't > indicate which record's acknowledgement has failed. > > 26. AlterShareGroupOffsetsResult alterShareGroupO

[jira] [Created] (KAFKA-16243) Idle kafka-console-consumer with new consumer group protocol preemptively leaves group

2024-02-12 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16243: Summary: Idle kafka-console-consumer with new consumer group protocol preemptively leaves group Key: KAFKA-16243 URL: https://issues.apache.org/jira/browse/KAFKA-16243

[jira] [Resolved] (KAFKA-8606) Provide a method to fetch committed offsets for a collection of TopicPartition

2024-01-30 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-8606. - Fix Version/s: 2.4.0 Resolution: Fixed Method added in AK 2.4. > Prov

[jira] [Created] (KAFKA-16184) Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16184: Summary: Flaky test - testAlterReplicaLogDirs(String).quorum=kraft – kafka.api.PlaintextAdminIntegrationTest Key: KAFKA-16184 URL: https://issues.apache.org/jira/browse

[jira] [Created] (KAFKA-16183) Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16183: Summary: Flaky test - testMetricsDuringTopicCreateDelete(String).quorum=zk – kafka.integration.MetricsDuringTopicCreationDeletionTest Key: KAFKA-16183 URL: https

[jira] [Created] (KAFKA-16182) Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest

2024-01-22 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16182: Summary: Flaky test - testClientInstanceId() - org.apache.kafka.clients.admin.KafkaAdminClientTest Key: KAFKA-16182 URL: https://issues.apache.org/jira/browse/KAFKA-16182

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-22 Thread Andrew Schofield
+1 (non-binding) Thanks, Andrew > On 22 Jan 2024, at 07:29, Federico Valeri wrote: > > +1 (non binding) > > Thanks. > > On Mon, Jan 22, 2024 at 7:03 AM Luke Chen wrote: >> >> Hi Ziming, >> >> +1(binding) from me. >> >> Thanks. >> Luke >> >> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash

[jira] [Created] (KAFKA-16137) ListClientMetricsResourcesResponse definition is missing field descriptions

2024-01-15 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16137: Summary: ListClientMetricsResourcesResponse definition is missing field descriptions Key: KAFKA-16137 URL: https://issues.apache.org/jira/browse/KAFKA-16137

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-12 Thread Andrew Schofield
I also agree with 2 & 2 with reasoning along the same lines as Artem. Thanks, Andrew > On 12 Jan 2024, at 09:15, Federico Valeri wrote: > > On Thu, Jan 11, 2024 at 10:43 PM Artem Livshits > wrote: >> >> Hi Proven, >> >> I'd say that we should do 2 & 2. The idea is that for small features

[jira] [Created] (KAFKA-16104) Enable additional PlaintextConsumerTest tests for new consumer

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16104: Summary: Enable additional PlaintextConsumerTest tests for new consumer Key: KAFKA-16104 URL: https://issues.apache.org/jira/browse/KAFKA-16104 Project

[jira] [Created] (KAFKA-16100) Consistent handling of timeouts and responses for new consumer ApplicationEvents

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16100: Summary: Consistent handling of timeouts and responses for new consumer ApplicationEvents Key: KAFKA-16100 URL: https://issues.apache.org/jira/browse/KAFKA-16100

[jira] [Created] (KAFKA-16099) Handle timeouts for AsyncKafkaConsumer.commitSync

2024-01-09 Thread Andrew Schofield (Jira)
Andrew Schofield created KAFKA-16099: Summary: Handle timeouts for AsyncKafkaConsumer.commitSync Key: KAFKA-16099 URL: https://issues.apache.org/jira/browse/KAFKA-16099 Project: Kafka

  1   2   3   >