Re: How Kafka handle partition leader change?

Looks like the core of the problem should still be the juggling game of consistency, availability and partition tolerance. If we want the cluster still work when brokers have inconsistent information due to network partition, we have to choose between consistency and availability. My proposal

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2407

See

Re: [DISCUSS] Road to Kafka 4.0

Hi Luke, I think we're conflating different things here. There are 3 separate points in your email, but only 1 of them requires 3.8: 1. JBOD may have some bugs in 3.7.0. Whatever bugs exist can be fixed in 3.7.x. We have already said that we will backport critical fixes to 3.7.x for some time.

Re: [DISCUSS] Road to Kafka 4.0

Hi Colin and Jose, I revisited the discussion of KIP-833 here , and you can see I'm the first one to reply to the discussion thread to express my excitement at that time. Till now, I personally still think having KRaft in Kafka is

Re: [DISCUSS] Should we continue to merge without a green build? No!

Hi, We have a dashboard already: [image: image.png] https://ge.apache.org/scans/tests?search.names=Git%20branch=P28D=kafka=America%2FLos_Angeles=trunk=FLAKY On Tue, Nov 14, 2023 at 10:41 PM Николай Ижиков wrote: > Hello guys. > > I want to tell you about one more approach to deal with flaky

Re: [DISCUSS] Should we continue to merge without a green build? No!

Thanks Sophie. Overall I agree with you. I think 50% is too high as a general rule, and I believe something like 30% might be more appropriate (going lower, given the infrastructure at hand might become too aggressive)? The difficult part about a policy like this is, that we don't really

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Thanks Jun. I updated the KIP with this information and also linked to a JIRA that captures the items required for 3.7. Ismael On Tue, Nov 21, 2023 at 2:53 PM Jun Rao wrote: > Hi, Ismael, > > Thanks for the KIP. +1 > > It would be useful to clarify in the KIP that the new metric and new >

[jira] [Created] (KAFKA-15874) Add metric and request log attribute for deprecated request api versions

Ismael Juma created KAFKA-15874: --- Summary: Add metric and request log attribute for deprecated request api versions Key: KAFKA-15874 URL: https://issues.apache.org/jira/browse/KAFKA-15874 Project:

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

Hey -- just catching up here, since I saw the vote thread. I had 2 questions that I'm not sure got answered from the previous discussion. 1. Can we update the KIP to include the name of the other image so if someone stumbles across this KIP they know the name of the other one? 2. Did we cover

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2406

See

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Thanks. LGTM. +1. On Tue, Nov 21, 2023 at 2:54 PM Jun Rao wrote: > > Hi, Ismael, > > Thanks for the KIP. +1 > > It would be useful to clarify in the KIP that the new metric and new > request log attribute will be added in Apache 3.7. > > Jun > > On Tue, Nov 21, 2023 at 1:57 PM Colin McCabe

[jira] [Created] (KAFKA-15873) Improve the performance of the DescribeTopicPartitions API

Calvin Liu created KAFKA-15873: -- Summary: Improve the performance of the DescribeTopicPartitions API Key: KAFKA-15873 URL: https://issues.apache.org/jira/browse/KAFKA-15873 Project: Kafka Issue

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Hi, Ismael, Thanks for the KIP. +1 It would be useful to clarify in the KIP that the new metric and new request log attribute will be added in Apache 3.7. Jun On Tue, Nov 21, 2023 at 1:57 PM Colin McCabe wrote: > Ah. I forget that KIP-724 not only deprecated, but proposed a removal in > 4.0.

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Ah. I forget that KIP-724 not only deprecated, but proposed a removal in 4.0. Great. +1 (binding) for KIP-896 best, Colin On Tue, Nov 21, 2023, at 12:36, Ismael Juma wrote: > Hi Colin, > > That change was proposed and approved via KIP-724: > >

[jira] [Resolved] (KAFKA-15774) Respect default.dsl.store Configuration Without Passing it to StreamsBuilder

[ https://issues.apache.org/jira/browse/KAFKA-15774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Almog Gavra resolved KAFKA-15774. - Fix Version/s: 3.7.0 Resolution: Fixed Note that we decided not to have

[jira] [Resolved] (KAFKA-15215) The default.dsl.store config is not compatible with custom state stores

[ https://issues.apache.org/jira/browse/KAFKA-15215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Almog Gavra resolved KAFKA-15215. - Fix Version/s: 3.7.0 Resolution: Fixed > The default.dsl.store config is not compatible

Re: [DISCUSS] Should we continue to merge without a green build? No!

For some concrete data, here are the stats for the latest build on two community PRs I am currently reviewing: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14648/16/tests - 18 unrelated test failures - 13 unique tests - only 1 out of 4 JDK builds were green

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2405

See

Re: [DISCUSS] Should we continue to merge without a green build? No!

In the interest of moving things forward, here is what we would need (in my opinion) to start enforcing this: 1. Get overall build failure percentage under a certain threshold 1. What is an acceptable number here? 2. How do we achieve this: wait until all of them are fixed, disable

Re: [DISCUSS] Should we continue to merge without a green build? No!

So...where did we land on this? To take a few steps back and make sure we're all talking about the same thing, I want to mention that I myself was responsible for merging a PR that broke the build a few weeks ago. There was a warning that only appeared in some of the versions, but when checking

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Hi Colin, That change was proposed and approved via KIP-724: https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1 Ismael On Tue, Nov 21, 2023, 12:21 PM Colin McCabe wrote: > Hi Ismael, > > Can we state somewhere that the

Re:[DISCUSS] KIP-1009: Add Broker-level Throttle Configurations

Hi All, (I sent the previous email prematurely, apologies) I'd like to start a discussion on KIP-1009: Add Broker-level Throttle Configurations - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1009%3A+Add+Broker-level+Throttle+Configurations At a high level, this KIP adds dynamic

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Hi Ismael, Can we state somewhere that the message.format.version configuration will be gone in 4.0? We only will support one message format version (for now, at least). If we do want more versions later, I don't think we'll want to configure them via a static config. best, Colin On Tue,

[DISCUSS] KIP-1009: Add Broker-level Throttle Configurations

Hi All, I'd like to start a discussion on KIP-1009: Add Broker-level Throttle Configurations.

Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0

I started a vote thread for this since I addressed the comments so far and it doesn't seem like there were any major concerns. Ismael On Tue, Jan 3, 2023 at 8:17 AM Ismael Juma wrote: > Hi all, > > I would like to start a discussion regarding the removal of very old > client protocol API

[VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

Hi all, I would like to start a vote on KIP-896. Please take a look and let us know what you think. Even though most of the changes in this KIP will be done for Apache Kafka 4.0, I would like to introduce a new metric and new request log attribute in Apache 3.7 to help users identify usage of

Re: [DISCUSS] KIP-896: Remove old client protocol API versions in Kafka 4.0

Hi Jose, I updated the KIP to include a new metric for deprecated request api versions and also a new attribute in the request log to make it easy to find such entries, Thanks, Ismael On Thu, Jan 12, 2023 at 1:03 AM Ismael Juma wrote: > Hi Jose, > > I think it's reasonable to add more

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Thanks to all for voting. So I consider KIP-968 as accepted. Cheers, Alieh On Tue, Nov 21, 2023 at 5:22 PM Matthias J. Sax wrote: > +1 (binding) > > On 11/21/23 4:52 AM, Lucas Brutschy wrote: > > Hi Alieh, > > > > thanks for the KIP! > > > > +1 (binding) > > > > Lucas > > > > On Tue, Nov 21,

Re: [VOTE] KIP-1001; CurrentControllerId Metric

Thanks, everyone! With binding +1s from: José Armando García Sancio Jason Gustafson David Arthur David Jacot the KIP passes. regards, Colin On Tue, Nov 21, 2023, at 09:25, José Armando García Sancio wrote: > LGTM. +1 binding. > > On Mon, Nov 20, 2023 at 1:48 PM Jason Gustafson > wrote: >> >>

Re: [DISCUSS] Road to Kafka 4.0

On Tue, Nov 21, 2023, at 03:47, Josep Prat wrote: > Hi Colin, > > I think it's great that Confluent runs KRaft clusters in production, > and it means that it is production ready for Confluent and it's users. > But luckily for Kafka, the community is bigger than this (self managed > in the cloud

Re: [VOTE] KIP-1001; CurrentControllerId Metric

LGTM. +1 binding. On Mon, Nov 20, 2023 at 1:48 PM Jason Gustafson wrote: > > The KIP makes sense. +1 > > On Mon, Nov 20, 2023 at 12:37 PM David Arthur > wrote: > > > Thanks Colin, > > > > +1 from me > > > > -David > > > > On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > > > > > Hi all, >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2404

See

[DISCUSS] KIP-1008: ParKa - the Marriage of Parquet and Kafka

Hi, all Can I ask for a discussion on the KIP just created KIP-1008: ParKa - the Marriage of Parquet and Kafka ? -- Xinli Shang

[jira] [Created] (KAFKA-15872) Investigate autocommit retry logic

Philip Nee created KAFKA-15872: -- Summary: Investigate autocommit retry logic Key: KAFKA-15872 URL: https://issues.apache.org/jira/browse/KAFKA-15872 Project: Kafka Issue Type: Sub-task

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

+1 (binding) On 11/21/23 4:52 AM, Lucas Brutschy wrote: Hi Alieh, thanks for the KIP! +1 (binding) Lucas On Tue, Nov 21, 2023 at 11:26 AM Alieh Saeedi wrote: Thanks, Matthias; I changed it to `ANY` which is the shortest and not misleading. Cheers, Alieh On Mon, Nov 20, 2023 at 7:42 PM

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Thanks! SGTM. Seems all open questions are resolved. Thanks for pushing this through! -Matthias On 11/21/23 2:29 AM, Alieh Saeedi wrote: Yes Matthias, Based on the discussion we had, it has now been changed to Optional and the default is empty (for the latest). Also, the `validTo()` method

[jira] [Created] (KAFKA-15871) Implement kafka-client-metrics.sh tool

Andrew Schofield created KAFKA-15871: Summary: Implement kafka-client-metrics.sh tool Key: KAFKA-15871 URL: https://issues.apache.org/jira/browse/KAFKA-15871 Project: Kafka Issue Type:

[jira] [Resolved] (KAFKA-15837) Throw error on use of Consumer.poll(long timeout)

[ https://issues.apache.org/jira/browse/KAFKA-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield resolved KAFKA-15837. -- Resolution: Fixed > Throw error on use of Consumer.poll(long timeout) >

[jira] [Created] (KAFKA-15870) Move new group coordinator metrics from Yammer to Metrics

Jeff Kim created KAFKA-15870: Summary: Move new group coordinator metrics from Yammer to Metrics Key: KAFKA-15870 URL: https://issues.apache.org/jira/browse/KAFKA-15870 Project: Kafka Issue

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #99

See Changes: -- [...truncated 470584 lines...] Gradle Test Run :streams:integrationTest > Gradle Test Executor 186 > TableTableJoinIntegrationTest > [caching enabled = true]

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2403

See

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

Heya! Thanks a lot for this. I have updated the KIP to include exposing the tiered-offset as well. Let me know whether the Public Interfaces section needs more explanations regarding the changes needed to the OffsetSpec or others. Best, Christo On Tue, 21 Nov 2023 at 04:20, Satish Duggana

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi Alieh, thanks for the KIP! +1 (binding) Lucas On Tue, Nov 21, 2023 at 11:26 AM Alieh Saeedi wrote: > > Thanks, Matthias; I changed it to `ANY` which is the shortest and not > misleading. > > Cheers, > Alieh > > On Mon, Nov 20, 2023 at 7:42 PM Matthias J. Sax wrote: > > > Adding an enum is

Re: Requesting permissions to contribute to Apache Kafka

Hi Ria, You are now set. Thanks for your interest in Apache Kafka! Best, On Mon, Nov 20, 2023 at 5:48 PM Ria Pradeep (BLOOMBERG/ 919 3RD A) < rprade...@bloomberg.net> wrote: > I would like to request permission to contribute to Apache Kafka. > > wiki ID: rpradeep > JIRA ID: rpradeep > >

Re: How Kafka handle partition leader change?

Hey De Gao, Message loss or duplication can actually happen even without a leadership change for a partition. For example if there are network issues and the producer never gets the ack from the server, it’ll retry and cause duplicates. Message loss can usually occur when you use acks=1 config

Re: [DISCUSS] Road to Kafka 4.0

Hi Colin, I think it's great that Confluent runs KRaft clusters in production, and it means that it is production ready for Confluent and it's users. But luckily for Kafka, the community is bigger than this (self managed in the cloud or in-prem, or customers of other SaaS companies). We've

[jira] [Created] (KAFKA-15869) Document semantics of nullable nested API entities

Anton Agestam created KAFKA-15869: - Summary: Document semantics of nullable nested API entities Key: KAFKA-15869 URL: https://issues.apache.org/jira/browse/KAFKA-15869 Project: Kafka Issue

Re: Apache Kafka 3.7.0 Release

Hi Stan Can you include KIP-951 to the 3.7 release plan? All PRs are merged in the trunk. On Wed, Nov 15, 2023 at 4:05 PM Stanislav Kozlovski wrote: > Friendly reminder to everybody that the KIP Freeze is *exactly 7 days away* > - November 22. > > A KIP must be accepted by this date in order

[jira] [Resolved] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client

[ https://issues.apache.org/jira/browse/KAFKA-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Shekhar Narula resolved KAFKA-15868. --- Resolution: Fixed > KIP-951 - Leader discovery optimisations for the client

[jira] [Created] (KAFKA-15868) KIP-951 - Leader discovery optimisations for the client

Mayank Shekhar Narula created KAFKA-15868: - Summary: KIP-951 - Leader discovery optimisations for the client Key: KAFKA-15868 URL: https://issues.apache.org/jira/browse/KAFKA-15868 Project:

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Yes Matthias, Based on the discussion we had, it has now been changed to Optional and the default is empty (for the latest). Also, the `validTo()` method returns an Optional. Cheers, Alieh On Mon, Nov 20, 2023 at 7:38 PM Matthias J. Sax wrote: > I think we should also discuss a little more

Re: [VOTE] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Thanks, Matthias; I changed it to `ANY` which is the shortest and not misleading. Cheers, Alieh On Mon, Nov 20, 2023 at 7:42 PM Matthias J. Sax wrote: > Adding an enum is a good idea! > > Wondering if `UNORDERED` is the best name? Want to avoid bike shedding, > just asking. > > We could also

[VOTE] 3.5.2 RC1

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.5.2. This is a bugfix release with several fixes since the release of 3.5.1, including dependency version bumps for CVEs. Release notes for the 3.5.2 release:

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2402

See

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

Hi All, It would be nice if we could have more people to review and vote for this KIP. Many thanks! Qichao On Mon, Nov 20, 2023 at 2:43 PM Qichao Chu wrote: > @Matthias: yeah it should be 977, sorry for the confusion. > Btw, do you want to cast another binding vote for it? > > Best, > Qichao

[jira] [Resolved] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Nee resolved KAFKA-15836. Resolution: Fixed PR merged. > KafkaConsumer subscribes to multiple topics does not respect

Re: [VOTE] KIP-1001; CurrentControllerId Metric

+1 from me. Thanks, David On Mon, Nov 20, 2023 at 10:48 PM Jason Gustafson wrote: > The KIP makes sense. +1 > > On Mon, Nov 20, 2023 at 12:37 PM David Arthur > wrote: > > > Thanks Colin, > > > > +1 from me > > > > -David > > > > On Tue, Nov 14, 2023 at 3:53 PM Colin McCabe wrote: > > > > >