Re: [DISCUSS] Should we continue to merge without a green build? No!

Hello guys. I want to tell you about one more approach to deal with flaky tests. We adopt this approach in Apache Ignite community, so may be it can be helpful for Kafka, also. TL;DR: Apache Ignite community have a tool that provide a statistic of tests and can tell if PR introduces new

Re: [DISCUSS] Should we continue to merge without a green build? No!

To use the pain analogy, people seem to have really good painkillers and hence they somehow don't feel the pain already. ;) The reality is that important and high quality tests will get fixed. Poor quality tests (low signal to noise ratio) might not get fixed and that's ok. I'm not opposed to

Re: [DISCUSS] Should we continue to merge without a green build? No!

One potential forcing function could be to allow an unconditional revert of any commit (including testing and non-testing changes) that can be shown to have introduced test flakiness. This could at least prevent us from accruing more flaky tests, though it would obviously not be viable for

Re: [DISCUSS] Should we continue to merge without a green build? No!

I agree on the test gap argument. However, my worry is, if we don't "force the pain", it won't get fixed at all. -- I also know, that we try to find an working approach for many years... My take is that if we disable a test and file a non-blocking Jira, it's basically the same as just

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Just catching up on this one. 50) I am also in favor of setting `validTo` in VersionedRecord for single-key single-ts lookup; it seems better to return the proper timestamp. The timestamp is already in the store and it's cheap to extract it and add to the result, and it might be valuable

Re: [DISCUSS] Should we continue to merge without a green build? No!

Matthias, Flaky tests are worse than useless. I know engineers find it hard to disable them because of the supposed test gap (I find it hard too), but the truth is that the test gap is already there! No-one blocks merging PRs on flaky tests, but they do get used to ignoring build failures. The

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2382

See

[jira] [Created] (KAFKA-15829) How to build Kafka 2.7 with maven instead of gradle?

rain.liang created KAFKA-15829: -- Summary: How to build Kafka 2.7 with maven instead of gradle? Key: KAFKA-15829 URL: https://issues.apache.org/jira/browse/KAFKA-15829 Project: Kafka Issue Type:

Re: [DISCUSS] Should we continue to merge without a green build? No!

Thanks Matthias -- I agree that there are lots of flaky test JIRAs but not a forcing function to fix them. One option is to mark them as blockers as you alluded to. One other thing to keep in mind is that we have the gradle enterprise tool to track flaky tests!

Re: [DISCUSS] Should we continue to merge without a green build? No!

Thanks for starting this discussion David! I totally agree to "no"! I think there is no excuse whatsoever for merging PRs with compilation errors (except an honest mistake for conflicting PRs that got merged interleaved). -- Every committer must(!) check the Jenkins status before merging to

[jira] [Created] (KAFKA-15828) Protect clients from broker hostname reuse

Jason Gustafson created KAFKA-15828: --- Summary: Protect clients from broker hostname reuse Key: KAFKA-15828 URL: https://issues.apache.org/jira/browse/KAFKA-15828 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-15827) KafkaBasedLog.withExistingClients leaks clients if start is not called

Greg Harris created KAFKA-15827: --- Summary: KafkaBasedLog.withExistingClients leaks clients if start is not called Key: KAFKA-15827 URL: https://issues.apache.org/jira/browse/KAFKA-15827 Project: Kafka

[VOTE] KIP-1001; CurrentControllerId Metric

Hi all, I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. Take a look here: https://cwiki.apache.org/confluence/x/egyZE best, Colin

Re: [DISCUSS] Road to Kafka 4.0

On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: > Hi Colin, > > Thank you for your thoughtful and comprehensive response. > >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed this in >> several KIPs that happened this year and last year. The most notable was >> probably KIP-866,

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #113

See Changes: -- [...truncated 308234 lines...] Gradle Test Run :streams:test > Gradle Test Executor 88 > RocksDBBlockCacheMetricsTest >

[jira] [Created] (KAFKA-15826) WorkerSinkTask leaks Consumer if plugin start or stop blocks indefinitely

Greg Harris created KAFKA-15826: --- Summary: WorkerSinkTask leaks Consumer if plugin start or stop blocks indefinitely Key: KAFKA-15826 URL: https://issues.apache.org/jira/browse/KAFKA-15826 Project:

[jira] [Resolved] (KAFKA-15825) KRaft controller writes empty state to ZK after migration

[ https://issues.apache.org/jira/browse/KAFKA-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15825. -- Resolution: Fixed This bug was fixed as part of KAFKA-15605 > KRaft controller writes empty

[jira] [Created] (KAFKA-15825) KRaft controller writes empty state to ZK after migration

David Arthur created KAFKA-15825: Summary: KRaft controller writes empty state to ZK after migration Key: KAFKA-15825 URL: https://issues.apache.org/jira/browse/KAFKA-15825 Project: Kafka

[jira] [Resolved] (KAFKA-15605) Topics marked for deletion in ZK are incorrectly migrated to KRaft

[ https://issues.apache.org/jira/browse/KAFKA-15605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur resolved KAFKA-15605. -- Fix Version/s: 3.7.0 Resolution: Fixed > Topics marked for deletion in ZK are

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

Thanks, Hailey. +1 (binding) Colin On Mon, Nov 13, 2023, at 15:13, Hailey Ni wrote: > Hi Colin, > > Thank you for your review. I removed the "absolute path need to be > provided" line from the KIP, and will modify the code to get the absolute > path to the config files using some bash in the

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

Heya everyone, Apologies for the delay in my response and thank you very much for all your comments! I will start answering in reverse: *re: Satish* 101. I am happy to scope down this KIP and start off by emitting those metrics on a topic level. I had a preference to emit them on a partition

[jira] [Created] (KAFKA-15824) SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet

Mayank Shekhar Narula created KAFKA-15824: - Summary: SubscriptionState's maybeValidatePositionForCurrentLeader should handle partition which isn't subscribed yet Key: KAFKA-15824 URL:

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi Alieh, I agree with Bruno that a weaker guarantee could be useful already, and it's anyway possible to strengthen the guarantees later on. On the other hand, it would be nice to provide a consistent view across all segments if it doesn't come with major downsides, because until now IQ does

[jira] [Created] (KAFKA-15823) NodeToControllerChannelManager: authentication error prevents controller update

Gaurav Narula created KAFKA-15823: - Summary: NodeToControllerChannelManager: authentication error prevents controller update Key: KAFKA-15823 URL: https://issues.apache.org/jira/browse/KAFKA-15823

[jira] [Created] (KAFKA-15822) Exception when using DumpLogSegments to dump log

jiawen created KAFKA-15822: -- Summary: Exception when using DumpLogSegments to dump log Key: KAFKA-15822 URL: https://issues.apache.org/jira/browse/KAFKA-15822 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

Hi Alieh, Regarding the semantics/guarantees of the query type: Do we need a snapshot semantic or can we specify a weaker but still useful semantic? An option could be to guarantee that: 1. the query returns the latest version when the query arrived at the state store 2. the query returns a

Re: [DISCUSS] Road to Kafka 4.0

Hi Colin, Thank you for your thoughtful and comprehensive response. > KIP-853 is not a blocker for either 3.7 or 4.0. We discussed this in several KIPs that happened this year and last year. The most notable was probably KIP-866, which was approved in May 2022. I understand this is the case,

[jira] [Resolved] (KAFKA-15658) Zookeeper.jar | CVE-2023-44981

[ https://issues.apache.org/jira/browse/KAFKA-15658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya resolved KAFKA-15658. -- Resolution: Fixed > Zookeeper.jar | CVE-2023-44981 > --- > >

Important Security Notice for Apache Kafka Users

Dear Apache Kafka Users, We want to bring to your attention a security vulnerability affecting all released versions of Apache Kafka that have a dependency on Zookeeper. The vulnerability, identified as CVE-2023-44981 [1], specifically impacts users utilizing SASL Quorum Peer authentication in

Re: [VOTE] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

Hi all, Closing this vote as it has enough votes and was open for more than two weeks. KIP-978 has passed the vote with: +3 binding votes +1 non-binding vote Thanks everyone for your votes. I will update the KIP page and open a PR targeting Kafka 3.7.0. Jakub On Tue, Oct 24, 2023 at 10:07 PM

[jira] [Reopened] (KAFKA-15376) Explore options of removing data earlier to the current leader's leader epoch lineage for topics enabled with tiered storage.

[ https://issues.apache.org/jira/browse/KAFKA-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya reopened KAFKA-15376: -- Assignee: (was: Satish Duggana) > Explore options of removing data earlier to the

Re: [DISCUSS] KIP-1002: Fetch remote segment indexes at once

> Offset and Transaction indexes are probably the only ones that make sense to cache as are used on every fetch. I do not think (correct me if I am wrong) that the transaction index is used on every fetch. It is only used when consumers want to include aborted transactions [1] i.e. when they use

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2381

See Changes: -- [...truncated 430965 lines...] Gradle Test Run :streams:test > Gradle Test Executor 88 > DefaultTaskExecutorTest >

[jira] [Resolved] (KAFKA-15376) Explore options of removing data earlier to the current leader's leader epoch lineage for topics enabled with tiered storage.

[ https://issues.apache.org/jira/browse/KAFKA-15376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash resolved KAFKA-15376. -- Resolution: Fixed This task was already addressed in the code, so closing the

Re: [VOTE] KIP-892: Transactional StateStores

Hi Nick! Thanks a lot for the KIP! Looking forward to the implementation! +1 (binding) Best, Bruno On 11/14/23 2:23 AM, Sophie Blee-Goldman wrote: +1 (binding) Thanks a lot for this KIP! On Mon, Nov 13, 2023 at 8:39 AM Lucas Brutschy wrote: Hi Nick, really happy with the final KIP.

[Question] About Kafka producer design decision making

Dear Apache Kakfa Developers, I'm 4-year SWE in South Korea. I have some questions while watching Kafka Producer API. *Why Use "Future" and Not "CompletableFuture"?* In the case of "Future", blocking occurs when calling "*get()*", so I thought "Computable Future" would be better when doing more