Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #49

See Changes: -- [...truncated 561204 lines...] Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > StreamsAssignmentScaleTest >

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

I have updated the KIP with the points as discussed above. @Guozhang, the suggested configuration makes it a little more awkward around the Materialized.as and Materialized.withStoreType APIs than it was when we were totally deprecating the old configuration. Let me know what you think. I will

[jira] [Resolved] (KAFKA-15218) NPE will be thrown while deleting topic and fetch from follower concurrently

[ https://issues.apache.org/jira/browse/KAFKA-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Chen resolved KAFKA-15218. --- Fix Version/s: 3.6.0 Resolution: Fixed > NPE will be thrown while deleting topic and fetch

[jira] [Created] (KAFKA-15251) Upgrade system test to use 3.5.1

Matthias J. Sax created KAFKA-15251: --- Summary: Upgrade system test to use 3.5.1 Key: KAFKA-15251 URL: https://issues.apache.org/jira/browse/KAFKA-15251 Project: Kafka Issue Type: Test

Re: [DISCUSS] KIP-955: Add stream-table join on foreign key

Igor, thanks for the KIP. Interesting proposal. I am wondering a little bit about the use-case and semantics, and if it's really required to add what you propose? Please correct me if I am wrong. In the end, a stream-table join is a "stream enrichment" (via a table lookup). Thus, it's

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #48

See Changes: -- [...truncated 474100 lines...] > Task :connect:json:generateMetadataFileForMavenJavaPublication > Task :connect:api:javadocJar > Task

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

Hi Ismael, I believe I have addressed all concerns. Please have a look, and consider a vote on this KIP. Thank you, -- Igor

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2037

See Changes: -- [...truncated 295536 lines...] Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > TableTableJoinIntegrationTest > [caching enabled =

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

Hey Omnia, Thanks for the KIP! I think that MM2 is responsible for providing an upgrade path for users, even if it isn't backwards-compatible by default due to a mistake. The non-configuration-based strategies I could think of aren't viable due to the danger of inferring the incorrect topic

[jira] [Created] (KAFKA-15250) DefaultBackgroundThread is running tight loop

Philip Nee created KAFKA-15250: -- Summary: DefaultBackgroundThread is running tight loop Key: KAFKA-15250 URL: https://issues.apache.org/jira/browse/KAFKA-15250 Project: Kafka Issue Type: Bug

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

On Tue, Jul 25, 2023, at 05:30, Luke Chen wrote: > Hi Colin, > > Some more comments: > 1. In the KIP, we mentioned "controller heartbeats", but it is not > explained anywhere. > I think "controller heartbeats" = controller registration", is that > correct? > If no, please explain more about it in

Debugging Jenkins test failures

Hi all! I’ve noticed that we’re back in the state where it’s tough to get a clean PR Jenkins test run. Spot checking the top ~10 pull request runs show this doesn’t appear to be an issue with just my PRs :P I know we have some chronic flaky tests, but I’ve seen at least two other classes of

Re: Increase in flaky test failures

Hi Divij, I’m hitting something similar in my pull request test run (https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13990/26/pipeline/10): > Task :clients:testClasses Unexpected exception thrown. org.gradle.internal.remote.internal.MessageIOException: Could

Re: [VOTE] KIP-930: Tiered Storage Metrics

Thank you for the KIP Abhinav. Although we should avoid changing customer-facing interfaces (such as metrics) after a KIP is accepted, in this case, I think that the divergence is minimal and the right thing to do in the longer run. Hence, I would consider this change as a one-off exception and

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

Hi Hector, Thanks for the KIP! +1 (non-binding) Thanks, Yash On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Thanks for the KIP. As you say, not that controversial. > > +1 (non-binding) > > Thanks, > Andrew > > > On 25 Jul 2023, at 18:22, Hector

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2036

See Changes: -- [...truncated 296992 lines...] Gradle Test Run :streams:integrationTest > Gradle Test Executor 180 > TableTableJoinIntegrationTest > [caching enabled =

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

Thanks Igor. Are there unaddressed concerns or is this ready for a vote again? Ismael On Tue, Jul 25, 2023 at 6:14 PM Igor Soarez wrote: > Hi everyone, > > Following a face-to-face discussion with Ron and Colin, > I have just made further improvements to this KIP: > > > 1. Every log directory

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

Thanks for the KIP. As you say, not that controversial. +1 (non-binding) Thanks, Andrew > On 25 Jul 2023, at 18:22, Hector Geraldino (BLOOMBERG/ 919 3RD A) > wrote: > > Hi everyone, > > The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have > a limited scope and

[VOTE] KIP-959 Add BooleanConverter to Kafka Connect

Hi everyone, The changes proposed by KIP-959 (Add BooleanConverter to Kafka Connect) have a limited scope and shouldn't be controversial. I'm opening a voting thread with the hope that it can be included in the next upcoming 3.6 release. Here are some links: KIP:

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

Hi everyone, Following a face-to-face discussion with Ron and Colin, I have just made further improvements to this KIP: 1. Every log directory gets a random UUID assigned, even if just one log dir is configured in the Broker. 2. All online log directories are registered, even if just one if

Re: Apache Kafka 3.6.0 release

Thanks for the update, Mayank Shekhar. On Tue, 25 Jul 2023 at 15:48, Mayank Shekhar Narula wrote: > > Hi Satish > > Heads up KIP-951 is under voting. This could be delayed by a day or two for > the 3.6 KIP deadline. > > On Tue, Jul 25, 2023 at 4:19 AM Satish Duggana > wrote: > > > Thanks Colin

Re: [VOTE] KIP-930: Tiered Storage Metrics

+1 for the KIP. Thanks, Satish. On Tue, 25 Jul 2023 at 18:31, Kamal Chandraprakash wrote: > > +1 (non-binding) > > -- > Kamal > > On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar > wrote: > > > Hi All, > > > > I would like to start the vote for KIP-930 Tiered Storage Metrics. > > > > The KIP is

Re: Re: [DISCUSS] KIP-951: Leader discovery optimisations for the client

Thanks Crispin! Ismael On Mon, Jul 24, 2023 at 8:16 PM Crispin Bernier wrote: > I updated the wiki to include both results along with their average. > > Thank you, > Crispin > > On Mon, Jul 24, 2023 at 10:58 AM Ismael Juma wrote: > > > Hi Crispin, > > > > One additional question, the wiki

Re: [DISCUSS] KIP-954: expand default DSL store configuration to custom types

Glad you like my KIP-secretary skills ;) A2. I'm definitely happy to take your suggestion here and not do anything special w.r.t. Versioned stores, I think it makes sense especially if we consider them implementation details of a specific store type. At EOD I'll update the KIP with all of these

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

Thanks Hector! LGTM On Tue, Jul 25, 2023 at 11:47 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Thanks Chris for your quick reply. > > Your suggestions make sense, I amended the KIP and added a note to the > class JavaDocs. Also added unit tests to the companion

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

Thanks Chris for your quick reply. Your suggestions make sense, I amended the KIP and added a note to the class JavaDocs. Also added unit tests to the companion PR [https://github.com/apache/kafka/pull/14093], and will mark it as "Ready for Review" in a few. Cheers From: dev@kafka.apache.org

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

KIP is updated now: https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations Looking forward to your feedback, Many thanks, Jorge. On Tue, 25 Jul 2023 at 16:59, Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > Hi Colin,

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2035

See Changes: -- [...truncated 296972 lines...] Gradle Test Run :streams:integrationTest > Gradle Test Executor 179 > RestoreIntegrationTest >

Re: [DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

Hi Hector, Thanks for the KIP! Really appreciate the tight scope, hoping this will be easy to review :) I only have one question: it looks like our existing primitive converters (string converter + subclasses of NumberConverter) are hardcoded to play nicely with null values during

[jira] [Created] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central

Chris Egerton created KAFKA-15249: - Summary: Verify Connect test-plugins artifact is published to Maven Central Key: KAFKA-15249 URL: https://issues.apache.org/jira/browse/KAFKA-15249 Project: Kafka

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

Hi Colin, sorry for the belated follow up. If I understand correctly, on your latest reply proposed to have a new API. >From the proposed alternatives, I lean towards the first alternative proposed with 2 config maps, old (before-alter) and new (after-alter). Deleting a config is effectively

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

Hey @Sagar Thanks again for the review. 1. "a null headers value is equivalent to invoking the older partition method", this is not true. If someone makes an implementation and the headers come as null, still the new implementation will take effect. Instead I have added : "Not overriding this

[jira] [Created] (KAFKA-15248) Add BooleanConverter to Kafka Connect

Hector Geraldino created KAFKA-15248: Summary: Add BooleanConverter to Kafka Connect Key: KAFKA-15248 URL: https://issues.apache.org/jira/browse/KAFKA-15248 Project: Kafka Issue Type:

[DISCUSS] KIP-959 Add BooleanConverter to Kafka Connect

Hi everyone, I'd like to start a discussion of KIP-959, which aims to add a BooleanConverter to Kafka Connect: https://cwiki.apache.org/confluence/display/KAFKA/KIP-959%3A+Add+BooleanConverter+to+Kafka+Connect This KIP is a counterpart of KIP-907: Add Boolean Serde to public interface

[jira] [Created] (KAFKA-15247) OutOfMemoryError in SaslClientAuthenticator during server restart

Dave Crighton created KAFKA-15247: - Summary: OutOfMemoryError in SaslClientAuthenticator during server restart Key: KAFKA-15247 URL: https://issues.apache.org/jira/browse/KAFKA-15247 Project: Kafka

[VOTE] KIP-934: Add DeleteTopicPolicy

Hi All, I'd like to start the vote for KIP-934: Add DeleteTopicPolicy: https://cwiki.apache.org/confluence/x/-xE0Dw Regards, Jorge.

Re: [VOTE] KIP-930: Tiered Storage Metrics

+1 (non-binding) -- Kamal On Tue, Jul 25, 2023 at 11:30 AM Abhijeet Kumar wrote: > Hi All, > > I would like to start the vote for KIP-930 Tiered Storage Metrics. > > The KIP is here: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics > > Regards > Abhijeet.

Re: [VOTE] KIP-930: Tiered Storage Metrics

+1 (binding) from me. Thanks. Luke On Tue, Jul 25, 2023 at 7:51 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > +1 (non-binding) > > Thanks, Abhijeet! > > > On Tue, 25 Jul 2023 at 14:22, Abhijeet Kumar > wrote: > > > Please find the updated link to the KIP: > > > > >

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

Hi Colin, Some more comments: 1. In the KIP, we mentioned "controller heartbeats", but it is not explained anywhere. I think "controller heartbeats" = controller registration", is that correct? If no, please explain more about it in the KIP. 2. Following this question: > Which endpoint will the

Re: [VOTE] KIP-930: Tiered Storage Metrics

+1 (non-binding) Thanks, Abhijeet! On Tue, 25 Jul 2023 at 14:22, Abhijeet Kumar wrote: > Please find the updated link to the KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics > > Updated the KIP as per our conversation on the

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

Hi Sagar, Thanks for the updates. I had a few more follow up questions: > I have added that a better way of doing that would be > via KIP-875. Also, I didn't want to include any mechamisms > for users to meddle with the offsets topic. Allowing tombstone > records via this method would be akin to

Re: [VOTE] KIP-930: Tiered Storage Metrics

Please find the updated link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics Updated the KIP as per our conversation on the discussion thread. On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar wrote: > Hi All, > > I would like to

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Abhijeet, Thanks for this KIP, I pretty much agree with the renaming and new names look good to me. Cheers, Jorge. On Tue, 25 Jul 2023 at 12:56, Satish Duggana wrote: > Hi Abhijeet, > Thanks for keeping this KIP only to renaming the existing metrics for > better clarity. These new names

Re: Apache Kafka 3.6.0 release

Hi Satish Heads up KIP-951 is under voting. This could be delayed by a day or two for the 3.6 KIP deadline. On Tue, Jul 25, 2023 at 4:19 AM Satish Duggana wrote: > Thanks Colin for the update on the mentioned KIPs. > > ~Satish. > > On Mon, 24 Jul 2023 at 23:09, Colin McCabe wrote: > > > > Hi

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Abhijeet, Thanks for keeping this KIP only to renaming the existing metrics for better clarity. These new names look good to me. ~Satish. On Tue, 25 Jul 2023 at 13:12, Luke Chen wrote: > > Hi Abhijeet, > > Thanks for the KIP! > I don't have much preference for the name changing. > But if it

Re: [DISCUSS] KIP-953: partition method to be overloaded to accept headers as well.

Hi Jack, Thanks I have a couple of final comments and then I am good. 1) Can you elaborate on the Javadocs of the partition headers argument to specify that a null headers value is equivalent to invoking the older partition method? It is apparent but generally good to call out. 2) In the

Question about Lag Calculations in Apache Kafka Source Code

Hi everyone, I am interested in understanding how the broker performs lag calculations. Specifically, I would like to explore the possibility of improving the calculation method to use the latest stable offset instead of the latest offset. I noticed that there might be differences between the

Re:Request permission to contribute

Hi Guozhang, Thanks for your attention. I'm a contributor of other Apache project (Ignite). Now I don't have permissions to create a page in `Apache Kafka` space on wiki (confluence) [1] [1] https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Abhijeet, Thanks for the KIP! I don't have much preference for the name changing. But if it could confuse other people, it's good to make it clear. Thank you. Luke On Tue, Jul 25, 2023 at 2:53 PM Abhijeet Kumar wrote: > Hi Kamal, > > As we discussed offline, I will rename this KIP so that

[jira] [Created] (KAFKA-15246) CoordinatorContext should be protected by a lock

David Jacot created KAFKA-15246: --- Summary: CoordinatorContext should be protected by a lock Key: KAFKA-15246 URL: https://issues.apache.org/jira/browse/KAFKA-15246 Project: Kafka Issue Type:

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2034

See

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Kamal, As we discussed offline, I will rename this KIP so that it only captures the aspect of renaming the previously added metrics to remove ambiguity. I will create another KIP for RemoteIndexCache metrics and other relevant tiered storage metrics. On Tue, Jul 25, 2023 at 12:03 PM Kamal

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

Hi Abhijeet, Thanks for the KIP! We are changing the metric names from what was proposed in the KIP-405 and adding new metrics for RemoteIndexCache. In the KIP, it's not clear whether we are renaming the aggregate broker level metrics for remote copy/fetch/failed-copy/failed-fetch. Are these

[VOTE] KIP-930: Tiered Storage Metrics

Hi All, I would like to start the vote for KIP-930 Tiered Storage Metrics. The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics Regards Abhijeet.