Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #3003

See Changes: -- [...truncated 395275 lines...] [2024-06-13T06:55:13.212Z] [2024-06-13T06:55:13.212Z] Gradle Test Run :core:test > Gradle Test Executor 96 > ZkConfigMigrat

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

Hi, 106 : Thanks for the clarification. Actually, this is not what I expected, but I better understand the performance issues regarding the state store iteration. If this is how it should be designed, it is fine for me as long as it is clear that the repartition must be done before the dedupli

[jira] [Resolved] (KAFKA-16557) Fix OffsetFetchRequestState.toString()

[ https://issues.apache.org/jira/browse/KAFKA-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16557. Resolution: Fixed > Fix OffsetFetchRequestState.toString() > -

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #38

See

Re: [VOTE] KIP-1035: StateStore managed changelog offsets

Thanks Nick! Great KIP! +1 (binding) Best, Bruno On 6/13/24 2:31 AM, Matthias J. Sax wrote: Thanks Nick. +1 (binding) Looking forward to get this all merged! -Matthias On 6/12/24 9:36 AM, Nick Telford wrote: Hi everyone, I'd like to call a vote on KIP-1035[1]. Regards, Nick 1: https

[jira] [Resolved] (KAFKA-16643) Add ModifierOrder checkstyle rule

[ https://issues.apache.org/jira/browse/KAFKA-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16643. Fix Version/s: 3.9.0 Resolution: Fixed > Add ModifierOrder checkstyle rule > --

Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

Hi Fan, Thanks for the feedback. In some ways, that takes us a step towards the behaviour of a schema registry converter such as io.confluent.connect.json.JsonSchemaConverter - with the converter able to fetch a schema hosted at a central location, which is an interesting idea. My preference

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi Greg, > However, the KIP still omits the MaskField and ValueToKey transformations. This looks like just a typo to me, should we update the KIP before closing the vote? Good catch. I have just updated the KIP. I think that we can close the voting and mark it as approved, right? Thank you all.

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Thanks Matthias and Greg. Have updated the KIP based on Matthias's comments. >> 100: Config names are part of the public interface, and the KIP should >> not say "none" in this section, but call out which configs are >> deprecated and which ones are newly added. Updated kip. >> 101: Nit. In "P

RE: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

Hi Hector, Currently, our focus is solely on Sink connectors. We have not encountered a scenario where it would be applicable for a source connector therefore we couldn’t identify a need for creating an external schema file when writing to a topic. If you can provide us a use case for implement

RE: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

Hi Greg Thanks for your comments/questions. 1. You raise a good point and I agree that when compression is enabled on a large batch size, then the impact on message size will not be as significant. However, this would still be useful enhancement for scenarios where the JSON message does not

Re: [VOTE] KIP-1049: Add config log.summary.interval.ms to Kafka Streams

+1 (binding) thanks for the KIP! On Thu, Jun 13, 2024 at 2:32 AM Matthias J. Sax wrote: > > +1 (binding) > > On 6/11/24 1:17 PM, Sophie Blee-Goldman wrote: > > +1 (binding) > > > > Thanks for the KIP! > > > > On Tue, Jun 11, 2024 at 5:37 AM jiang dou wrote: > > > >> HI > >> I would like to star

Re: [VOTE] 3.7.1 RC1

Hi Edoardo, It is late, but not too late. I have cherry-picked your change to the 3.7 branch and I'll build a second release candidate. If you could have a look at the first RC, please let me know if you spot any issues with it that can be avoided in the next RC. Thanks, -- Igor

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #39

See

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Hi everyone, On a semantic note, would it perhaps make more sense to rename them " uncaught." instead? These handlers are essentially the "last resort" exception handlers, because Exceptions can be caught *within* a component, e.g. a Deserializer can catch and handle an exception without the confi

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #177

See Changes: -- [...truncated 2938 lines...] [2024-06-13T12:32:18.908Z] > Task :connect:json:checkstyleTest [2024-06-13T12:32:22.085Z] > Task :connect:basic-auth-extension:spo

Re: [VOTE] 3.7.1 RC1

Thanks ! On Thu, 13 Jun 2024 at 13:20, Igor Soarez wrote: > > Hi Edoardo, > > It is late, but not too late. I have cherry-picked your change > to the 3.7 branch and I'll build a second release candidate. > > If you could have a look at the first RC, please let me know if > you spot any issues wit

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

Hi everyone, Bumping this thread, as I haven't yet had any replies. Kind regards, Harry On Thursday, 6 June 2024 at 17:59, Harry Fallows wrote: > Hi everyone, > > I would like to propose a change to allow the static configuration of leader > and follower replication throttling rates. > > T

[jira] [Created] (KAFKA-16949) System test test_dynamic_logging in connect_distributed_test is failing

Sagar Rao created KAFKA-16949: - Summary: System test test_dynamic_logging in connect_distributed_test is failing Key: KAFKA-16949 URL: https://issues.apache.org/jira/browse/KAFKA-16949 Project: Kafka

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi all, I noticed that the PR of Mario effected only InsertField and ExtractField, so I'd like to amend others SMTs in that KIP because I have experienced to develop custom SMTs. Can I join to contribute (open PR)? On 2024/06/13 15:59:36 Greg Harris wrote: > Hey Mario, > > Thanks for updating t

[jira] [Resolved] (KAFKA-13298) Improve documentation on EOS KStream requirements

[ https://issues.apache.org/jira/browse/KAFKA-13298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13298. - Fix Version/s: 3.1.0 Assignee: Andy Chambers Resolution: Fixed > Improve

RE: [DISCUSS] KIP-1052: Align the naming convention for config and default variables in *Config classes

Hi Eric, Apologies for the KIP number conflict. I'm not sure how the community handles these race conditions, but I think we've got it sorted. Also not sure if this is the right action, but it might be less confusing if you re-send your DISCUSS email to the dev list with the correct title and

[jira] [Created] (KAFKA-16950) Define Persister and Share Coordinator RPCs

Apoorv Mittal created KAFKA-16950: - Summary: Define Persister and Share Coordinator RPCs Key: KAFKA-16950 URL: https://issues.apache.org/jira/browse/KAFKA-16950 Project: Kafka Issue Type: Sub

RE: [DISCUSS] KIP-1052: Enable warmup in producer performance test

Hello again Kafka devs, I'd like to again call attention to this KIP for discussion. Apparently, we encountered a race condition when choosing KIP numbers, but hopefully it's straightened out now. Regards, Matt -Original Message- From: Welch, Matt Sent: Thursday, June 6, 2024 4:44 P

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Bruno, We focused our effort (well, mostly Seb and Loic :)) on KIP-1033, that's why not much progress has been made on this one yet. Regarding your points: B1: It is up to the user to specify the DLQ topic name and to implement a potential differentiation. I tend to think that having one DLQ p

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hey Mario, Thanks for updating the KIP. +1 (binding) Greg On Thu, Jun 13, 2024 at 1:32 AM Mario Fiore Vitale wrote: > Hi Greg, > > > However, the KIP still omits the MaskField and ValueToKey > transformations. > This looks like just a typo to me, should we update the KIP before closing > the v

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Hi Punsak, If nobody has signaled their intent to contribute that work yet (which I believe is the case), you are welcome to take it on yourself! Cheers, Cheers On Thu, Jun 13, 2024, 12:52 Punsak Incham wrote: > My customers are using these SMTs, I think it can affect to their project > (abou

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

Hi Hector, I did see your colleague's KIP, and I actually mentioned it in the KIP that I have written. As I see it, both of these KIPs move towards more easily configurable replication throttling and both should be implemented. KIP-1009 makes it easier to enable throttling and KIP-1051 makes it

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

My customers are using these SMTs, I think it can affect to their project (about data correctness) in the future. (You may think why I'm not developing it and send to my customers directly? Because they will use only SMTs that published by Kafka.) On 2024/06/13 16:38:45 Punsak Incham wrote: > H

[jira] [Created] (KAFKA-16953) Implement and test listeners in DescribeQuorumResponse

José Armando García Sancio created KAFKA-16953: -- Summary: Implement and test listeners in DescribeQuorumResponse Key: KAFKA-16953 URL: https://issues.apache.org/jira/browse/KAFKA-16953 Pro

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hey all -- was just wondering where we currently stand Re: delaying 3. for the KRaft KIPs vs doing a 3.9 release I know we don't want to have to wait for a whole release cycle to ship these KRaft features, but delaying 3.8 up to month is also rather difficult to swallow. I just wanted to throw an

Re: Kafka certification

Hi Matheus, The are only one that I know, provided by Confluent: https://www.confluent.io/certification/ If someone knows more, I am also interested to know. Best regards , Caio Guedes On Thu, 13 Jun 2024 at 01:55 Matheus Pereira dos Santos < matheuspereira8...@gmail.com> wrote: > Hi! > > I a

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hey Sophie, Thanks for bringing this up, since I was also lacking clarity here. One thing to note is that such a proposal (parallel development) was discussed as part of KIP-1012 and it was rejected https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Sophie, I have a call tomorrow with José to clarify the estimates for KIP-853. I also wouldn't like to delay the release for a month or more. Regarding your proposal, I find it would be a good way forward, +1 from my side. I also find this release and what should include is a hot topic. What

[jira] [Created] (KAFKA-16954) Move consumer leave operations on close to background thread

Lianet Magrans created KAFKA-16954: -- Summary: Move consumer leave operations on close to background thread Key: KAFKA-16954 URL: https://issues.apache.org/jira/browse/KAFKA-16954 Project: Kafka

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Justine, I know we discarded parallel branching, but it was under the scope of 3.8.0 and with the KIPs no yet approved. We could also not do a parallel release, but rather "quick" 3.9 and then start with 4.0. Best - Josep Prat Open Source Engineering Director, Aiven josep.p...@

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Sophie and Justine, I share your concerns about delaying 3.8 in order to give the Kraft KIPs more time for implementation. I raised them in the discussion for KIP-1012 [1]: > I think there is a > risk that features that are on-time and eligible for a 3.8 release > could be delayed by some KIPs

Re: [DISCUSS] Apache Kafka 3.8.0 release

is this where I ask to join? On Thu, Jun 13, 2024 at 1:20 PM Greg Harris wrote: > Hi Sophie and Justine, > > I share your concerns about delaying 3.8 in order to give the Kraft KIPs > more time for implementation. I raised them in the discussion for KIP-1012 > [1]: > > > I think there is a > >

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi, We follow a time based release process precisely to avoid this type of issues. Rushing to complete a feature and merging it just before the release puts pressure on the contributors, and leaves little time to properly test it. Especially on a complex feature like KIP-853. I'd be +1 on releasi

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Mickael, I agree +1 to proceeding with the 3.8 release on-time. I'm fine with cutting the 3.9 branch immediately after the Kraft KIPS are feature-complete, or 4 months after the 3.8 release, whichever comes first. Thanks, Greg On Thu, Jun 13, 2024 at 1:29 PM Mickael Maison wrote: > Hi, > >

[jira] [Resolved] (KAFKA-16001) Migrate ConsumerNetworkThreadTest away from ConsumerTestBuilder

[ https://issues.apache.org/jira/browse/KAFKA-16001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16001. - Resolution: Fixed > Migrate ConsumerNetworkThreadTest away from ConsumerTestBuilder > --

Re: [DISCUSS] KIP-1051 Statically configured log replication throttling

Hi Harry, A colleague of mine opened KIP-1009: Add Broker-level Throttle Configurations, which aims to achieve the same goal (although from a different angle). Can you please take a look and see if this would work for the things you have in mind? Maybe we can partner and coalesce around either

[jira] [Created] (KAFKA-16952) Do not bump broker epoch when re-registering the same incarnation

Colin McCabe created KAFKA-16952: Summary: Do not bump broker epoch when re-registering the same incarnation Key: KAFKA-16952 URL: https://issues.apache.org/jira/browse/KAFKA-16952 Project: Kafka

[jira] [Created] (KAFKA-16951) TransactionManager should rediscover coordinator on disconnection

Graham Campbell created KAFKA-16951: --- Summary: TransactionManager should rediscover coordinator on disconnection Key: KAFKA-16951 URL: https://issues.apache.org/jira/browse/KAFKA-16951 Project: Kafk

Kafka upgrade recommendation

Hi there, We have few clusters in AWS world and being forced to bump cluster version. So, we are considering options to be in safer way possible, mitigating risks, and without outage as possible (our business is 24/7). Upgrade would be from 2.2.1 / 2.6.2 to 3.7.0 (MSK recommended version). Thi

Re: [DISCUSS] KIP-1052: Enable warmup in producer performance test

Hi Matt, Yes I forgot to update the KIP counter after creating a KIP. I changed mine to 1053. We should be all good now. Cheers, Eric On Thu, Jun 13, 2024 at 3:08 PM Welch, Matt wrote: > Hello again Kafka devs, > > I'd like to again call attention to this KIP for discussion. > Apparently, we e

[jira] [Created] (KAFKA-16955) ConcurrentModification exception thrown by KafkaStream threadState access

Rohan Desai created KAFKA-16955: --- Summary: ConcurrentModification exception thrown by KafkaStream threadState access Key: KAFKA-16955 URL: https://issues.apache.org/jira/browse/KAFKA-16955 Project: Kafk

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #178

See Changes: -- [...truncated 2919 lines...] [2024-06-13T22:08:13.874Z] Note: /home/jenkins/workspace/Kafka_kafka_3.7/core/src/main/java/kafka/log/remote/RemoteLogManager.jav

[jira] [Resolved] (KAFKA-16076) RestClient Interrupting the thread in case of InterruptedException

[ https://issues.apache.org/jira/browse/KAFKA-16076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16076. Fix Version/s: 3.9.0 Resolution: Fixed > RestClient Interrupting the thread in case

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #41

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #179

See Changes: -- [...truncated 2961 lines...] [2024-06-14T00:13:23.859Z] > Task :connect:basic-auth-extension:compileTestJava [2024-06-14T00:13:23.859Z] > Task :connect:basic-a

[jira] [Resolved] (KAFKA-16879) SystemTime should use singleton mode

[ https://issues.apache.org/jira/browse/KAFKA-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16879. Fix Version/s: 3.9.0 Resolution: Fixed > SystemTime should use singleton mode > ---

Re: [DISCUSS] KIP-1052: Align the naming convention for config and default variables in *Config classes

Might be good to restart a new discuss thread using the new KIP-1035 number to split both threads? I added the KIP to the table in the wiki page in the mean time. In Kafka Streams we did rename config names in the past in the exact same way as proposed: deprecate the old config, and add the

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Using `uncaught.` prefix would be fine with me. (Even if I find it a little be redundant personally?) 101 follow-up: Given that `DESERIALIZATION_EXCEPTION_HANDLER_CLASS_DOC` is public we cannot just rename it, but must follow the same deprecation pattern as for the config name itself. (It s

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #109

See

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.3 #194

See

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Thanks! Chris. One question : We will need to wait to fix all SMTs in the KIP and release it to user, or we can split some fixed SMTs to user early? (I noticed that the PR of Mario ready to merge and it effect only ExtractField and InsertField) On 2024/06/13 17:01:49 Chris Egerton wrote: > Hi

Re: [DISCUSS] Apache Kafka 3.8.0 release

I'm also +1 to have 3.8 release on time, and have a shorter release cycle for v3.9. But we have to explicitly define what "shorter release cycle" means. I agree with Greg's suggestion: cutting the 3.9 branch immediately after the Kraft KIPS are feature-complete, or 4 months after the 3.8 release, w

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.5 #110

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #42

See

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #3007

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #176

See Changes: -- [...truncated 528405 lines...] [2024-06-14T05:53:05.874Z] [2024-06-14T05:53:05.874Z] Gradle Test Run :core:integrationTest > Gradle Test Executor 169 > Kafka

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

We don't have to patch every SMT in the same release, we can definitely move incrementally. We'll just have to note in the release notes and the KIPs page ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) when the change for the KIP was applied to each SMT. On Thu, J

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #195

See Changes: -- [...truncated 339865 lines...] [2024-06-14T06:03:13.659Z] org.apache.kafka.streams.integration.StoreQueryIntegrationTest > shouldQuerySpecificActivePartition

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

Thanks! Chris. Waiting @Mario Fiore Vitale to close the voting and mark it as approved. Cheers. On 2024/06/14 06:10:15 Chris Egerton wrote: > We don't have to patch every SMT in the same release, we can > definitely move incrementally. We'll just have to note in the release > notes and the KIPs