Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Very interesting discussion. Seems a central point is the question about "how generic" we approach this, and some people think we need to be conservative and others think we should try to be as generic as possible. Personally, I think following a limited scope for this KIP (by explicitly

[jira] [Resolved] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest

[ https://issues.apache.org/jira/browse/KAFKA-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16678. Fix Version/s: 3.8.0 Resolution: Fixed > Remove unimplementedquorum from

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

@Loic, yes, what you describe is exactly what I had in mind. @Sophie, can you elaborate a little bit? First of all, I agree that it makes sense to maintain the two separate callbacks for the ProductionExceptionHandler, since one of them is specifically for serialization exceptions while the

[jira] [Created] (KAFKA-16691) Support for nested structures: TimestampConverter

Jorge Esteban Quilcate Otoya created KAFKA-16691: Summary: Support for nested structures: TimestampConverter Key: KAFKA-16691 URL: https://issues.apache.org/jira/browse/KAFKA-16691

[jira] [Created] (KAFKA-16690) Support for nested structures: HeaderFrom

Jorge Esteban Quilcate Otoya created KAFKA-16690: Summary: Support for nested structures: HeaderFrom Key: KAFKA-16690 URL: https://issues.apache.org/jira/browse/KAFKA-16690 Project:

[jira] [Resolved] (KAFKA-14226) Introduce support for nested structures

[ https://issues.apache.org/jira/browse/KAFKA-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge Esteban Quilcate Otoya resolved KAFKA-14226. -- Resolution: Fixed Merged:

[jira] [Created] (KAFKA-16689) Move LogValidatorTest to storage module

Chia-Ping Tsai created KAFKA-16689: -- Summary: Move LogValidatorTest to storage module Key: KAFKA-16689 URL: https://issues.apache.org/jira/browse/KAFKA-16689 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Alieh, Thanks for the KIP. The motivation talks about very specific cases, but the interface is generic. [AL1] If the interface evolves in the future I think we could have the following confusion: 1. A user implemented SWALLOW action for both RecordTooLargeException and

RE: DISCUSS KIP-984 Add pluggable compression interface to Kafka

Hi Greg, Thank you for your thoughtful response. Our motivation here is to accelerate compression with the use of hardware accelerators. If the community prefers, we would be happy to contribute code to support compression accelerators, but we believe that introducing a pluggable

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2881

See

[jira] [Created] (KAFKA-16688) SystemTimer leaks resources on close

Gaurav Narula created KAFKA-16688: - Summary: SystemTimer leaks resources on close Key: KAFKA-16688 URL: https://issues.apache.org/jira/browse/KAFKA-16688 Project: Kafka Issue Type: Test

Re: [VOTE] KIP-924: customizable task assignment for Streams

Thanks all! KIP-924 is accepted with 4 +1 (binding) votes from Sophia Blee-Goldman, Matthias Sax, Bruno Cadonna, and Lucas Brutschy On Tue, Apr 16, 2024 at 1:46 PM Rohan Desai wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-924%3A+customizable+task+assignment+for+Streams > > As

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

First of all, I agree that it makes sense to maintain the two separate callbacks for the ProductionExceptionHandler, since one of them is specifically for serialization exceptions while the other is used for everything/anything else. I also think we can take advantage of this fact to simplify

Re: Request version not enabled errors upgrading from Kafka 3.5 -> 3.6

Hi Johnson, Thanks for bringing this issue to the mailing list. I'm familiar with the change you are referring to. However, during the upgrade you should be hitting this code path and we should not sending requests to older version brokers.

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

+1 (binding) thanks for the KIP! On Fri, May 3, 2024 at 9:13 AM Matthias J. Sax wrote: > +1 (binding) > > On 5/3/24 8:52 AM, Federico Valeri wrote: > > Hi Fred, this is a useful addition. > > > > +1 non binding > > > > Thanks > > > > On Fri, May 3, 2024 at 4:11 PM Andrew Schofield > > wrote:

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Alieh and Chris, Thanks for clarifying 1) but I saw the motivation. I guess I just didn't understand how that would be ensured on the producer side. I saw the sample code -- is it just an if statement checking for the error before the handler is invoked? That seems a bit fragile. Can you clarify

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Omnia, Thanks for the detailed response. I agree that the client ID solution can be tricky (and could even run into the same problem if the client ID is not unique). As for the waiting one day -- that was not meant to be an exact value, but my point was that there will be some time where

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi, Thank you, Chris and Justine, for the feedback. @Chris 1) Flexibility: it has two meanings. The first meaning is the one you mentioned. We are going to cover more exceptions in the future, but as Justine mentioned, we must be very conservative about adding more exceptions. Additionally,

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Justine Thanks for the feedback > So consider a case where there is a storm for a given principal. We could > have a large mass of short lived producers in addition to some > "well-behaved" ones. My understanding is that if the "well-behaved" one > doesn't produce as frequently ie less than

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Hi Igor, thanks for the feedback and sorry for the late response. > 10 Given the goal is to prevent OOMs, do we also need to > limit the number of KafkaPrincipals in use? None of the Kafka quotas ever limited number of KafkaPrincipals and I don’t really think this is the issue as you just need

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2880

See

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Justine, The method signatures for the interface are indeed open-ended, but the KIP states that its uses will be limited. See the motivation section: > We believe that the user should be able to develop custom exception handlers for managing producer exceptions. On the other hand, this will

Request version not enabled errors upgrading from Kafka 3.5 -> 3.6

Hi folks, Awesome work you have been doing on this project! I was hoping I could get some help on an issue we are having in one of our Kafka clusters. Most of the clients on this cluster use exactly-once-semantics. The Kafka cluster currently runs version 3.5.2 and we were attempting an upgrade

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Alieh, I was out for KSB and then was also sick. :( To your point 1) Chris, I don't think it is limited to two specific scenarios, since the interface accepts a generic Exception e and can be implemented to check if that e is an instanceof any exception. I didn't see anywhere that specific

[jira] [Resolved] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

[ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15018. --- Fix Version/s: 3.8.0 Resolution: Fixed > Potential tombstone offsets corruption for

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

Hi Alieh, Sorry for the delay, I've been out sick. I still have some thoughts that I'd like to see addressed before voting. 1) If flexibility is the motivation for a pluggable interface, why are we only limiting the uses for this interface to two very specific scenarios? Why not also allow,

[jira] [Created] (KAFKA-16687) Native memory leak by Unsafe_allocatememory in Kafka Clients 3.7.0

FTR created KAFKA-16687: --- Summary: Native memory leak by Unsafe_allocatememory in Kafka Clients 3.7.0 Key: KAFKA-16687 URL: https://issues.apache.org/jira/browse/KAFKA-16687 Project: Kafka Issue

[jira] [Created] (KAFKA-16686) Flakey tests in TopicBasedRemoteLogMetadataManagerTest

Gaurav Narula created KAFKA-16686: - Summary: Flakey tests in TopicBasedRemoteLogMetadataManagerTest Key: KAFKA-16686 URL: https://issues.apache.org/jira/browse/KAFKA-16686 Project: Kafka

[VOTE] KIP-1038: Add Custom Error Handler to Producer

Hi all, It seems that we have no more comments, discussions, or feedback on KIP-1038; therefore, I’d like to open voting for the KIP: Add Custom Error Handler to Producer Cheers, Alieh

[DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Hi Matthias, To sum up with the ProductionExceptionHandler callback methods (106) proposed changes. A new method ProductionExceptionHandler#handle is added with the following signature: > ProductionExceptionHandlerResponse handle(final ErrorHandlerContext context, > final ProducerRecord

[jira] [Resolved] (KAFKA-13329) Connect does not perform preflight validation for per-connector key and value converters

[ https://issues.apache.org/jira/browse/KAFKA-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-13329. --- Fix Version/s: 3.8.0 Resolution: Fixed > Connect does not perform preflight

[jira] [Resolved] (KAFKA-13328) Connect does not perform preflight validation for per-connector header converters

[ https://issues.apache.org/jira/browse/KAFKA-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-13328. --- Fix Version/s: 3.8.0 Resolution: Fixed > Connect does not perform preflight

Creating kafka wiki id

Hello, I want to create a KIP, but I don't have Kafka wiki id. I go to the page (https://cwiki.apache.org/confluence/signup.action) but it doesn't have a button to regist an account Please help me to create an account, Thank you

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

I have updated KIP-1042 with a proposal for how to reduce the time spent looking for matching wildcard patterns. Experimentally I see a reduction of 66-75% execution time. On Mon, May 6, 2024 at 9:03 PM Greg Harris wrote: > Hi Murali, > > Thanks for the KIP! > > I think I understand the

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2879

See

[jira] [Resolved] (KAFKA-16307) fix EventAccumulator thread idle ratio metric

[ https://issues.apache.org/jira/browse/KAFKA-16307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16307. - Fix Version/s: 3.8.0 Reviewer: David Jacot Resolution: Fixed > fix

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #150

See

[jira] [Created] (KAFKA-16685) RSM Task warn logs do not include parent exception trace

Jorge Esteban Quilcate Otoya created KAFKA-16685: Summary: RSM Task warn logs do not include parent exception trace Key: KAFKA-16685 URL: https://issues.apache.org/jira/browse/KAFKA-16685

[jira] [Created] (KAFKA-16684) FetchResponse#responseData could return incorrect data

Chia-Ping Tsai created KAFKA-16684: -- Summary: FetchResponse#responseData could return incorrect data Key: KAFKA-16684 URL: https://issues.apache.org/jira/browse/KAFKA-16684 Project: Kafka

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

Hi Christo, Thanks for the update! For both the policies "retain" and "delete", can we maintain the same approach to delete the segments async? > If the disablement policy is set to delete, the Log start offset (LSO) is updated to match the Local Log Start Offset and the remote log is deleted

[jira] [Created] (KAFKA-16683) Extract security-related helpers from scala.TestUtils to java class

Chia-Ping Tsai created KAFKA-16683: -- Summary: Extract security-related helpers from scala.TestUtils to java class Key: KAFKA-16683 URL: https://issues.apache.org/jira/browse/KAFKA-16683 Project:

[jira] [Created] (KAFKA-16682) Rewrite JassTestUtils by Java

Chia-Ping Tsai created KAFKA-16682: -- Summary: Rewrite JassTestUtils by Java Key: KAFKA-16682 URL: https://issues.apache.org/jira/browse/KAFKA-16682 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16681) Rewrite MiniKDC by Java

Chia-Ping Tsai created KAFKA-16681: -- Summary: Rewrite MiniKDC by Java Key: KAFKA-16681 URL: https://issues.apache.org/jira/browse/KAFKA-16681 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-16680) Make ClusterTestExtensions support SASL

Chia-Ping Tsai created KAFKA-16680: -- Summary: Make ClusterTestExtensions support SASL Key: KAFKA-16680 URL: https://issues.apache.org/jira/browse/KAFKA-16680 Project: Kafka Issue Type: New

Re: [VOTE] KIP-924: customizable task assignment for Streams

Hi, thanks for the KIP! +1 (binding) Best, Lucas On Tue, May 7, 2024 at 9:37 AM Bruno Cadonna wrote: > > Thanks for the KIP! > > Looking forward to a well-structured task assignor! > > +1 (binding) > > Best, > Bruno > > On 5/3/24 2:44 AM, Matthias J. Sax wrote: > > I left one more nit on the

Re: [VOTE] KIP-924: customizable task assignment for Streams

Thanks for the KIP! Looking forward to a well-structured task assignor! +1 (binding) Best, Bruno On 5/3/24 2:44 AM, Matthias J. Sax wrote: I left one more nit on the discuss thread. But overall LGTM. +1 (binding) Thanks Rohan and Sophie for driving this KIP. -Matthias On 4/29/24 2:07

[jira] [Created] (KAFKA-16679) Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into

Chia-Ping Tsai created KAFKA-16679: -- Summary: Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into Key: KAFKA-16679 URL:

Re: [PR] MINOR: Add code signing key for Igor [kafka-site]

soarez merged PR #600: URL: https://github.com/apache/kafka-site/pull/600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Resolved] (KAFKA-16615) JoinGroup API for upgrading ConsumerGroup

[ https://issues.apache.org/jira/browse/KAFKA-16615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-16615. - Fix Version/s: 3.8.0 Reviewer: David Jacot Assignee: Dongnuo Lyu

[jira] [Created] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest

Chia-Ping Tsai created KAFKA-16678: -- Summary: Remove unimplementedquorum from EndToEndAuthorizationTest Key: KAFKA-16678 URL: https://issues.apache.org/jira/browse/KAFKA-16678 Project: Kafka

[jira] [Resolved] (KAFKA-16593) Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions

[ https://issues.apache.org/jira/browse/KAFKA-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-16593. Fix Version/s: 3.8.0 Resolution: Fixed > Rewrite DeleteConsumerGroupsTest by