Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2810

See

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

Somewhat minor point overall, but it actually drives me crazy that you can't get access to the taskId of a StateStore until #init is called. This has caused me a huge headache personally (since the same is true for processors and I was trying to do something that's probably too hacky to actually

Re: [DISCUSS] KIP-1036: Extend RecordDeserializationException exception

I think the bigger question here is: why is checkstyle complaining about this import? Does anyone know? On Thu, Apr 11, 2024 at 11:12 AM Frédérik Rouleau wrote: > Hi everyone, > > I have made some changes to take in account comments. I have replaced the > ConsumerRecord by Record. As it was not

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

Fully agree about creating a new class for the bits of ProcessingContext that are specific to metadata only. In fact, more or less this same point just came up in the related KIP 1034 for DLQs, since the RecordMetadata can't always be trusted to remain immutable. Maybe it's possible to solve both

[jira] [Created] (KAFKA-16545) Auto adjust the replica factor according to number of broker when using ClusterTestExtensions

Chia-Ping Tsai created KAFKA-16545: -- Summary: Auto adjust the replica factor according to number of broker when using ClusterTestExtensions Key: KAFKA-16545 URL: https://issues.apache.org/jira/browse/KAFKA-16545

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Thanks for the KIP, this will make a lot of people very happy. Wanted to chime in on a few points that have been raised so far and add some of my own (numbering with an S to distinguish my points from the previous ones) S1. > 1.a I really meant ProducerRecord, that's the class used to forward

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Thanks Ivan! LGTM On Fri, Apr 12, 2024, 13:38 Ivan Yurchenko wrote: > Hi Chris and all, > > Thank you for your feedback. Your proposals seems good to me. I did these > changed to the KIP, please have a look at the change [1] > > Best, > Ivan > > [1] >

[jira] [Created] (KAFKA-16544) DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames should return null instead of throwing NPE

Chia-Ping Tsai created KAFKA-16544: -- Summary: DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames should return null instead of throwing NPE Key: KAFKA-16544 URL:

Re: [VOTE] KIP-1022 Formatting and Updating Features

Hi, Justine, Thanks for the KIP. +1 Jun On Wed, Apr 10, 2024 at 9:13 AM José Armando García Sancio wrote: > Hi Justine, > > +1 (binding) > > Thanks for the improvement. > -- > -José >

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Andrew, Thanks a lot for your review, plenty of good points! 11. Typo fixed, good cach. 12. I do agree with you and Nick also mentioned it, I updated the KIP to mention that context headers should be forwarded. 13. Good catch, to be consistent with KIP-298, and without a strong opinion from

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2809

See

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi Jun, Ok sounds good. Justine On Fri, Apr 12, 2024 at 10:17 AM Jun Rao wrote: > Hi, Justine, > > unstable.metadata.versions.enable is an internal configuration. So, we > could probably just remove it instead of depreciation. Also, it would be > useful to make it clear that

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

Hi Chris and all, Thank you for your feedback. Your proposals seems good to me. I did these changed to the KIP, please have a look at the change [1] Best, Ivan [1] https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=14=12 On Thu, Apr 11, 2024, at 10:49, Chris

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

Hi, Justine, unstable.metadata.versions.enable is an internal configuration. So, we could probably just remove it instead of depreciation. Also, it would be useful to make it clear that unstable.feature.versions.enable is an internal configuration. Thanks, Jun On Thu, Apr 11, 2024 at 11:16 AM

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Nick, 1. Good point, that's less impactful than a custom interface, I just updated the KIP with the new signature. 1.a I really meant ProducerRecord, that's the class used to forward to downstream processors in the PAPI. The only information missing in this class is the topic name. I also

Re: [DISCUSS] Transition Required from RemoteLogSegment to LocalLogSegment during Fetch Request

@Divij Vaidya Satish @Kamal Chandraprakash ,Luke As you are more aware of the code flow path , Your suggestion would help me to advance further on this. On Thu, Apr 11, 2024, 21:20 Arpit Goyal wrote: > Hi All, > When tackling the issue outlined in >

[jira] [Created] (KAFKA-16543) There may be ambiguous deletions in the `cleanupGroupMetadata` when the generation of the group is less than or equal to 0

hudeqi created KAFKA-16543: -- Summary: There may be ambiguous deletions in the `cleanupGroupMetadata` when the generation of the group is less than or equal to 0 Key: KAFKA-16543 URL:

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Damien, Sebastien and Loic, Thanks for the KIP. The DLQ pattern is well established and bringing this to Kafka Streams is a good improvement. I do plan to add DLQ support to share groups in the future, once KIP-932 is complete. Having broad support in Kafka for DLQs is great. I have a few

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2808

See Changes: -- [...truncated 341788 lines...] [2024-04-12T10:26:10.208Z] [2024-04-12T10:26:10.208Z] Gradle Test Run :streams:test > Gradle Test Executor 94 >

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Damien and Sebastien, 1. I think you can just add a `String topic` argument to the existing `withDeadLetterQueueRecord(ProducerRecord deadLetterQueueRecord)` method, and then the implementation of the exception handler could choose the topic to send records to using whatever logic the user

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

I think there is an issue in the KIP. Basically the kip says, if the PID is found in either of the Bloom filters then no action is taken If the PID is not found then it is added and the quota rating metrics are incremented. In this case long running PIDs will be counted multiple times. Let's

Re: [DISCUSS] KIP-936 Throttle number of active PIDs

Initial code is available at https://github.com/Claudenw/kafka/blob/KIP-936/storage/src/main/java/org/apache/kafka/storage/internals/log/ProducerIDQuotaManager.java On Tue, Apr 9, 2024 at 2:37 PM Claude Warren wrote: > I should also note that the probability of false positives does not fall >

Re: [DISCUSS] KIP-932: Queues for Kafka

Hi, 77. I’ve updated the KIP to use log retention rather than log compaction. The basic ideas of what to persist are unchanged. It makes a few changes: * It changes the record names: ShareCheckpoint -> ShareSnapshot and ShareDelta -> ShareUpdate. They’re equivalent, but renaming makes it

[DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hello Nick, Thanks for your remark. 5. The Record posted in the DLQ topic is not intended to be reprocessed by the Kafka Streams application. If users want to reprocess the records, it is up to them to take into considerations the stream time and / or if the window has closed regards

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

+1 (binding), thanks Omnia! On Fri, Apr 12, 2024, 03:46 Mickael Maison wrote: > Hi Omnia, > > +1 (binding), thanks for the KIP! > > Mickael > > On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim > wrote: > > > > Hi everyone, I would like to start a voting thread for KIP-1031: Control > offset

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Nick, Thanks a lot for your review and your useful comments! 1. It is a good point, as you mentioned, I think it would make sense in some use cases to have potentially multiple DLQ topics, so we should provide an API to let users do it. Thinking out-loud here, maybe it is a better approach to

Re: [VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

Hi Ivan, Thanks for reviving this KIP, I think it will be a useful addition to Connect! +1 (binding) Cheers, Yash On Tue, Apr 9, 2024 at 4:23 AM Knowles Atchison Jr wrote: > +1 (non binding) > > On Mon, Apr 8, 2024, 3:30 PM Chris Egerton > wrote: > > > Thanks Ivan! +1 (binding) from me. > >

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Oh, and one more thing: 5. Whenever you take a record out of the stream, and then potentially re-introduce it at a later date, you introduce the potential for record ordering issues. For example, that record could have been destined for a Window that has been closed by the time it's re-processed.

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Damien, Thanks for the KIP! Dead-letter queues are something that I think a lot of users would like. I think there are a few points with this KIP that concern me: 1. It looks like you can only define a single, global DLQ for the entire Kafka Streams application? What about applications that

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

In a general way, if the user does not configure the right ACL, that would be a security issue, but that's true for any topic. This KIP allows users to configure a Dead Letter Queue without writing custom Java code in Kafka Streams, not at the topic level. A lot of applications are already

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

On further thought, it's clear that this can't work for one simple reason: StateStores don't know their associated TaskId (and hence, their StateDirectory) until the init() call. Therefore, committedOffset() can't be called before init(), unless we also added a StateStoreContext argument to

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.7 #133

See

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

My concern is that someone would create a dead letter queue on a sensitive topic and not get the ACL correct from the start. Thus causing potential confidential data leak. Is there anything in the proposal that would prevent that from happening? If so I did not recognize it as such. On Fri,

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

Hi Omnia, +1 (binding), thanks for the KIP! Mickael On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim wrote: > > Hi everyone, I would like to start a voting thread for KIP-1031: Control > offset translation in MirrorSourceConnector >

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

Hi Claude, In this KIP, the Dead Letter Queue is materialized by a standard and independant topic, thus normal ACL applies to it like any other topic. This should not introduce any security issues, obviously, the right ACL would need to be provided to write to the DLQ if configured. Cheers,

[VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

Hi everyone, I would like to start a voting thread for KIP-1031: Control offset translation in MirrorSourceConnector https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector For comments or feedback please check the discussion thread here

Re: [DISCUSS] KIP-1034: Dead letter queue in Kafka Streams

I am new to the Kafka codebase so please excuse any ignorance on my part. When a dead letter queue is established is there a process to ensure that it at least is defined with the same ACL as the original queue? Without such a guarantee at the start it seems that managing dead letter queues will

[jira] [Created] (KAFKA-16542) The remote node is not a BROKER that supports the METADATA api. (org.apache.kafka.clients.admin.internals.AdminMetadataManager)

Hiro created KAFKA-16542: Summary: The remote node is not a BROKER that supports the METADATA api. (org.apache.kafka.clients.admin.internals.AdminMetadataManager) Key: KAFKA-16542 URL:

[jira] [Resolved] (KAFKA-16510) java.lang.OutOfMemoryError in kafka-metadata-quorum.sh

[ https://issues.apache.org/jira/browse/KAFKA-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiro resolved KAFKA-16510. -- Resolution: Fixed > java.lang.OutOfMemoryError in kafka-metadata-quorum.sh >

[jira] [Resolved] (KAFKA-16473) KafkaDockerWrapper uses wrong cluster ID when formatting log dir

[ https://issues.apache.org/jira/browse/KAFKA-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-16473. --- Fix Version/s: 3.8.0 3.7.1 Resolution: Fixed > KafkaDockerWrapper uses

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2807

See Changes: -- [...truncated 465869 lines...] [2024-04-12T05:22:43.131Z] [2024-04-12T05:22:43.131Z] Gradle Test Run :streams:test > Gradle Test Executor 106 >