Re: outerjoin not joining after window

2024-05-22 Thread Matthias J. Sax
of the application will process records? For example, if the input topics each have 6 partitions, and I use the repartition method to set the number of partitions for the streams to 2, how many instances of the application will process records? Thanks, Chad On Wed, May 1, 2024 at 6:47 PM Matthias J. Sax

[jira] [Commented] (KAFKA-16586) Test TaskAssignorConvergenceTest failing

2024-05-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848476#comment-17848476 ] Matthias J. Sax commented on KAFKA-16586: - I did suspect a bug, too, and did include the seed

[jira] [Updated] (KAFKA-16586) Test TaskAssignorConvergenceTest failing

2024-05-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16586: Description: {code:java} java.lang.AssertionError: Assertion failed in randomized test

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024-05-22 Thread Matthias J. Sax
/main/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java On Mon, May 20, 2024 at 9:05 PM Matthias J. Sax wrote: Had a discussion on https://issues.apache.org/jira/browse/KAFKA-15242 and it was pointed out, that we also need to do something about `FixedKeyRecord`. It does not have

Re: Permission to contribute to Apache Kafka

2024-05-22 Thread Matthias J. Sax
You should be all set. On 5/21/24 8:30 AM, Harry Fallows wrote: Hello, I am following the[Getting Started guide for writing KIPs](https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals). Could someone give me the permissions to write a KIP please? My Wiki ID is

Re: Request for Authorization to Create KIP

2024-05-22 Thread Matthias J. Sax
You should be all set. On 5/21/24 6:58 PM, 黃竣陽 wrote: I want to create a KIP, and my wiki id : m1a2st and Jira id : m1a2st, Thanks for your help. jiang dou 於 2024年5月22日 上午9:01 寫道: You should send your jira ID and wiki ID, Please refer to this address :

Re: Request to be added to kafka contributors list

2024-05-21 Thread Matthias J. Sax
_____ From: Matthias J. Sax Sent: Saturday, May 18, 2024 4:06 To: users@kafka.apache.org Subject: Re: 回复: Request to be added to kafka contributors list Did you sign out and sign in again? On 5/17/24 9:49 AM, Yang Fan wrote: Thanks Matthias, I still can't find "Assign to me" button beside A

[jira] [Resolved] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-15242. - Assignee: (was: Alexander Aghili) Resolution: Duplicate > FixedKeyProces

[jira] [Resolved] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-15242. - Assignee: (was: Alexander Aghili) Resolution: Duplicate > FixedKeyProces

Re: Fwd: Request to be added to kafka contributors list

2024-05-20 Thread Matthias J. Sax
Done. You should be all set :) -Matthias On 5/20/24 10:10 AM, bou...@ulukai.net wrote: Dear Apache Kafka Team,     I hope to post in the right place: my name is Franck LEDAY, under Apache-Jira ID "handfreezer".     I opened an issue as Improvement KAFKA-16707 but I failed to assigned

Re: Request for contributor list

2024-05-20 Thread Matthias J. Sax
What is your Jira ID? -Matthias On 5/20/24 9:55 AM, Brenden Deluna wrote: Hello, I am requesting to be added to the contributor list to take care of some tickets. Thank you.

Re: Release plan required

2024-05-20 Thread Matthias J. Sax
Zookeeper is already deprecated (since 3.5): https://kafka.apache.org/documentation/#zk_depr It's planned to be fully removed in 4.0 release. It's not confirmed yet, but there is a high probability that there won't be a 3.9 release, and that 4.0 will follow 3.8. -Matthias On 5/20/24 2:11

[jira] [Commented] (KAFKA-16801) Streams upgrade :test target doesn't find any junit tests

2024-05-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848022#comment-17848022 ] Matthias J. Sax commented on KAFKA-16801: - There packages contain code for system tests. We put

[jira] [Updated] (KAFKA-16794) Can't open videos in streams documentation

2024-05-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16794: Component/s: streams > Can't open videos in streams documentat

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024-05-20 Thread Matthias J. Sax
would extend `FixedKeyRecord`? -- Or maybe an even better solution? I could not think of something else so far. Thoughts? On 5/3/24 9:46 AM, Matthias J. Sax wrote: Please also update the KIP. To get a wiki account created, please request it via a commet on this ticket: https

[jira] [Commented] (KAFKA-15143) MockFixedKeyProcessorContext is missing from test-utils

2024-05-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848021#comment-17848021 ] Matthias J. Sax commented on KAFKA-15143: - As pointed out on KAFKA-15242 cf comments, we also

[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848020#comment-17848020 ] Matthias J. Sax commented on KAFKA-15242: - I don't think that `TestRecord` has anything to do

Re: 回复: Request to be added to kafka contributors list

2024-05-17 Thread Matthias J. Sax
Did you sign out and sign in again? On 5/17/24 9:49 AM, Yang Fan wrote: Thanks Matthias, I still can't find "Assign to me" button beside Assignee and Reporter. Could you help me set it again? Best regards, Fan 发件人: Matthias J. Sax 发送时间: 2024年5月17

[jira] [Commented] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

2024-05-16 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847064#comment-17847064 ] Matthias J. Sax commented on KAFKA-16774: - Ah. Nice. Glad it's not a bug. Thanks for the PR

[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-16 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17847063#comment-17847063 ] Matthias J. Sax commented on KAFKA-15242: - The example ([https://github.com/zveljkovic/kafka

Re: [DISCUSS] KIP-989: RocksDB Iterator Metrics

2024-05-16 Thread Matthias J. Sax
of a more obvious metric name. Nit: the KIP says it's a store-level metric, but I think it would be good to say explicitly that it's recorded with DEBUG level only? Yes, I've already updated the KIP with this information in the table. Regards, Nick On Sun, 31 Mar 2024 at 10:53, Matthias J. S

Re: Kafka streams stores key in multiple state store instances

2024-05-16 Thread Matthias J. Sax
Hello Kay, What you describe is "by design" -- unfortunately. The problem is, that when we build the `Topology` we don't know the partition count of the input topics, and thus, StreamsBuilder cannot insert a repartition topic for this case (we always assume that the partition count is the

Re: Request to be added to kafka contributors list

2024-05-16 Thread Matthias J. Sax
Thanks for reaching out Yang. You should be all set. -Matthias On 5/16/24 7:40 AM, Yang Fan wrote: Dear Apache Kafka Team, I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I kindly request to be added to the contributors list for Apache Kafka. Being part of this

[jira] [Assigned] (KAFKA-16448) Add Kafka Streams exception handler for exceptions occuring during processing (KIP-1033)

2024-05-16 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-16448: --- Assignee: Loïc Greffier > Add Kafka Streams exception handler for excepti

[jira] [Commented] (KAFKA-16333) Removed Deprecated methods KTable#join

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846821#comment-17846821 ] Matthias J. Sax commented on KAFKA-16333: - In general yes. But we should wait until a final

[jira] [Commented] (KAFKA-16329) Remove Deprecated Task/ThreadMetadata classes and related methods

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846820#comment-17846820 ] Matthias J. Sax commented on KAFKA-16329: - In general yes. But we should wait until a final

[jira] [Commented] (KAFKA-16330) Remove Deprecated methods/variables from TaskId

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846819#comment-17846819 ] Matthias J. Sax commented on KAFKA-16330: - In general yes. But we should wait until a final

[jira] [Commented] (KAFKA-16448) Add Kafka Streams exception handler for exceptions occuring during processing (KIP-1033)

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846818#comment-17846818 ] Matthias J. Sax commented on KAFKA-16448: - [~muralibasani] – Thanks for you interest

[jira] [Commented] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846815#comment-17846815 ] Matthias J. Sax commented on KAFKA-16774: - Wondering if this is actually a flaky test

[jira] [Updated] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16774: Labels: flaky-test (was: ) > fix flaky StreamThreadT

[jira] [Updated] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16774: Component/s: streams unit tests > fix flaky StreamThreadT

[jira] [Comment Edited] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846814#comment-17846814 ] Matthias J. Sax edited comment on KAFKA-15242 at 5/16/24 5:32 AM

[jira] [Commented] (KAFKA-15242) FixedKeyProcessor testing is unusable

2024-05-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846814#comment-17846814 ] Matthias J. Sax commented on KAFKA-15242: - There is https://issues.apache.org/jira/browse/KAFKA

Re: Query regarding groupbykey in streams

2024-05-15 Thread Matthias J. Sax
If I read this correctly, your upstream producer which writes into the input topic of you KS app is using a custom partitioner? If you do a `groupByKey()` and change the key upstream, it would result in a repartition step, which would fall back to the default partioner. If you want to use a

Re: [VOTE] KIP-989: RocksDB Iterator Metrics

2024-05-14 Thread Matthias J. Sax
+1 (binding) On 5/14/24 9:19 AM, Lucas Brutschy wrote: Hi Nick! Thanks for the KIP. +1 (binding) On Tue, May 14, 2024 at 5:16 PM Nick Telford wrote: Hi everyone, I'd like to call a vote on the Kafka Streams KIP-989: RocksDB Iterator Metrics:

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

2024-05-13 Thread Matthias J. Sax
+1 (binding) On 5/13/24 5:54 PM, Sophie Blee-Goldman wrote: Thanks for the KIP guys! +1 (binding) On Mon, May 13, 2024 at 6:02 AM Bill Bejeck wrote: Thanks for the KIP, this will be a great addition! +1(binding) -Bill On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna wrote: Hi Damien,

[jira] [Commented] (KAFKA-16584) Make log processing summary configurable or debug

2024-05-09 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845111#comment-17845111 ] Matthias J. Sax commented on KAFKA-16584: - I am on the PMC and can help you. :)  After you wiki

Re: Creating kafka wiki id

2024-05-09 Thread Matthias J. Sax
Self-service to create an account is currently not working. Please reply on https://issues.apache.org/jira/browse/INFRA-25451 to request a wiki account. I'll update the wiki page for now until the issue is resolved. -Matthias On 5/7/24 8:25 AM, 黃竣陽 wrote: Hello, I want to create a KIP, but

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-05-09 Thread Matthias J. Sax
ProductionExceptionHandler... Just explaining my reasoning -- in the end I leave it up to the KIP authors and anyone who will actually be using this feature in their applications :) On Tue, May 7, 2024 at 8:35 PM Matthias J. Sax wrote: @Loic, yes, what you describe is exactly what I had i

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-07 Thread Matthias J. Sax
rRecord as the input parameter of the handle() method as well. - increased the response types to 3 to have fail and two types of continue. - The default behaviour is having no custom handler, having the corresponding config parameter set to null. Therefore, the KIP provides no default

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-05-07 Thread Matthias J. Sax
@Loic, yes, what you describe is exactly what I had in mind. @Sophie, can you elaborate a little bit? First of all, I agree that it makes sense to maintain the two separate callbacks for the ProductionExceptionHandler, since one of them is specifically for serialization exceptions while the

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

2024-05-03 Thread Matthias J. Sax
On 5/3/24 6:29 PM, Matthias J. Sax wrote: 101: Yes, but what I am saying is, that we don't need to flush the .position file to disk periodically, but only maintain it in main memory, and only write it to disk on close() to preserve it across restarts. This way, it would never be ahead

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

2024-05-03 Thread Matthias J. Sax
-3910-4c25-bfad-ea2b98953db3-StreamThread-9 Message: [Consumer clientId=kafka-streams-exec-0-test-store-6d676cf0-3910-4c25-bfad-ea2b98953db3-StreamThread-9-consumer, groupId=kafka-streams-exec-0-test-store ] Node 102 disconnected. On Mon, Apr 22, 2024 at 7:16 AM Matthias J. Sax wrote

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024-05-03 Thread Matthias J. Sax
, May 1, 2024 at 11:00 PM Matthias J. Sax wrote: Any updates on this KIP? On 3/28/24 4:11 AM, Matthias J. Sax wrote: It seems that `MockRecordMetadata` is a private class, and thus not part of the public API. If there are any changes required, we don't need to discuss on the KIP

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-05-03 Thread Matthias J. Sax
:43 AM, Matthias J. Sax wrote: Thanks Sophie. My bad. You are of course right about `TaskAssignment` and the StreamsPartitionAssignor's responsibitliy to map tasks of a instance to consumers. When I wrote my reply, I forgot about this detail. Seems you did not add `UNKNOWN_TASK_ID` error yet as pr

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

2024-05-03 Thread Matthias J. Sax
eams/state/internals/RocksDBStore.java#L397 On 5/1/24 3:00 AM, Matthias J. Sax wrote: Thanks Bruno. 101: I think I understand this better now. But just want to make sure I do. What do you mean by "they can diverge" and "Recovering after a failure might load inconsistent offse

Re: [VOTE] KIP-1036: Extend RecordDeserializationException exception

2024-05-03 Thread Matthias J. Sax
+1 (binding) On 5/3/24 8:52 AM, Federico Valeri wrote: Hi Fred, this is a useful addition. +1 non binding Thanks On Fri, May 3, 2024 at 4:11 PM Andrew Schofield wrote: Hi Fred, Thanks for the KIP. It’s turned out nice and elegant I think. Definitely a worthwhile improvement. +1

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-05-03 Thread Matthias J. Sax
What about (106) to unify both exiting callback methods of `ProductionExceptionHandler` into a single one, instead of adding two new ones? Damien's last reply about it was: I will think about unifying, I do agree it would be cleaner. There was not follow up on this question, and the KIP

Re: [VOTE] KIP-924: customizable task assignment for Streams

2024-05-02 Thread Matthias J. Sax
I left one more nit on the discuss thread. But overall LGTM. +1 (binding) Thanks Rohan and Sophie for driving this KIP. -Matthias On 4/29/24 2:07 PM, Sophie Blee-Goldman wrote: +1 (binding) thanks for driving this KIP! On Tue, Apr 16, 2024 at 1:46 PM Rohan Desai wrote:

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-05-02 Thread Matthias J. Sax
have a lower level indention? It caught me for a sec until I realized there are just two categories. * In TaskAssignmentUtils , why not let those util functions return `TaskAssignment` (to me it feels more consistent with the user APIs), but instead return a Map? Guozhang On Tue, Apr 30, 2024

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Component/s: clients > Deprecate window.size.ms and window.inner.serde.cl

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Description: {{window.size.ms}} and {{window.inner.serde.class}} are not a true

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Description: {{indwindow.size.ms}} and `is not a true KafkaStreams config, and results

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Description: {{window.size.ms}} and `is not a true KafkaStreams config, and results

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and window.inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Summary: Deprecate window.size.ms and window.inner.serde.class in StreamsConfig

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms and inner.serde.class in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Summary: Deprecate window.size.ms and inner.serde.class in StreamsConfig (was: Deprecate

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Description: {{window.size.ms}}  is not a true KafkaStreams config, and results

[jira] [Updated] (KAFKA-16260) Deprecate window.size.ms in StreamsConfig

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16260: Labels: KIP (was: needs-kip) > Deprecate window.size.ms in StreamsCon

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024-05-01 Thread Matthias J. Sax
Any updates on this KIP? On 3/28/24 4:11 AM, Matthias J. Sax wrote: It seems that `MockRecordMetadata` is a private class, and thus not part of the public API. If there are any changes required, we don't need to discuss on the KIP. For `CapturedPunctuator` and `CapturedForward` it's

Re: outerjoin not joining after window

2024-05-01 Thread Matthias J. Sax
oes process one sided joins after the skipped record. Do you have any docs on the "dropper records" metric? I did a Google search and didn't find many good results for that. Thanks, Chad On Tue, Apr 30, 2024 at 2:49 PM Matthias J. Sax wrote: Thanks for the information. I ran the cod

[jira] [Commented] (KAFKA-16514) Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.

2024-05-01 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842805#comment-17842805 ] Matthias J. Sax commented on KAFKA-16514: - Thanks for the background! Makes sense. > Ka

[jira] [Resolved] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16644. - Resolution: Duplicate > FK join emits duplicate tombstone on left-side del

[jira] [Resolved] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-16644. - Resolution: Duplicate > FK join emits duplicate tombstone on left-side del

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-04-30 Thread Matthias J. Sax
Thanks for the update. I am wondering if we should use `ReadOnlyHeaders` instead of `ImmutableHeaders` as interface name? Also, the returned `Header` interface is technically not immutable either, because `Header#key()` returns a mutable byte-array... Would we need a `ReadOnlyHeader`

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

2024-04-30 Thread Matthias J. Sax
that we should try to support downgrades without wipes. At least Nick should state in the KIP why we do not support it. Best, Bruno On 4/23/24 8:13 AM, Matthias J. Sax wrote: Thanks for splitting out this KIP. The discussion shows, that it is a complex beast by itself, so worth to discuss by

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-04-30 Thread Matthias J. Sax
I like the idea of error codes. Not sure if the name are ideal? UNKNOWN_PROCESS_ID makes sense, but the other two seems a little bit difficult to understand? Should we be very descriptive (and also try to avoid coupling it to the threading model -- important for the first error code): -

[jira] [Commented] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842514#comment-17842514 ] Matthias J. Sax commented on KAFKA-16644: - Sorry. Wrong link. Fixed -> https://issues.apache.

[jira] [Updated] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16644: Description: We introduced a regression bug in 3.7.0 release via KAFKA-14748. When

Re: outerjoin not joining after window

2024-04-30 Thread Matthias J. Sax
n on to tell me what is going on? Basically, I'm looking for some pointers on where I can start looking. Thanks, Chad On Tue, Apr 30, 2024 at 10:26 AM Matthias J. Sax wrote: I expect the join to execute after the 25 with one side of the join containing a record and the other being null Give

Re: outerjoin not joining after window

2024-04-30 Thread Matthias J. Sax
I expect the join to execute after the 25 with one side of the join containing a record and the other being null Given that you also have a grace period of 5 minutes, the result will only be emitted after the grace-period passed and the window is closed (not when window end time is reached).

[jira] [Commented] (KAFKA-16382) Kafka Streams drop NULL values after reset

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842423#comment-17842423 ] Matthias J. Sax commented on KAFKA-16382: - Not yet from our side... Working on other things atm

[jira] [Commented] (KAFKA-16645) CVEs in 3.7.0 docker image

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842419#comment-17842419 ] Matthias J. Sax commented on KAFKA-16645: - I believe fixing these CVEs should be a blocker

[jira] [Updated] (KAFKA-16645) CVEs in 3.7.0 docker image

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16645: Priority: Blocker (was: Major) > CVEs in 3.7.0 docker im

[jira] [Updated] (KAFKA-16645) CVEs in 3.7.0 docker image

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16645: Fix Version/s: 3.8.0 3.7.1 > CVEs in 3.7.0 docker im

[jira] [Commented] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842417#comment-17842417 ] Matthias J. Sax commented on KAFKA-16644: - Thanks for the pointer. I did not have time yet

[jira] [Commented] (KAFKA-16514) Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.

2024-04-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842229#comment-17842229 ] Matthias J. Sax commented on KAFKA-16514: - Cool. You can find details on the wiki: [https

[jira] [Updated] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16644: Description: We introduced a regression bug in 3.7.0 release via KAFKA-14778. When

[jira] [Updated] (KAFKA-16644) FK join emits duplicate tombstone on left-side delete

2024-04-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16644: Summary: FK join emits duplicate tombstone on left-side delete (was: FK join emit

[jira] [Created] (KAFKA-16644) FK join emit duplicate tombstone on left-side delete

2024-04-29 Thread Matthias J. Sax (Jira)
Matthias J. Sax created KAFKA-16644: --- Summary: FK join emit duplicate tombstone on left-side delete Key: KAFKA-16644 URL: https://issues.apache.org/jira/browse/KAFKA-16644 Project: Kafka

[jira] [Created] (KAFKA-16644) FK join emit duplicate tombstone on left-side delete

2024-04-29 Thread Matthias J. Sax (Jira)
Matthias J. Sax created KAFKA-16644: --- Summary: FK join emit duplicate tombstone on left-side delete Key: KAFKA-16644 URL: https://issues.apache.org/jira/browse/KAFKA-16644 Project: Kafka

[jira] [Commented] (KAFKA-16514) Kafka Streams: stream.close(CloseOptions) does not respect options.leaveGroup flag.

2024-04-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842183#comment-17842183 ] Matthias J. Sax commented on KAFKA-16514: - I would not call it a bug, but more like a feature

Wiki account sign-up

2024-04-29 Thread Matthias J. Sax
Hi, as many of you know, there is a issue with creating new wiki accounts right now: https://issues.apache.org/jira/browse/INFRA-25451 A fix might take some more time. In the meantime, please reply to the INFRA ticket with your email address, and accounts will be created manually in the

Re: Assistance Needed with Creating Wiki ID for Kafka Contribution

2024-04-29 Thread Matthias J. Sax
It's a known issue and INFRA is working on a solution: https://issues.apache.org/jira/browse/INFRA-25451 In the mean time, users can be added manually (cf the ticket, and reply there to get added). -Matthias On 3/28/24 5:10 AM, Prashant Jagtap wrote: Hi, I hope this email finds you well.

Re: How to find out the end of the session window

2024-04-29 Thread Matthias J. Sax
Did you look into .windowedBy(...).emitStrategy(...) ? Using emit-final you would get an downstream even only after the window closed. -Matthias On 4/29/24 1:43 AM, Santhoshi Mekala wrote: Hi Team, We have the below requirement: We are processing batch logs in kstreams. Currently, we are

[jira] [Commented] (KAFKA-16584) Make log processing summary configurable or debug

2024-04-26 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841332#comment-17841332 ] Matthias J. Sax commented on KAFKA-16584: - There is an issue with creating new account: https

Re: Confluence edit access

2024-04-26 Thread Matthias J. Sax
Thanks. You should be all set. -Matthias On 4/25/24 10:49 PM, Claude Warren wrote: My Confluence ID is "claude" On Thu, Apr 25, 2024 at 8:40 PM Matthias J. Sax wrote: What's your wiki ID? We can grant write access on our side if you have already an account. -Matthias On 4/25/

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-04-25 Thread Matthias J. Sax
believe that we shouldn't call certain kinds of assignments "invalid" (such as a task not being assigned to anyone). For the clearly-invalid assignment cases, I'd err on the side of not holding users hands too much for now, but again: would be happy to defer if anyone has another suggestion and/or s

Re: Confluence edit access

2024-04-25 Thread Matthias J. Sax
What's your wiki ID? We can grant write access on our side if you have already an account. -Matthias On 4/25/24 4:06 AM, Claude Warren wrote: I would like to get edit access to the Kafka confluence so that I can work on KIP-936. Can someone here do that or do I need to go through Infra?

[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor

2024-04-25 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840907#comment-17840907 ] Matthias J. Sax commented on KAFKA-16585: - {quote}I can use the regular Processor, but as I

Re: [DISCUSS] KIP-924: customizable task assignment for Streams

2024-04-24 Thread Matthias J. Sax
aying -- but I'm now realizing that I completely misinterpreted what your concern was. Sorry for the long-winded and ultimately irrelevant answer. I'm completely fine with having the return type be a simple Set with additional info such as TaskId in the AssignedTask class (and I see Rohan already made this

[jira] [Commented] (KAFKA-16584) Make log processing summary configurable or debug

2024-04-24 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840639#comment-17840639 ] Matthias J. Sax commented on KAFKA-16584: - Yes, it does required to write code. – It does also

Re: [ANNOUNCE] New committer: Igor Soarez

2024-04-24 Thread Matthias J. Sax
Congrats! On 4/24/24 2:29 PM, Bill Bejeck wrote: Congrats Igor! -Bill On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley wrote: Congratulations Igor! On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai wrote: Congratulations, Igor! you are one of the best Kafka developers!!! Mickael Maison 於

Re: [ANNOUNCE] New committer: Igor Soarez

2024-04-24 Thread Matthias J. Sax
Congrats! On 4/24/24 2:29 PM, Bill Bejeck wrote: Congrats Igor! -Bill On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley wrote: Congratulations Igor! On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai wrote: Congratulations, Igor! you are one of the best Kafka developers!!! Mickael Maison 於

[jira] [Commented] (KAFKA-16584) Make log processing summary configurable or debug

2024-04-23 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840232#comment-17840232 ] Matthias J. Sax commented on KAFKA-16584: - I would prefer to make if configurable personally

[jira] [Updated] (KAFKA-16584) Make log processing summary configurable or debug

2024-04-23 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-16584: Labels: needs-kip newbie (was: ) > Make log processing summary configurable or de

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-04-23 Thread Matthias J. Sax
et to null. Therefore, the KIP provides no default implementation of the interface. - We follow the interface solution as described in the Rejected Alternetives section. Cheers, Alieh On Thu, Apr 18, 2024 at 8:11 PM Matthias J. Sax wrote: Thanks for the KIP Alieh! It addresses an important case

Re: [DISCUSS] KIP-1035: StateStore managed changelog offsets

2024-04-23 Thread Matthias J. Sax
Thanks for splitting out this KIP. The discussion shows, that it is a complex beast by itself, so worth to discuss by its own. Couple of question / comment: 100 `StateStore#commit()`: The JavaDoc says "must not be called by users" -- I would propose to put a guard in place for this, by

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

2024-04-22 Thread Matthias J. Sax
gaps in a separate KIP. Best, Bruno On 4/11/24 12:06 AM, Matthias J. Sax wrote: Thanks for the KIP. Great discussion. I am not sure if I understand the proposal from Bruno to hand in the processor node id? Isn't this internal (could not even find it quickly). We do have a processor name, r

[jira] [Commented] (KAFKA-16585) No way to forward message from punctuation method in the FixedKeyProcessor

2024-04-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839910#comment-17839910 ] Matthias J. Sax commented on KAFKA-16585: - Well, the use-case make sense, but the question

[jira] [Commented] (KAFKA-16567) Add New Stream Metrics based on KIP-869

2024-04-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839907#comment-17839907 ] Matthias J. Sax commented on KAFKA-16567: - Thanks Bruno – makes sense to me – should we move

  1   2   3   4   5   6   7   8   9   10   >