[jira] [Updated] (KAFKA-14415) ThreadCache is getting slower with every additional state store

2022-11-22 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14415: Component/s: streams > ThreadCache is getting slower with every additional state st

Re: [DISCUSS] KIP-890 Server Side Defense

2022-11-21 Thread Matthias J. Sax
Thanks for the KIP. Couple of clarification questions (I am not a broker expert do maybe some question are obvious for others, but not for me with my lack of broker knowledge). (10) The delayed message case can also violate EOS if the delayed message comes in after the next

Re: [DISCUSS] KIP-889 Versioned State Stores

2022-11-21 Thread Matthias J. Sax
Thanks for the KIP Victoria. Very well written! Couple of questions (many might just require to add some more details to the KIP): (1) Why does the new store not extend KeyValueStore, but StateStore? In the end, it's a KeyValueStore? (2) Should we have a ReadOnlyVersionedKeyValueStore?

[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value

2022-11-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636907#comment-17636907 ] Matthias J. Sax edited comment on KAFKA-14400 at 11/21/22 10:40 PM

[jira] [Comment Edited] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value

2022-11-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636907#comment-17636907 ] Matthias J. Sax edited comment on KAFKA-14400 at 11/21/22 10:40 PM

[jira] [Commented] (KAFKA-14400) KStream - KStream - LeftJoin() does not call ValueJoiner with null value

2022-11-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636907#comment-17636907 ] Matthias J. Sax commented on KAFKA-14400: - The old and new API have different semantics. The old

Re: [VOTE] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

2022-11-21 Thread Matthias J. Sax
+1 (binding) On 11/21/22 7:39 AM, John Roesler wrote: I'm +1 (binding) Thanks for the KIP! -John On 2022/11/17 21:06:29 Hao Li wrote: Hi all, I would like start a vote on KIP-884:

[jira] [Commented] (KAFKA-14184) Kafka streams application crashes due to "UnsupportedOperationException: this should not happen: timestamp() is not supported in standby tasks."

2022-11-17 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635680#comment-17635680 ] Matthias J. Sax commented on KAFKA-14184: - The stack trace is highly suspicious. Based

[jira] [Commented] (KAFKA-14374) Kafka streams losing messages in State Store during first launch of app

2022-11-17 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17635667#comment-17635667 ] Matthias J. Sax commented on KAFKA-14374: - Thanks for creating a ticket. I am not sure right now

Re: [DISCUSS] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

2022-11-17 Thread Matthias J. Sax
it the `DefaultKafkaClientSupplier` Thanks, Hao On Tue, Nov 15, 2022 at 4:11 PM Matthias J. Sax wrote: Thanks for the KIP Hao. What is the behavior if users set the config and also pass in a client supplier into the constructor? Following other config/API patterns we use, it seems the best thing

Re: [DISCUSS] KIP-884: Add config to configure KafkaClientSupplier in Kafka Streams

2022-11-15 Thread Matthias J. Sax
Thanks for the KIP Hao. What is the behavior if users set the config and also pass in a client supplier into the constructor? Following other config/API patterns we use, it seems the best thing would be if the code would overwrite the config? If we do this, should we change the config name

[jira] [Resolved] (KAFKA-14360) Documentation: Streams Security page has broken links

2022-11-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-14360. - Resolution: Fixed > Documentation: Streams Security page has broken li

[jira] [Resolved] (KAFKA-14360) Documentation: Streams Security page has broken links

2022-11-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-14360. - Resolution: Fixed > Documentation: Streams Security page has broken li

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Matthias J. Sax
don't think that there is any guarantee that you might "see" concurrent modification (IIRC, RocksDB uses snapshot isolation for iterators). But maybe that's good enough for you? -Matthias On 11/7/22 11:13 AM, Joshua Suskalo wrote: "Matthias J. Sax" writes: In ge

Re: [DISCUSS] KIP-878: Autoscaling for Statically Partitioned Streams

2022-11-07 Thread Matthias J. Sax
Thanks for the KIP Sophie. Seems there is a lively discussion going on. I tried to read up on the history and I hope I don't repeat what was already discussed. And sorry for the quite long email... (1) Stateless vs Stateful I agree that stateless apps should be supported, even if I am not

[jira] [Commented] (KAFKA-14294) Kafka Streams should commit transaction when no records are processed

2022-11-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630098#comment-17630098 ] Matthias J. Sax commented on KAFKA-14294: - Why do you think we don't commit? Based on the code

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-07 Thread Matthias J. Sax
used iterator is concurrent, there is no API contract about it. -Matthias On 11/7/22 7:41 AM, Joshua Suskalo wrote: Hello Matthias, thanks for the response! "Matthias J. Sax" writes: Spanning your own thread and calling context.forward() is _not_ safe, and there is currently no w

Re: Thread Safety of Punctuator Functions and Processor Contexts

2022-11-04 Thread Matthias J. Sax
Your observation is correct. The Processor#process() and punctuation callback are executed on a single thread. It's by design to avoid the issue of concurrency (writing thread safe code is hard and we want to avoid putting this burden onto the user). There is currently no plans to make

[jira] [Commented] (KAFKA-13813) left/outer joins can wait indefinitely for emitted record with spurious record fix

2022-11-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17629223#comment-17629223 ] Matthias J. Sax commented on KAFKA-13813: - Sorry for late reply – I was on leave. {quote

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-02 Thread Matthias J. Sax
Congrats! On 11/1/22 7:08 PM, Luke Chen wrote: Congrats Bruno! Well deserved! Luke On Wed, Nov 2, 2022 at 10:07 AM John Roesler wrote: Congratulations, Bruno!!! On Tue, Nov 1, 2022, at 15:16, Lucas Brutschy wrote: Wow, congratulations! On Tue, Nov 1, 2022 at 8:55 PM Chris Egerton

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-02 Thread Matthias J. Sax
Congrats! On 11/1/22 7:08 PM, Luke Chen wrote: Congrats Bruno! Well deserved! Luke On Wed, Nov 2, 2022 at 10:07 AM John Roesler wrote: Congratulations, Bruno!!! On Tue, Nov 1, 2022, at 15:16, Lucas Brutschy wrote: Wow, congratulations! On Tue, Nov 1, 2022 at 8:55 PM Chris Egerton

[jira] [Updated] (KAFKA-14090) Allow Kafka Streams to be configured to not create internal topics

2022-07-25 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14090: Component/s: streams (was: clients) > Allow Kafka Stre

Re: [ANNOUNCE] New Committer: Chris Egerton

2022-07-25 Thread Matthias J. Sax
Congrats! Well deserved! -Matthias On 7/25/22 1:08 PM, Bill Bejeck wrote: Congrats Chris! -Bill On Mon, Jul 25, 2022 at 3:58 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: Congratulations Chris! On Mon, 25 Jul 2022 at 20:27, Robin Moffatt wrote: Congrats Chris! --

[jira] [Updated] (KAFKA-14076) Fix issues with KafkaStreams.CloseOptions

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14076: Affects Version/s: 3.3.0 > Fix issues with KafkaStreams.CloseOpti

[jira] [Updated] (KAFKA-13217) Reconsider skipping the LeaveGroup on close() or add an overload that does so

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13217: Labels: kip newbie newbie++ (was: needs-kip newbie newbie++) > Reconsider skipp

[jira] [Updated] (KAFKA-13217) Reconsider skipping the LeaveGroup on close() or add an overload that does so

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13217: Description: In Kafka Streams, when an instance is shut down via the close() API, we

[jira] [Commented] (KAFKA-13217) Reconsider skipping the LeaveGroup on close() or add an overload that does so

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569283#comment-17569283 ] Matthias J. Sax commented on KAFKA-13217: - Just reviewed the PR and linked the tickets. PR LGTM

[jira] [Commented] (KAFKA-14076) Fix issues with KafkaStreams.CloseOptions

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569282#comment-17569282 ] Matthias J. Sax commented on KAFKA-14076: - Marking this as blocker for 3.3, because this fixes

[jira] [Updated] (KAFKA-14076) Fix issues with KafkaStreams.CloseOptions

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14076: Priority: Blocker (was: Major) > Fix issues with KafkaStreams.CloseOpti

[jira] [Updated] (KAFKA-14076) Fix issues with KafkaStreams.CloseOptions

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-14076: Fix Version/s: 3.3.0 > Fix issues with KafkaStreams.CloseOpti

[jira] [Assigned] (KAFKA-14076) Fix issues with KafkaStreams.CloseOptions

2022-07-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-14076: --- Assignee: Jim Hughes > Fix issues with KafkaStreams.CloseOpti

[jira] [Commented] (KAFKA-12887) Do not trigger user-customized ExceptionalHandler for RTE

2022-07-19 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568659#comment-17568659 ] Matthias J. Sax commented on KAFKA-12887: - This feature broke some stuff, and we revert

[jira] [Updated] (KAFKA-13152) Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2022-07-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13152: Labels: kip (was: needs-kip) > Replace "buffered.records.per.p

[jira] [Reopened] (KAFKA-13152) Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2022-07-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-13152: - > Replace "buffered.records.per.partition" with "inpu

[jira] [Updated] (KAFKA-13152) Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2022-07-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13152: Fix Version/s: (was: 3.3.0) > Replace "buffered.records.per.p

[jira] [Reopened] (KAFKA-13152) Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2022-07-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-13152: - > Replace "buffered.records.per.partition" with "inpu

Re: [DISCUSS] KIP-837 Allow MultiCasting a Result Record.

2022-07-07 Thread Matthias J. Sax
class devoid of any StreamPartitioner. I just thought it's cleaner to keep it this way as StreamPartitioner already dpes the partitioning. Let me know what you think. Thanks! Sagar. On Wed, Jun 29, 2022 at 5:34 AM Matthias J. Sax wrote: Thanks for the KIP. Overall a good addition. I am actually

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

2022-07-07 Thread Matthias J. Sax
Thanks for the KIP. I read it and I am also worried by the complexity of the new configurations. They are not easy to grasp. I need to digest it a bit more, I think. Best, David Le mer. 29 juin 2022 à 02:25, Matthias J. Sax a écrit : Thanks for the KIP. I don't think I fully digested the proposa

[jira] [Commented] (KAFKA-14049) Relax Non Null Requirement for KStreamGlobalKTable Left Join

2022-07-06 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563510#comment-17563510 ] Matthias J. Sax commented on KAFKA-14049: - {quote}Null Values in the Stream for a Left Join

[jira] (KAFKA-14049) Relax Non Null Requirement for KStreamGlobalKTable Left Join

2022-07-06 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14049 ] Matthias J. Sax deleted comment on KAFKA-14049: - was (Author: mjsax): Sounds like a duplicate to https://issues.apache.org/jira/browse/KAFKA-12317 ? > Relax Non Null Requirem

[jira] [Commented] (KAFKA-14049) Relax Non Null Requirement for KStreamGlobalKTable Left Join

2022-07-06 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563509#comment-17563509 ] Matthias J. Sax commented on KAFKA-14049: - Sounds like a duplicate to https://issues.apache.org

[jira] [Updated] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13936: Fix Version/s: 3.3.0 > Invalid consumer lag when monitoring from a kafka stre

[jira] [Assigned] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-13936: --- Assignee: Prashanth Joseph Babu > Invalid consumer lag when monitoring f

[jira] [Resolved] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13936. - Resolution: Fixed > Invalid consumer lag when monitoring from a kafka stre

[jira] [Resolved] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-30 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13936. - Resolution: Fixed > Invalid consumer lag when monitoring from a kafka stre

Re: [VOTE] KIP-825: introduce a new API to control when aggregated results are produced

2022-06-30 Thread Matthias J. Sax
On Thu, Mar 24, 2022 at 5:13 PM Matthias J. Sax wrote: +1 (binding) On 3/24/22 1:52 PM, Hao Li wrote: I hit reply on my phone in the mail app and changed the title and text hoping it will start a new thread. Apparently it doesn't work... On Thu, Mar 24, 2022 at 12:36 PM Bruno Cadonna wro

[jira] [Commented] (KAFKA-10493) KTable out-of-order updates are not being ignored

2022-06-29 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560622#comment-17560622 ] Matthias J. Sax commented on KAFKA-10493: - There is nothing official yet, but we started to work

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

2022-06-28 Thread Matthias J. Sax
Thanks for the KIP. I don't think I fully digested the proposal yet, but my first reaction is: this is quite complicated. Frankly, I am worried about complexity and usability. Especially the option `safe_latest` is a "weird" one IMHO, and `nearest` is even more complex. The problem at

Re: [DISCUSS] KIP-837 Allow MultiCasting a Result Record.

2022-06-28 Thread Matthias J. Sax
Thanks for the KIP. Overall a good addition. I am actually not sure if we need to add a new class? From my understanding, if there is exactly one abstract method, the interface is still functional? Thus, we could add a new method to `StreamsPartitioner` with a default implementation (that

[jira] [Commented] (KAFKA-13531) Flaky test NamedTopologyIntegrationTest

2022-06-17 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555781#comment-17555781 ] Matthias J. Sax commented on KAFKA-13531: - Could be. We usually also report if we say more

[jira] [Resolved] (KAFKA-13531) Flaky test NamedTopologyIntegrationTest

2022-06-17 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13531. - Resolution: Cannot Reproduce > Flaky test NamedTopologyIntegrationT

[jira] [Resolved] (KAFKA-13531) Flaky test NamedTopologyIntegrationTest

2022-06-17 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13531. - Resolution: Cannot Reproduce > Flaky test NamedTopologyIntegrationT

[jira] [Commented] (KAFKA-13939) Memory Leak When Logging Is Disabled In InMemoryTimeOrderedKeyValueBuffer

2022-06-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554742#comment-17554742 ] Matthias J. Sax commented on KAFKA-13939: - Thanks for the PR. I added you to list

[jira] [Assigned] (KAFKA-13939) Memory Leak When Logging Is Disabled In InMemoryTimeOrderedKeyValueBuffer

2022-06-15 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-13939: --- Assignee: Jackson Newhouse > Memory Leak When Logging Is Disab

Re: How it is safe to break message ordering but not idempotency after getting an OutOfOrderSequenceException?

2022-06-15 Thread Matthias J. Sax
es include a message being written to topic A, could messages from batch with sn X+1 end up being persisted with an offset lesser than the ones from the batch with sn X? Does this question make sense? El mar, 7 jun 2022 a las 16:13, Matthias J. Sax () escribió: Yes, the broker de-dupes using

[jira] [Commented] (SUREFIRE-2076) BufferOverflowException when encoding message with null runMode

2022-06-14 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/SUREFIRE-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554376#comment-17554376 ] Matthias J. Sax commented on SUREFIRE-2076: --- Thanks for the update! – I opened a PR for our

[jira] [Commented] (KAFKA-13963) Topology Description ignores context.forward

2022-06-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551333#comment-17551333 ] Matthias J. Sax commented on KAFKA-13963: - {quote}Is it worth updating the java doc to mention

[jira] [Commented] (KAFKA-13963) Topology Description ignores context.forward

2022-06-07 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17551271#comment-17551271 ] Matthias J. Sax commented on KAFKA-13963: - TopologyDescription only describes the structure

Re: How it is safe to break message ordering but not idempotency after getting an OutOfOrderSequenceException?

2022-06-07 Thread Matthias J. Sax
Yes, the broker de-dupes using the sequence number. But for example, if a sequence number is skipped, you could get this exception: the current batch of messages cannot be appended to the log, as one batch is missing, and the producer would need to re-send the previous/missing batch with

Re: Newbie how to get key/value pojo out of a stream?

2022-06-07 Thread Matthias J. Sax
`enable.auto.commit` is a Consumer config and does not apply to Kafka Stream. In Kafka Streams, you basically always have auto commit enabled, and you can control how frequently commits happen via `commit.interval.ms`. Also on `close()` Kafka Streams would commit offsets. -Matthias On

[jira] [Commented] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-03 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17546407#comment-17546407 ] Matthias J. Sax commented on KAFKA-13936: - The docs are in the same repository as the code

[jira] [Commented] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-06-02 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545686#comment-17545686 ] Matthias J. Sax commented on KAFKA-13936: - Might be worth to document for this case :) – Would

[jira] [Updated] (KAFKA-13945) Add task-level metrics to Streams for bytes/records Produced

2022-05-31 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13945: Description: KIP-846: [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId

[jira] [Commented] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-05-31 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544647#comment-17544647 ] Matthias J. Sax commented on KAFKA-13936: - As mentioned above, offsets are by default committed

[jira] [Commented] (KAFKA-13939) Memory Leak When Logging Is Disabled In InMemoryTimeOrderedKeyValueBuffer

2022-05-31 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544645#comment-17544645 ] Matthias J. Sax commented on KAFKA-13939: - Thanks for reporting this issue – sound rather severs

[jira] [Updated] (KAFKA-13939) Memory Leak When Logging Is Disabled In InMemoryTimeOrderedKeyValueBuffer

2022-05-31 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13939: Component/s: streams > Memory Leak When Logging Is Disab

[jira] [Updated] (KAFKA-13939) Memory Leak When Logging Is Disabled In InMemoryTimeOrderedKeyValueBuffer

2022-05-31 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13939: Priority: Blocker (was: Major) > Memory Leak When Logging Is Disab

[jira] [Commented] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

2022-05-25 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542359#comment-17542359 ] Matthias J. Sax commented on KAFKA-13936: - > we see in the kafka UI  There is no "

Re: kafka stream - sliding window - getting unexpected output

2022-05-20 Thread Matthias J. Sax
ane wrote: @Matthias J. Sax / All Have added below line : .suppress(Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded())) Here is the output : (for uuid (*2cbef750-325b-4a2f-ac39-b2c23fa0313f)*, expecting single output but that is not the case here. Which 1 is the final output from tho

Re: Are timestamps available for records stored in Kafka Streams state stores?

2022-05-20 Thread Matthias J. Sax
Added. Feel free to update wiki pages with smaller details like this directly. -Matthias On 5/19/22 2:27 PM, James Cheng wrote: Thanks Guozhang! Based on your comment, I searched through the repo and found the associated pull requests and JIRAs. It looks like most of the support was added

[jira] [Commented] (KAFKA-13817) Schedule nextTimeToEmit to system time every time instead of just once

2022-05-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540294#comment-17540294 ] Matthias J. Sax commented on KAFKA-13817: - Thanks for the PR! I put it into my review backlog

[jira] [Commented] (KAFKA-13913) Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

2022-05-19 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539718#comment-17539718 ] Matthias J. Sax commented on KAFKA-13913: - There was some discussion about this in the past (cf

[jira] [Updated] (KAFKA-13913) Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

2022-05-19 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13913: Labels: kip (was: ) > Provide builders for KafkaProducer/KafkaConsumer and KafkaStre

[jira] [Updated] (KAFKA-13913) Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

2022-05-19 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13913: Component/s: streams > Provide builders for KafkaProducer/KafkaConsumer and KafkaStre

Re: kafka stream - sliding window - getting unexpected output

2022-05-18 Thread Matthias J. Sax
Emitting intermediate result is by-design. If you don't want to get intermediate result, you can add `suppress()` after the aggregation and configure it to only "emit on window close". -Matthias On 5/17/22 3:20 AM, Shankar Mane wrote: Hi All, Our use case is to use sliding window. (for

[jira] [Commented] (KAFKA-6520) When a Kafka Stream can't communicate with the server, it's Status stays RUNNING

2022-05-16 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537860#comment-17537860 ] Matthias J. Sax commented on KAFKA-6520: It's still an open item – and it's complex (for details

[jira] [Commented] (KAFKA-13817) Schedule nextTimeToEmit to system time every time instead of just once

2022-05-13 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536926#comment-17536926 ] Matthias J. Sax commented on KAFKA-13817: - > as it will eventually self-correct and conti

[jira] [Commented] (SUREFIRE-2076) BufferOverflowException when encoding message with null runMode

2022-05-11 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/SUREFIRE-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17535072#comment-17535072 ] Matthias J. Sax commented on SUREFIRE-2076: --- [~tibordigana] [~zoltan.meze] – Did you see my

[jira] [Resolved] (KAFKA-7527) Enable Dependency Injection for Kafka Streams handlers (KIP-378)

2022-05-10 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-7527. Resolution: Fixed > Enable Dependency Injection for Kafka Streams handlers (KIP-

[jira] [Resolved] (KAFKA-7527) Enable Dependency Injection for Kafka Streams handlers (KIP-378)

2022-05-10 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-7527. Resolution: Fixed > Enable Dependency Injection for Kafka Streams handlers (KIP-

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-10 Thread Matthias J. Sax
I had one minor question on the discuss thread. It's mainly about clarifying and document the user contract. I am fine either way. +1 (binding) -Matthias On 5/10/22 12:32 PM, Sophie Blee-Goldman wrote: Thanks for the KIP! +1 (binding) On Tue, May 10, 2022, 12:24 PM Bruno Cadonna wrote:

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-10 Thread Matthias J. Sax
d point about getting the paused state to new members; it is seeming like the "building block" approach is a good one to keep things simple at first. Cheers, Jim On Fri, May 6, 2022 at 8:31 PM Matthias J. Sax < mj...@apache.org wrote: I think it's tricky to propagate a pa

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-06 Thread Matthias J. Sax
I think it's tricky to propagate a pauseAll() via the rebalance protocol. New members joining the group would need to get paused, too? Could there be weird race conditions with overlapping pauseAll() and resumeAll() calls on different instanced while there could be a errors / network

[jira] [Commented] (KAFKA-8769) Consider computing stream time independently per key

2022-05-05 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532427#comment-17532427 ] Matthias J. Sax commented on KAFKA-8769: Just a "random" comment, based on some other

[jira] [Updated] (KAFKA-13873) Add ability to Pause / Resume KafkaStreams Topologies

2022-05-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13873: Description: In order to reduce resources used or modify data pipelines, users may want

[jira] [Updated] (KAFKA-13873) Add ability to Pause / Resume KafkaStreams Topologies

2022-05-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13873: Component/s: streams > Add ability to Pause / Resume KafkaStreams Topolog

[jira] [Updated] (KAFKA-13873) Add ability to Pause / Resume KafkaStreams Topologies

2022-05-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13873: Labels: kip (was: ) > Add ability to Pause / Resume KafkaStreams Topolog

[jira] [Updated] (KAFKA-13870) support both Suppressed untilTimeLimit and maxBytes without using emitEarlyWhenFull()

2022-05-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13870: Labels: needs-kip (was: ) > support both Suppressed untilTimeLimit and maxBytes with

[jira] [Commented] (KAFKA-13870) support both Suppressed untilTimeLimit and maxBytes without using emitEarlyWhenFull()

2022-05-04 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531836#comment-17531836 ] Matthias J. Sax commented on KAFKA-13870: - I think you mix up two concepts: windowing is about

[jira] [Updated] (KAFKA-13813) left/outer joins can wait indefinitely for emitted record with spurious record fix

2022-05-03 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13813: Summary: left/outer joins can wait indefinitely for emitted record with spurious record

[jira] [Created] (KAFKA-13866) Support more advanced time retention policies

2022-05-02 Thread Matthias J. Sax (Jira)
Matthias J. Sax created KAFKA-13866: --- Summary: Support more advanced time retention policies Key: KAFKA-13866 URL: https://issues.apache.org/jira/browse/KAFKA-13866 Project: Kafka Issue

[jira] [Created] (KAFKA-13866) Support more advanced time retention policies

2022-05-02 Thread Matthias J. Sax (Jira)
Matthias J. Sax created KAFKA-13866: --- Summary: Support more advanced time retention policies Key: KAFKA-13866 URL: https://issues.apache.org/jira/browse/KAFKA-13866 Project: Kafka Issue

[jira] [Commented] (SUREFIRE-2076) BufferOverflowException when encoding message with null runMode

2022-04-28 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/SUREFIRE-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529670#comment-17529670 ] Matthias J. Sax commented on SUREFIRE-2076: --- Seems we are hitting this issue trying

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

2022-04-27 Thread Matthias J. Sax
. Levani On 23. Apr 2022, at 02:43, Matthias J. Sax wrote: Ivan, are you still interested in this KIP? I think it would be a good addition. -Matthias On 8/16/21 5:30 PM, Matthias J. Sax wrote: Your point about the IQ problem is an interesting one. I missed the point that the "new key&q

[jira] [Commented] (KAFKA-13647) RocksDb metrics 'number-open-files' is not correct

2022-04-26 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528288#comment-17528288 ] Matthias J. Sax commented on KAFKA-13647: - Should we close this ticket? \cc [~guozhang

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

2022-04-22 Thread Matthias J. Sax
Ivan, are you still interested in this KIP? I think it would be a good addition. -Matthias On 8/16/21 5:30 PM, Matthias J. Sax wrote: Your point about the IQ problem is an interesting one. I missed the point that the "new key" would be a "superkey", and thus, it shoul

Re: [DISCUSS] KIP-819: Merge multiple KStreams in one operation

2022-04-22 Thread Matthias J. Sax
/javase/8/docs/api/java/util/Collection.html#toArray-T:A- Cheers, Chris On Tue, Mar 29, 2022 at 11:14 PM Matthias J. Sax wrote: My understand was, that the original proposal was to have: merge(KStream stream); merge(KStream... streams); Maybe I misunderstood. I am not really a fan

[jira] [Comment Edited] (KAFKA-12909) Allow users to opt-into spurious left/outer stream-stream join improvement

2022-04-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17522502#comment-17522502 ] Matthias J. Sax edited comment on KAFKA-12909 at 4/21/22 9:13 PM

[jira] [Commented] (KAFKA-12909) Allow users to opt-into spurious left/outer stream-stream join improvement

2022-04-21 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526086#comment-17526086 ] Matthias J. Sax commented on KAFKA-12909: - Ah sorry. That was my bad... Fixed. > Allow us

[jira] [Updated] (KAFKA-13842) Add per-aggregation step before repartitioning

2022-04-20 Thread Matthias J. Sax (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-13842: Description: Kafka Streams follows a continuous refinement model for aggregation

<    7   8   9   10   11   12   13   14   15   16   >