Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Currently we, don't cleanup dead threads, but the KIP proposes to change this: > Stream threads that are in state DEAD will be removed from the stream threads > of a Kafka Streams client. -Matthias On 9/8/20 2:37 PM, Sophie Blee-Goldman wrote: > Ah, I forgot about localThreadsMetadata(). In

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

+1 (binding) On 9/8/20 2:49 PM, Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas wrote: > >> Hi, >> >> I would like to call a vote on the following KIP. >> >> *KIP *- >> >>

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #49

See Changes: [github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader epoch 0 (#9219) -- [...truncated 6.57 MB...]

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

+1. Thanks! Guozhang On Tue, Sep 8, 2020 at 3:04 PM Ron Dagostino wrote: > +1 (non-binding) -- Thanks, Jason! > > Ron > > On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-664: > > > > >

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #49

See Changes: [github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader epoch 0 (#9219) -- [...truncated 6.57 MB...]

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Bill, Could you also add KIP-450 to the release plan? It's been merged. https://cwiki.apache.org/confluence/display/KAFKA/KIP-450%3A+Sliding+Window+Aggregations+in+the+DSL Cheers, Leah On Tue, Sep 8, 2020 at 9:32 AM Bill Bejeck wrote: > Hi Bruno, > > Thanks for letting me know, I've added

[jira] [Resolved] (KAFKA-5636) Add Sliding-Window support for Aggregations

[ https://issues.apache.org/jira/browse/KAFKA-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leah Thomas resolved KAFKA-5636. Resolution: Fixed > Add Sliding-Window support for Aggregations >

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #48

See Changes: [github] KAFKA-10432; LeaderEpochCache is incorrectly recovered for leader epoch 0 (#9219) -- [...truncated 3.26 MB...]

Re: [VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

+1 (non-binding) -- Thanks, Jason! Ron On Tue, Sep 8, 2020 at 2:04 PM Jason Gustafson wrote: > Hi All, > > I'd like to start a vote on KIP-664: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions > . > Thanks for all the

Re: [VOTE] KIP-654 Aborting Transaction with non-flushed data should throw a non-fatal Exception

+1 Thanks for the KIP! On Thu, Sep 3, 2020 at 12:25 PM Gokul Srinivas wrote: > Hi, > > I would like to call a vote on the following KIP. > > *KIP *- > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-654:+Aborted+transaction+with+non-flushed+data+should+throw+a+non-fatal+exception > < >

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Ah, I forgot about localThreadsMetadata(). In that. case I agree, there's no reason to introduce a new method when we can get both the names and number of all running threads from this. I assume that we would update localThreadsMetadata to only return currently alive threads as part of this KIP

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

From my understanding, the KIP aims for the case when a user does not control the code, eg, when using the command line consumer (or similar tools). If the user writes code, we should always encourage her to instantiate the deserializer explicitly and not relying on reflection+config. I also

[jira] [Created] (KAFKA-10471) TimeIndex handling may cause data loss in certain back to back failure

Rohit Shekhar created KAFKA-10471: - Summary: TimeIndex handling may cause data loss in certain back to back failure Key: KAFKA-10471 URL: https://issues.apache.org/jira/browse/KAFKA-10471 Project:

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

I am not sure if we need a new method? There is already `localThreadsMetadata()`. What do we gain by adding a new one? Returning the thread's name (as `Optional`) for both add() and remove() is fine with me. -Matthias On 9/8/20 12:58 PM, Sophie Blee-Goldman wrote: > Sorry Bruno, I think I

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Sorry Bruno, I think I missed the end of your message with the numberOfAliveStreamThreads() proposal. I agree, that would be better than the alternatives I listed. That said: > They rather suggest that the method returns a list of handles to the stream threads. I hadn't thought of that

[jira] [Resolved] (KAFKA-10432) LeaderEpochCache is incorrectly recovered on segment recovery for epoch 0

[ https://issues.apache.org/jira/browse/KAFKA-10432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-10432. - Fix Version/s: 2.6.1 Resolution: Fixed > LeaderEpochCache is incorrectly

[GitHub] [kafka-site] ableegoldman commented on a change in pull request #301: Redo 2.6 docs commit

ableegoldman commented on a change in pull request #301: URL: https://github.com/apache/kafka-site/pull/301#discussion_r485154723 ## File path: 26/streams/developer-guide/memory-mgmt.html ## @@ -202,7 +202,10 @@ RocksDB} } -1. INDEX_FILTER_BLOCK_RATIO can

[GitHub] [kafka-site] ableegoldman commented on a change in pull request #301: Redo 2.6 docs commit

ableegoldman commented on a change in pull request #301: URL: https://github.com/apache/kafka-site/pull/301#discussion_r485154723 ## File path: 26/streams/developer-guide/memory-mgmt.html ## @@ -202,7 +202,10 @@ RocksDB} } -1. INDEX_FILTER_BLOCK_RATIO can

[GitHub] [kafka-site] vvcephei commented on a change in pull request #301: Redo 2.6 docs commit

vvcephei commented on a change in pull request #301: URL: https://github.com/apache/kafka-site/pull/301#discussion_r485142442 ## File path: 26/streams/developer-guide/memory-mgmt.html ## @@ -202,7 +202,10 @@ RocksDB} } -1. INDEX_FILTER_BLOCK_RATIO can be

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

Thank you again for the feedback Sophie! As I tried to point out in my previous e-mail, removing a stream thread from a Kafka Streams client that does not have alive stream threads is nothing exceptional for the client per se. However, it can become exceptional within the context of the user.

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hi Sophie, Seems I do have some mis-understanding of the KIP's motivation here :) Just for clarification my reasoning is that: 1) today Streams itself never uses a windowed deserializer itself since its built-in operators only need the serializer and users do not need to override it, plus

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hey Sophie, Are you advocating that we change the name back to *window.size.ms *? I wasn't thinking that by including the config we would steer people away from using the constructors that pass in a window size, so in that sense I suppose the config isn't "default." I

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #48

See

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #47

See Changes: [github] MINOR: Update scala default version in readme (#9260) -- [...truncated 6.52 MB...] org.apache.kafka.streams.TopologyTestDriverTest >

Re: [DISCUSS] KIP-663: API to Start and Shut Down Stream Threads and to Request Closing of Kafka Streams Clients

> it's never a good sign when the discussion moves into the vote thread Hah, sorry, the gmail consolidation of [VOTE] and [DISCUSS] threads strikes again. Thanks for redirecting me Bruno I suppose it's unfair to expect the callers to keep perfect track of the current number of stream threads,

[VOTE] KIP-664: Provide tooling to detect and abort hanging transactions

Hi All, I'd like to start a vote on KIP-664: https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions. Thanks for all the feedback! Best, Jason

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hey Guozhang & Leah, I want to push back a bit on the assumption that we would fall back on this config in the case of an unspecified window size in a Streams serde. I don't think it should be a default at all, either in name or in effect. To borrow the rhetorical question that John raised

Re: [VOTE] KIP-667: Remove deprecated methods from ReadOnlyWindowStore

Thanks John! KIP is fixed now. On Tue, Sep 8, 2020 at 3:45 PM John Roesler wrote: > Hello Jorge, > > Thanks for the KIP! > > FYI, it looks like there's still a boilerplate paragraph in > the KIP document about it being a template. > > The KIP looks good to me; I'm +1 (binding) for doing this in

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Thanks Leah, I'm +1 on the KIP proposal. Guozhang On Tue, Sep 8, 2020 at 10:40 AM Leah Thomas wrote: > Hi Guozhang, > > Yes, the config would read them as a single window size. I think this > relates to John's comments about having variably sized windows, which this > config doesn't handle. I

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hi Guozhang, Yes, the config would read them as a single window size. I think this relates to John's comments about having variably sized windows, which this config doesn't handle. I like the name change and updated the wiki to reflect that, and to clarify that the default value will still be

[jira] [Created] (KAFKA-10470) zstd decompression with small batches is slow and causes excessive GC

Robert Wagner created KAFKA-10470: - Summary: zstd decompression with small batches is slow and causes excessive GC Key: KAFKA-10470 URL: https://issues.apache.org/jira/browse/KAFKA-10470 Project:

Dynamics 365 Inbound and Outbound integrations with Kafka topics

Hi there, I am doing some research using Kafka topics to support inbound and outbound messaging to D365. >From my research, it appears that you can: create an Azure Function and retrieve the function URL parse the the URL to obtain webhook URL Register new web hook in D365 plugin registration

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hello Leah, Thanks for initiating this. I just have one minor clarification question here: the config "window.size.ms" seems to be used as the default window size when reading from a topic that represents windowed records right? I.e. if there are multiple topics that represent windowed records

Re: [VOTE] KIP-659: Improve TimeWindowedDeserializer and TimeWindowedSerde to handle window size

Hey all, We should be good to wrap up voting now that the discussion has been resolved. Cheers, Leah On Wed, Sep 2, 2020 at 7:23 PM Matthias J. Sax wrote: > +1 (binding) > > On 8/26/20 8:02 AM, John Roesler wrote: > > Hi all, > > > > I've just sent a new message to the DISCUSS thread. We > >

[jira] [Created] (KAFKA-10469) describeConfigs() for broker loggers returns incorrect values

Tom Bentley created KAFKA-10469: --- Summary: describeConfigs() for broker loggers returns incorrect values Key: KAFKA-10469 URL: https://issues.apache.org/jira/browse/KAFKA-10469 Project: Kafka

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Hi Jason, Thank you again for all of your feedback, it is greatly appreciated. Here are some more changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615=21=17 > 1. I wonder if we need to bother with `enable.dynamic.config`, especially > if the

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

Hi Jason, Thank you again for all of your feedback, it is greatly appreciated. Here are some more changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615=21=17 > 1. I wonder if we need to bother with `enable.dynamic.config`, especially > if the

Re: [VOTE] KIP-667: Remove deprecated methods from ReadOnlyWindowStore

Hello Jorge, Thanks for the KIP! FYI, it looks like there's still a boilerplate paragraph in the KIP document about it being a template. The KIP looks good to me; I'm +1 (binding) for doing this in 3.0. Thanks, -John On Mon, 2020-09-07 at 12:18 +0100, Jorge Esteban Quilcate Otoya wrote: > Hi

Re: [DISCUSS] Apache Kafka 2.7.0 release

Hi Bruno, Thanks for letting me know, I've added KIP-662 to the release plan. -Bill On Mon, Sep 7, 2020 at 11:33 AM Bruno Cadonna wrote: > Hi Bill, > > Could you add KIP-662 [1] to the release plan. The KIP has been already > implemented. > > Best, > Bruno > > > [1] > >

[jira] [Created] (KAFKA-10468) Log4jController.getLoggers serialization

Tom Bentley created KAFKA-10468: --- Summary: Log4jController.getLoggers serialization Key: KAFKA-10468 URL: https://issues.apache.org/jira/browse/KAFKA-10468 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP idea: Support of multipart messages

LinkedIn had something like this. Becket did a talk on it a few years ago. It would be interesting to know what became of it and if there were lessons learned. https://www.youtube.com/watch?v=ZrbaXDYUZY8 On Fri, 4 Sep 2020 at 08:17, Alexander Sibiryakov < sibirya...@scrapinghub.com> wrote: >

Re: Apply for JIRA contributor authority

Hi, I already added you last week and replied to your thread. See https://lists.apache.org/thread.html/r723dfe2906bb52a364bc938726f3be654e50d13e3c6606b3c817c245%40%3Cdev.kafka.apache.org%3E On Tue, Sep 8, 2020 at 4:16 AM 季伟 wrote: > > Hi, > I want to contribute to Apache Kafka, > Would you

Re: Request to join Assignee list

Done. Thanks for your interest in Apache Kafka! On Tue, Sep 8, 2020 at 5:04 AM Yingshuan Song wrote: > > Hi, > > Could you please add me to the JIRA Assignee list. I would like to start > contributing. > > Jira username : ymxz > Full Name: songyingshuan > > Apologies in case I've sent this

[DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

Hi Everyone, I'd like to restart the discussion on this. Since the KIP has been revamped I thought I'd start a new discussion thread. Link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-567%3A+Kafka+Cluster+Audit Short summary: - Would like to introduce a new interface similar to the

Re: Backports of KAFKA-9839 onto 2.2, 2.3 and 2.4

Hi again, Would someone please have a look at these? Thanks, Viktor On Fri, Aug 28, 2020 at 10:11 AM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi folks, > > I have backported KAFKA-9839 > onto 2.2, 2.3 and 2.4 > respectively as