Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

Hello Boyang, Just realized one thing about timeout configurations that we should consider including in this KIP as well: 1) In Producer we have: max.block.ms (default value 60sec), request.timeout (30sec), delivery.timeout.ms (120sec), transaction.timeout (60sec) 2) In Consumer we have:

Jenkins build is back to normal : kafka-trunk-jdk8 #3889

See

Re: [VOTE] KIP-447: Producer scalability for exactly once semantics

Thanks Boyang, I'm +1 on the KIP. Could you also update the detailed design doc https://docs.google.com/document/d/1LhzHGeX7_Lay4xvrEXxfciuDWATjpUXQhrEIkph9qRE/edit which seems a bit out-dated with the latest proposal? Guozhang On Wed, Sep 4, 2019 at 10:45 AM Boyang Chen wrote: > Hey all, >

Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and TrustStore

Hi all It seems we are in some sort of agreement so far apart from code review/comments. However, I have a fundamental question - going forward how this work from the process standpoint? What do we do with this KIP-486 vs KIP-383? I feel that both needs to come together since in order to make

Jenkins build is back to normal : kafka-trunk-jdk11 #796

See

[jira] [Resolved] (KAFKA-4729) Stores for kstream-kstream join cannot be in-memory

[ https://issues.apache.org/jira/browse/KAFKA-4729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-4729. Resolution: Duplicate > Stores for kstream-kstream join cannot be in-memory >

[jira] [Created] (KAFKA-8880) Augment Consumer.committed(partition) to allow multiple partitions

Guozhang Wang created KAFKA-8880: Summary: Augment Consumer.committed(partition) to allow multiple partitions Key: KAFKA-8880 URL: https://issues.apache.org/jira/browse/KAFKA-8880 Project: Kafka

[jira] [Reopened] (KAFKA-1534) transient unit test failure in testBasicPreferredReplicaElection

[ https://issues.apache.org/jira/browse/KAFKA-1534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-1534: Assignee: (was: Abhishek Sharma) Failed again: 

[DISCUSS] 2.3.1 Bug Fix Release

Hey everyone, I'd like to volunteer for the Kafka 2.3.1 bug fix release. Kafka 2.3.0 was released last month on August 6 and a number of issues have been fixed since then including several critical and blocker bugs. Here is a complete list:

[jira] [Resolved] (KAFKA-8793) StickyTaskAssignor throws java.lang.ArithmeticException

[ https://issues.apache.org/jira/browse/KAFKA-8793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8793. -- Resolution: Not A Problem > StickyTaskAssignor throws java.lang.ArithmeticException >

Re: [VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

+1 (binding). On Thu, Sep 5, 2019 at 2:47 PM John Roesler wrote: > Hello, all, > > After a great discussion, I'd like to open voting on KIP-441, > to avoid long restore times in Streams after rebalancing. > Please cast your votes! > > >

[jira] [Resolved] (KAFKA-8879) GlobalStateUpdateTask uses wrong javaType to deserialize value

[ https://issues.apache.org/jira/browse/KAFKA-8879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vlad Olevsky resolved KAFKA-8879. - Resolution: Invalid > GlobalStateUpdateTask uses wrong javaType to deserialize value >

Build failed in Jenkins: kafka-2.3-jdk8 #101

See Changes: [jason] KAFKA-8724; Improve range checking when computing cleanable partitions -- [...truncated 2.96 MB...] kafka.zk.KafkaZkClientTest >

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Thanks, all. FWIW, the most recent formulation from Guozhang + Bruno's addendum would have my support. Thanks, -John On Thu, Sep 5, 2019 at 4:05 PM Bruno Cadonna wrote: > Hi Guozhang, > > Your summary corresponds to my proposal. > > A new value would only be added if in future we change the

[VOTE] KIP-441: Smooth Scaling Out for Kafka Streams

Hello, all, After a great discussion, I'd like to open voting on KIP-441, to avoid long restore times in Streams after rebalancing. Please cast your votes! https://cwiki.apache.org/confluence/display/KAFKA/KIP-441:+Smooth+Scaling+Out+for+Kafka+Streams Thanks, -John

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

Hi, Ismael, Yes, I agree that there are 2 separate discussion points, one on the API and the other on the implementation. First on the API, in general, my feeling is that the authorize() call is expected to be fast to reduce request latency. Adding an async API could be mis-leading since it

Re: [DISCUSS] KIP-448: Add State Stores Unit Test Support to Kafka Streams Test Utils

Thanks Sophie! I took a look at the issue and the mailing thread. So in other words, people are having issues writing unit tests using in-memory stores (which is a very common practice due to the lack of a better alternative), so we try to provide a better solution for testings, and hopefully

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Hi Guozhang, Your summary corresponds to my proposal. A new value would only be added if in future we change the metrics in a backward-incompatible way, i.e., 2.4-. "latest" will always stay the default. Best, Bruno On Thu, Sep 5, 2019 at 10:57 PM Guozhang Wang wrote: > > Hello Bruno, > > I

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Hello Bruno, I think your concern makes sense, let's adopt this suggestion in KIP-444 instead. Just to clarify: 1. The default value would be "latest". 2. The only other valid value is "0.10.0-2.3". And moving forward this config may stay without any new values. Guozhang On Thu, Sep 5, 2019

Re: Kafka

Hi Ghullam, Apache Kafka is open source. See license under https://github.com/apache/kafka/blob/trunk/LICENSE Best, Bruno On Thu, Sep 5, 2019 at 10:19 PM Ghullam Mohiyudin wrote: > > Hi , > I read the information about kafka. Now i want to create a degree final > project using kafka. Can you

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Aha! I did miss that detail in your email before. Sorry for my density. It does seem like, if it turns out to be a problem, it would be pretty straightforward to add your proposal in. It wouldn't even require a version bump, because the wire protocol and the assignment algorithm would be mutually

[jira] [Created] (KAFKA-8879) GlobalStateUpdateTask uses wrong javaType to deserialize value

Vlad Olevsky created KAFKA-8879: --- Summary: GlobalStateUpdateTask uses wrong javaType to deserialize value Key: KAFKA-8879 URL: https://issues.apache.org/jira/browse/KAFKA-8879 Project: Kafka

Re: [VOTE] KIP-401: TransformerSupplier/ProcessorSupplier StateStore connecting

Thanks all! I updated the KIP status, and will get to the remaining TODOs in my PR hopefully this weekend. > On Sep 3, 2019, at 6:16 PM, Guozhang Wang wrote: > > Hi Paul, thanks for the confirmation! > > Since we have three binding votes now I think you can proceed and mark it > as accepted.

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Hi Guozhang, I think user experience and code maintenance are tightly related. The harder to maintain the code the worse the user experience will get. Making the config optional does not solve the issue. Wouldn't users be puzzled when we release 2.5 and they cannot set built.in.metrics.version

Re: [VOTE] KIP-444: Refactor and Augment Metrics for Kafka Streams

+1 from myself. I'm now officially closing this voting thread with the following tally: binding +1: 3 (Guozhang, Bill, Matthias voted on the DISCUSS thread). non-binding +1: 2 (Bruno, John). Guozhang On Thu, Aug 22, 2019 at 8:16 AM Bill Bejeck wrote: > +1 (binding) > > -Bill > > On Thu,

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

Hi Bruno, Thanks for raising this point. I think the main motivation behind this proposal is, like Matthias said, to ease the understanding burden from users to our own shoulders. Testing wise, I think we do not necessarily need to explode the testing matrix but just test the last version before

Re: [DISCUSS] KIP-444: Augment Metrics for Kafka Streams

I share Bruno's concern about future releases, however, I would make slightly different proposal. Instead of using "latest" we can just make the config optional and if not set, we use the new metrics code? This way we don't need to add a new version number each time we do a new release (note,

Build failed in Jenkins: kafka-trunk-jdk8 #3888

See Changes: [github] KAFKA-8866; Return exceptions as Optional in authorizer [github] KAFKA-8857; Don't check synonyms while determining if config is readOnly --

[jira] [Created] (KAFKA-8878) Flaky Test AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS

Matthias J. Sax created KAFKA-8878: -- Summary: Flaky Test AssignedStreamsTasksTest#shouldCloseCleanlyWithSuspendedTaskAndEOS Key: KAFKA-8878 URL: https://issues.apache.org/jira/browse/KAFKA-8878

[jira] [Resolved] (KAFKA-8724) log cleaner thread dies when attempting to clean a __consumer_offsets partition after upgrade from 2.0->2.3

[ https://issues.apache.org/jira/browse/KAFKA-8724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-8724. Resolution: Fixed > log cleaner thread dies when attempting to clean a __consumer_offsets

[DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

Hi All, I'd like to propose a new consumer metric that measures the time between calls to poll() for use in issues related to hitting max.poll.interval.ms due to long processing time.

Re: [DISCUSS] KIP-441: Smooth Scaling Out for Kafka Streams

Hello John, What I was thinking is that for other hosts of non-logged store only tasks, they will be ranked with a lag of "1" (as I tried to illustrate with "StatefulTasksToRankedCandidates[task][1] := instance" in the previous email), so that the current host would be ranked first, while others

Build failed in Jenkins: kafka-trunk-jdk11 #795

See Changes: [github] KAFKA-8866; Return exceptions as Optional in authorizer [github] KAFKA-8857; Don't check synonyms while determining if config is readOnly --

Re: [DISCUSS] KIP-470: TopologyTestDriver test input and output usability improvements

Hi Jukka! I just came across your work - it looks great! I was taking a stab at improving the existing API, but yours already looks great and just about complete! Are you planning on continuing your work and submitting a PR? If you want some help, I'd be happy to jump in. Regards, Antony. On

Build failed in Jenkins: kafka-trunk-jdk11 #794

See Changes: [manikumar.reddy] MINOR: Fix few typos in the javadocs/docs -- [...truncated 2.62 MB...]

[jira] [Resolved] (KAFKA-8866) Make Authorizer create/delete exceptions Optional

[ https://issues.apache.org/jira/browse/KAFKA-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8866. --- Reviewer: Ismael Juma Resolution: Fixed > Make Authorizer create/delete exceptions

[jira] [Resolved] (KAFKA-8857) Config describe should not return isReadOnly=false based on synonyms

[ https://issues.apache.org/jira/browse/KAFKA-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-8857. --- Reviewer: Manikumar Resolution: Fixed > Config describe should not return

[jira] [Created] (KAFKA-8877) Race condition on partition counter

Oleg Kuznetsov created KAFKA-8877: - Summary: Race condition on partition counter Key: KAFKA-8877 URL: https://issues.apache.org/jira/browse/KAFKA-8877 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-trunk-jdk8 #3887

See Changes: [wangguoz] KAFKA-8729, pt 1: Add 4 new metrics to keep track of various types of [manikumar.reddy] MINOR: Fix few typos in the javadocs/docs --