[jira] [Resolved] (KAFKA-7101) Session Window store should set topic policy `compact,cleanup`

[ https://issues.apache.org/jira/browse/KAFKA-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-7101. -- Resolution: Fixed Fix Version/s: 2.1.0 > Session Window store should set topic policy

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hey Lucas, Thanks for the reply. Some follow up questions below. Regarding 1, if each ProduceRequest covers 20 partitions that are randomly distributed across all partitions, then each ProduceRequest will likely cover some partitions for which the broker is still leader after it quickly

Re: [DISCUSS] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

> Can you provide a link, please? Pardon me. I put the page in the incorrect location. https://cwiki.apache.org/confluence/display/KAFKA/KIP-331+Add+default+implementation+to+close%28%29+and+configure%28%29+for+Serializer%2C+Deserializer+and+Serde Cheers, Chia-Ping On 2018/07/02 19:45:19, John

Build failed in Jenkins: kafka-2.0-jdk8 #68

See Changes: [rajinisivaram] MINOR: Close timing window in SimpleAclAuthorizer startup (#5318) -- [...truncated 867.77 KB...] kafka.controller.ReplicaStateMachineTest >

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

For #1, I don't know what would be good approximation for M. Maybe use max((TO / 2) / N, M / N) as default value for poll timeout ? For #2, I don't see the picture in email :-) Can you use third party website ? Thanks On Mon, Jul 2, 2018 at 5:17 PM, Lucas Wang wrote: > Hi Ted, > > 1. I'm

Build failed in Jenkins: kafka-0.10.2-jdk7 #223

See Changes: [rajinisivaram] MINOR: Close timing window in SimpleAclAuthorizer startup (#5318) -- [...truncated 1.31 MB...]

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

Hi Ted, 1. I'm neutral on making the poll timeout parameter configurable. Mainly because as a config, it could be confusing for operators who try to choose a value for it. To understand the implication of this value better, let's use TO to represent the timeout value under discussion, M to

Build failed in Jenkins: kafka-0.11.0-jdk7 #388

See Changes: [rajinisivaram] MINOR: Close timing window in SimpleAclAuthorizer startup (#5318) -- [...truncated 1.56 MB...]

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hey John, Obviously I'm too lazy on email replying diligence compared with you :) Will try to reply them separately: - To reply your email on "Mon, Jul 2, 2018 at 8:23 AM": I'm aware of this use case, but again, the

Jenkins build is back to normal : kafka-1.1-jdk7 #159

See

Re: [DISCUSS] KIP-326: Schedulable KTable as Graph source

Hi Flávio, Sure thing. And apologies in advance if I missed the point. Below is some more-or-less realistic Java code to demonstrate how, given a high-volume (heavily partitioned) stream of purchases, we can "step down" the update rate with rate-limited intermediate aggregations. Please bear in

Re: [VOTE] KIP-330: Add retentionPeriod in SessionBytesStoreSupplier

+1 from myself as well. I'm closing this thread with the tally below: binding +1: 3 (Damian, Matthias, myself) non-binding +1: 3 (Ted, Bill, John) Thanks for everyone's votes! Guozhang On Fri, Jun 29, 2018 at 1:15 PM, Matthias J. Sax wrote: > Thx. That makes sense. Just want to make sure

Re: [VOTE] 2.0.0 RC1

+1.  1) Ran unit tests  2) 3 node cluster , tested basic operations.  Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" wrote: > > > > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu (with Java 8). > > > Minor: It seems this doc

Re: [VOTE] 2.0.0 RC1

+1 . * Ran unit tests * Verified signatures * Ran 3 node cluster with basic operations Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:13 AM, "Vahid S Hashemian" wrote: > > > > +1 (non-binding) > > Built from source and ran quickstart successfully on Ubuntu (with Java 8). > > > Minor: It

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Dong, Thanks for your valuable comments. Please see my reply below. 1. The Google doc showed only 1 partition. Now let's consider a more common scenario where broker0 is the leader of many partitions. And let's say for some reason its IO becomes slow. The number of leader partitions on

Re: [DISCUSS] KIP-326: Schedulable KTable as Graph source

Thanks for clarifying the real usage of KIP-328. Now I understood a bit better. I didn't see how that feature would be used to minimize the number of publications to the single partitioned output topic. When it is falls into supression, the graph stops going down? Could you explain better? If

Re: [kafka-clients] [VOTE] 1.0.2 RC1

+1.      1) Ran unit tests 2) 3 node cluster , tested basic operations. Thanks, Harsha On Mon, Jul 2nd, 2018 at 11:57 AM, Jun Rao wrote: > > > > Hi, Matthias, > > Thanks for the running the release. Verified quickstart on scala 2.12 > binary. +1 > > Jun > > On Fri, Jun 29, 2018 at 10:02

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

In fact, to push the idea further (which IIRC is what Matthias originally proposed), if we can accept "Suppression#finalResultsOnly" in my last email, then we could also consider whether to eliminate "suppressLateEvents" entirely. We could always add it later, but you've both expressed doubt that

Re: [DISCUSS] KIP-326: Schedulable KTable as Graph source

Hi Flávio, Thanks for the KIP. I'll apologize that I'm arriving late to the discussion. I've tried to catch up, but I might have missed some nuances. Regarding KIP-328, the idea is to add the ability to suppress intermediate results from all KTables, not just windowed ones. I think this could

[jira] [Created] (KAFKA-7128) Lagging high watermark can lead to committed data loss after ISR expansion

Jason Gustafson created KAFKA-7128: -- Summary: Lagging high watermark can lead to committed data loss after ISR expansion Key: KAFKA-7128 URL: https://issues.apache.org/jira/browse/KAFKA-7128

Re: [DISCUSS] KIP-331 Add default implementation to close() and configure() for Serializer, Deserializer and Serde

Hi Chia-Ping, I couldn't find KIP-331 in the list of KIPs ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals ). Can you provide a link, please? Thanks, -John On Sun, Jul 1, 2018 at 11:33 AM Chia-Ping Tsai wrote: > hi folks, > > KIP-331 is waiting for any

Re: [kafka-clients] [VOTE] 1.0.2 RC1

Hi, Matthias, Thanks for the running the release. Verified quickstart on scala 2.12 binary. +1 Jun On Fri, Jun 29, 2018 at 10:02 PM, Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 1.0.2. > > This is a

Build failed in Jenkins: kafka-0.11.0-jdk7 #387

See Changes: [matthias] Bump version to 0.11.0.3 [jason] Revert "MINOR: Use kill_java_processes when killing ConsoleConsumer in [matthias] Bump version to 0.11.0.4-SNAPSHOT

Re: [VOTE] 2.0.0 RC1

+1 (non-binding) Built from source and ran quickstart successfully on Ubuntu (with Java 8). Minor: It seems this doc update PR is not included in the RC: https://github.com/apache/kafka/pull/5280 Guozhang seems to have wanted to cherry-pick it to 2.0. Thanks Rajini! --Vahid From: Rajini

Re: [DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hi Dong, Thanks, I've been thinking about your suggestions a bit. It is challenging to make this work given the current APIs. One of the difficulties is that we don't have an API to find the leader epoch for a given offset at the moment. So if the user does a seek to offset 5, then we'll need a

Build failed in Jenkins: kafka-0.10.2-jdk7 #222

See Changes: [matthias] Bump version to 0.10.2.2 [jason] Revert "MINOR: Use kill_java_processes when killing ConsoleConsumer in [matthias] Bump version to 0.10.2.3-SNAPSHOT

Re: [VOTE] KIP-280: Enhanced log compaction

Hi Luis, I believe that compaction property is indeed overridable at per-topic level, as in https://github.com/apache/kafka/blob/0cacbcf30e0a90ab9fad7bc310e5477cf959f1fd/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L116 And also documented in

Re: [RESULTS] [VOTE] Release Kafka version 0.11.0.3

Correcting the vote breakdown. Harash should be listed as a committer. Sorry for the mistake! This vote passes with 8 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Jun * Rajini * Ismael Committers: * Harash * Matthias Community: * Vahid * Manikumar * Ted 0 votes * No

Re: Contribute : assign JIRAs and create KIP permissions

Done. On 7/2/18 9:35 AM, Saïd Bouras wrote: > Hi everyone, > > I am looking for a while now to contribute to Apache Kafka. I would like to > have permissions to create KIP and assign JIRAs to myself. > > My jira id is : sbouras > > Thanks in advance :-) > signature.asc Description: OpenPGP

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hi again, Guozhang ;) Here's the second part of my response... It seems like your main concern is: "if I'm a user who wants final update semantics, how complicated is it for me to get it?" I think we have to assume that people don't always have time to become deeply familiar with all the nuances

[jira] [Resolved] (KAFKA-6583) Metadata should include number of state stores for task

[ https://issues.apache.org/jira/browse/KAFKA-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6583. Resolution: Duplicate > Metadata should include number of state stores for task >

[jira] [Reopened] (KAFKA-6583) Metadata should include number of state stores for task

[ https://issues.apache.org/jira/browse/KAFKA-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reopened KAFKA-6583: > Metadata should include number of state stores for task >

[RESULTS] [VOTE] Release Kafka version 0.11.0.3

This vote passes with 8 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Jun * Rajini * Ismael Committers: * Matthias Community: * Vahid * Manikumar * Ted * Harash 0 votes * No votes -1 votes * No votes Vote thread:

Re: [kafka-clients] Re: [VOTE] 0.11.0.3 RC0

+1 -Matthias On 7/1/18 11:28 AM, Manikumar wrote: > +1 (non-binding)   Verified quick start, ran producer/consumer perf tests, > > Thanks > > > On Sun, Jul 1, 2018 at 8:29 PM Ismael Juma > wrote: > > +1 (binding) > > Verified signature of source artifact,

Re: Please, add me to the committers!

Hello Matthias, thank you very much the informaton and for making me welcome in Apache Kafka community. I look forward to contribute! Kind Regards, Daniele Ascione Il giorno ven 29 giu 2018 alle ore 19:31 Matthias J. Sax < matth...@confluent.io> ha scritto: > Daniele, > > thanks for your

Contribute : assign JIRAs and create KIP permissions

Hi everyone, I am looking for a while now to contribute to Apache Kafka. I would like to have permissions to create KIP and assign JIRAs to myself. My jira id is : sbouras Thanks in advance :-) -- *Saïd BOURAS*

[RESULTS] [VOTE] Release Kafka version 0.10.2.2

This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Jason * Guozhang * Jun Committers: * Matthias Community: * Ted 0 votes * No votes -1 votes * No votes Vote thread: http://search-hadoop.com/m/Kafka/uyzND1Wt6721GMzOE1?subj=+VOTE+0+10+2+2+RC1 I'll

Re: [kafka-clients] [VOTE] 0.10.2.2 RC1

+1 -Matthias On 6/29/18 2:46 PM, Jun Rao wrote: > Hi, Matthias, > > Thanks for running the release. Verified quickstart on scala 2.12 binary. +1 > > Jun > > On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax > wrote: > > Hello Kafka users, developers and

Request Permission to Create KIP and Assign JIRAs

Hi everyone, I am looking for a while now to contribute to Apache Kafka. I would like to have permissions to create KIP and assign JIRAs to myself. Thanks in advance :-) -- *Saïd BOURAS* Data Engineer Mobile: 0662988731 Zenika Paris 10 rue de Milan 75009 Paris Standard : +33(0)1 45 26 19 15

[jira] [Resolved] (KAFKA-6583) Metadata should include number of state stores for task

[ https://issues.apache.org/jira/browse/KAFKA-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6583. Resolution: Fixed > Metadata should include number of state stores for task >

Re: [DISCUSS] KIP-328: Ability to suppress updates for KTables

Hi Guozhang, Thanks for the clarification. To answer your questions: 1. Yes, specifically Y < X makes sense and is by design. The scenario is to support IQ queries over windows that are closed but not evicted. For example, suppose we have a metrics application backed by Streams. Let's say we do

Re: [VOTE] KIP-280: Enhanced log compaction

Hi Guozhang, You are right that it is not straightforward to add a dependent property validation.  Though it is possible to re-design it to allow for this, that effort would be better placed under its own KIP, if it really becomes useful for other properties as well. Given this, the

Re: [DISCUSS] KIP-322: Return new error code for DeleteTopics API when topic deletion disabled.

Since this is a minor KIP, If there are no concerns I will start voting thread. On Wed, Jun 27, 2018 at 2:24 PM Ted Yu wrote: > bq. set the serialVersionUID = 1L in Kafka exception classes > > Indeed. Should have checked earlier. > > On Wed, Jun 27, 2018 at 1:38 AM, Manikumar > wrote: > > >

Re: [VOTE] KIP-280: Enhanced log compaction

Hi Guozhang, At the moment the KIP has your vote, Matthias' and Ted's. Should I ask someone else to have a look? Cheers, Luis On Monday, July 2, 2018, 12:16:48 PM GMT+2, Mickael Maison wrote: +1 (non binding). Thanks for the KIP! On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang

Re: [VOTE] KIP-280: Enhanced log compaction

+1 (non binding). Thanks for the KIP! On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang wrote: > Hi Luis, > > Regarding the minor suggest, I agree it would be better to make it as > mandatory, but it might be a bit tricky because it is a conditional > mandatory one depending on the other config's

Re: [kafka-clients] [VOTE] 2.0.0 RC1

+1 (non-binding) - Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag - Ran through quickstart of core/streams on builds. Thanks, Satish. On Sun, Jul 1, 2018 at 6:00 AM, Ted Yu wrote: > +1 > > Checked signatures > Ran test suite > > On Sat, Jun 30, 2018 at 2:37 AM, Rajini Sivaram > wrote: > > >

[jira] [Created] (KAFKA-7127) Add asynchronous support for methods in KafkaConsumer

Richard Yu created KAFKA-7127: - Summary: Add asynchronous support for methods in KafkaConsumer Key: KAFKA-7127 URL: https://issues.apache.org/jira/browse/KAFKA-7127 Project: Kafka Issue Type: