Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Hi Jun, same for me. jeffc...@me.com Thanks, Jeff Chao On Thu, Apr 5, 2018 at 9:06 PM, Jun Rao wrote: > Hi, James, Jeff, > > Added you to the invitation. > > Jun > > On Thu, Apr 5, 2018 at 6:11 PM, Jeff Widman wrote: > > > Please add me as well:

[jira] [Created] (KAFKA-6756) client.id and group.id validation in the old vs new consumer

Badai Aqrandista created KAFKA-6756: --- Summary: client.id and group.id validation in the old vs new consumer Key: KAFKA-6756 URL: https://issues.apache.org/jira/browse/KAFKA-6756 Project: Kafka

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Hi, James, Jeff, Added you to the invitation. Jun On Thu, Apr 5, 2018 at 6:11 PM, Jeff Widman wrote: > Please add me as well: j...@jeffwidman.com > > thanks > > On Thu, Apr 5, 2018 at 5:16 PM, James Cheng wrote: > > > Jun, > > > > Can you add me as

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Thanks for the KIP! +1 (binding) On Thu, Apr 5, 2018 at 3:11 PM, Matthias J. Sax wrote: > +1 (binding) > > > -Matthias > > On 4/5/18 4:36 AM, Ted Yu wrote: > > +1 > > Original message From: Mickael Maison < > mickael.mai...@gmail.com> Date: 4/5/18 1:42

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Hey guys, thanks for the discussion. One good point Matthias pointed out was that the whole "consumer config separation" is for advanced users. In case someone wants to set different consumer configs, he must already be searching the codebase to figure out how to do that; if a user doesn't

Jenkins build is back to normal : kafka-1.1-jdk7 #113

See

[jira] [Created] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null

Randall Hauch created KAFKA-6755: Summary: MaskField SMT should optionally take a literal value to use instead of using null Key: KAFKA-6755 URL: https://issues.apache.org/jira/browse/KAFKA-6755

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Please add me as well: j...@jeffwidman.com thanks On Thu, Apr 5, 2018 at 5:16 PM, James Cheng wrote: > Jun, > > Can you add me as well? wushuja...@gmail.com > > Thanks, > -James > > > On Apr 5, 2018, at 1:34 PM, Jun Rao

Jenkins build is back to normal : kafka-trunk-jdk9 #537

See

[jira] [Created] (KAFKA-6754) Allow Kafka to be used for horizontally-scalable real-time stream visualization

Bill DeStein created KAFKA-6754: --- Summary: Allow Kafka to be used for horizontally-scalable real-time stream visualization Key: KAFKA-6754 URL: https://issues.apache.org/jira/browse/KAFKA-6754 Project:

[jira] [Created] (KAFKA-6753) Speed up event processing on the controller

Lucas Wang created KAFKA-6753: - Summary: Speed up event processing on the controller Key: KAFKA-6753 URL: https://issues.apache.org/jira/browse/KAFKA-6753 Project: Kafka Issue Type: Improvement

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Jun, Can you add me as well? wushuja...@gmail.com Thanks, -James > On Apr 5, 2018, at 1:34 PM, Jun Rao wrote: > > Hi, Ted, Vahid, > > Added you to the invite. > > Thanks, > > Jun > > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <

Re: [DISCUSS] KIP-257 - Configurable Quota Management

Hi Rajini, Can you share the motivation for the change? Not sure if the new approach is better. Ismael On Thu, Apr 5, 2018 at 4:06 PM, Rajini Sivaram wrote: > The quota callback interface was updated based on Jun's suggestion during > the PR review. I have updated the

Jenkins build is back to normal : kafka-1.0-jdk7 #184

See

Re: [DISCUSS] KIP-257 - Configurable Quota Management

The quota callback interface was updated based on Jun's suggestion during the PR review. I have updated the KIP to reflect this. Let me know if there are any concerns. Thanks, Rajini On Thu, Apr 5, 2018 at 1:04 PM, Rajini Sivaram wrote: > Thanks, Ismael. > > I have

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, For the password example, you could define a login CallbackHandler that processes PasswordCallback to provide passwords. We don't currently do this with PLAIN/SCRAM because login callback handlers were not configurable earlier and we haven't updated the login modules to do this. But that

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Ewen, I cannot completely follow your argument. Can you elaborate a little bit? After reading you mail, I am not sure if you prefer config inheritance or not? And if, to what extend? > In that mode if you override *anything*, you >> should specify *everything*. Can you give an example for

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 (binding) -Matthias On 4/5/18 4:36 AM, Ted Yu wrote: > +1 > Original message From: Mickael Maison > Date: 4/5/18 1:42 AM (GMT-08:00) To: dev > Subject: Re: [VOTE] KIP-211: Revise Expiration > Semantics of Consumer Group

Re: [DISCUSS] KIP-219 - Improve Quota Communication

Hi, I have been implementing KIP-219. I discussed the interface changes with Becket Qin and Dong Lin, and we decided to bump up the protocol version of ApiVersionsRequest and ApiVersionsResponse only, instead of bumping up all requests/responses that may be throttled, to indicate clients whether

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Rajini. Thanks for the questions. I could see someone wanting to retrieve a password from a vended password vault solution (for example); that is the kind of scenario that the ability to add new substitutable types would be meant for. I do still consider this KIP 269 to be a prerequisite for

Build failed in Jenkins: kafka-1.1-jdk7 #112

See Changes: [wangguoz] KAFKA-6748: double check before scheduling a new task after the -- [...truncated 414.03 KB...] kafka.controller.ReplicaStateMachineTest >

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Hi, Ted, Vahid, Added you to the invite. Thanks, Jun On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jun, > > I used to receive these invites, but didn't get this one. > Please send me an invite. Thanks. > > Regards, > --Vahid > > > > From: Jun

Re: [DISCUSS] KIP-279: Fix log divergence between leader and follower after fast leader fail over

For the second alternative which was rejected (The follower sends all sequences of {leader_epoch, end_offset}) bq. also increases the size of OffsetForLeaderEpoch request by at least 64bit Though the size increases, the number of roundtrips is reduced meaningfully which would increase the

Build failed in Jenkins: kafka-trunk-jdk9 #536

See Changes: [wangguoz] KAFKA-6748: double check before scheduling a new task after the -- [...truncated 1.48 MB...] kafka.server.epoch.LeaderEpochFileCacheTest >

Jenkins build is back to normal : kafka-trunk-jdk9 #535

See

[DISCUSS] KIP-279: Fix log divergence between leader and follower after fast leader fail over

Hi, I just created KIP-279 to fix edge cases of log divergence for both clean and unclean leader election configs. https://cwiki.apache.org/confluence/display/KAFKA/KIP-279%3A+Fix+log+divergence+between+leader+and+follower+after+fast+leader+fail+over The KIP is basically a follow up to

Jenkins build is back to normal : kafka-trunk-jdk8 #2529

See

Re: [DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

Sure, updated the table under 'Public Interfaces' by adding the TimeUnit column. Thank you > In the table under 'Public Interfaces', please add a column with TimeUnit. > Looks good overall. ‌

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

Ewen, thanks for your thoughts. Just to clarify the current KIP does not propose for four new prefixes, but three plus the existing one. So it is "consumer" "main.consumer" "restore.consumer" "global.consumer" If we design the configs for the first time, I'd be in favor of only keeping the

[jira] [Resolved] (KAFKA-6748) Scheduler cannot be cancelled from Punctuator

[ https://issues.apache.org/jira/browse/KAFKA-6748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6748. -- Resolution: Fixed Fix Version/s: (was: 1.0.2) 1.2.0 > Scheduler

[jira] [Resolved] (KAFKA-6413) ReassignPartitionsCommand#parsePartitionReassignmentData() should give better error message when JSON is malformed

[ https://issues.apache.org/jira/browse/KAFKA-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6413. -- Resolution: Duplicate Resolving as duplicate of KAFKA-6084. >

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Hi Jun, I used to receive these invites, but didn't get this one. Please send me an invite. Thanks. Regards, --Vahid From: Jun Rao To: dev Date: 04/05/2018 10:25 AM Subject:Kafka KIP meeting on Apr. 9 at 9:00am PDT Hi, Everyone,

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Jun: I would be interested. Is there going to be an online meeting ? Thanks On Thu, Apr 5, 2018 at 10:24 AM, Jun Rao wrote: > Hi, Everyone, > > We plan to have a Kafka KIP meeting this coming Monday (Apr. 9) at 9:00am > PDT. If you plan to attend but haven't received an

Kafka KIP meeting on Apr. 9 at 9:00am PDT

Hi, Everyone, We plan to have a Kafka KIP meeting this coming Monday (Apr. 9) at 9:00am PDT. If you plan to attend but haven't received an invite, please let me know. The following is the agenda. Agenda: KIP-253: Partition expansion Thanks, Jun

Build failed in Jenkins: kafka-trunk-jdk9 #534

See Changes: [rajinisivaram] KAFKA-6741: Disable Selector's idle connection timeout in -- [...truncated 1.48 MB...] kafka.message.ByteBufferMessageSetTest >

[DISCUSS] KIP-282: Add the listener name to the authentication context

Hi all, I have submitted KIP-282 to add the listener name to the authentication context: https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+authentication+context This is a very minor KIP to simplify identifying the source of a connection instead of relying

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

Thanks everyone, Voting for KIP-274 has passed with 3 binding and 4 non-binding +1s (and no objections). Thanks for your time in considering this KIP, -John On Thu, Apr 5, 2018 at 4:48 AM, Damian Guy wrote: > +1 > > On Thu, 5 Apr 2018 at 01:56 Matthias J. Sax

Build failed in Jenkins: kafka-trunk-jdk8 #2528

See Changes: [rajinisivaram] KAFKA-6694: The Trogdor Coordinator should support filtering task [rajinisivaram] KAFKA-6741: Disable Selector's idle connection timeout in

[jira] [Created] (KAFKA-6751) Make max.connections.per.ip.overrides a dynamic config

Jason Gustafson created KAFKA-6751: -- Summary: Make max.connections.per.ip.overrides a dynamic config Key: KAFKA-6751 URL: https://issues.apache.org/jira/browse/KAFKA-6751 Project: Kafka

Security Vulnerabilities

Is there a list including Kafka's outstanding security vulnerability issues like the ones posted in the majority of CVE databases. I am looking for something like this --> https://www.cvedetails.com/product/27453/Apache-Zookeeper.html?vendor_id=45

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Hi Matthias, Thanks a lot for reviewing the KIP and the clarification on streams protocol type for consumer groups. I have updated the KIP with your suggestion. I'm assuming the cast vote so far will remain valid since this is a minor change. Cheers, --Vahid From: "Matthias J. Sax"

Re: [DISCUSS] KIP-280: Enhanced log compaction

In the 'Proposed Changes' section, can you expand 'OCC' ? bq. Specifically changing this to anything other than "*offset*" Is it possible to enumerate the keys ? In the future, more metadata would be defined in record header - it is better to avoid collision. Cheers On Thu, Apr 5, 2018 at 2:05

Re: [DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

In the table under 'Public Interfaces', please add a column with TimeUnit. Looks good overall. On Thu, Apr 5, 2018 at 7:14 AM, Alex Dunayevsky wrote: > Hello friends, > > I would like to start a discussion thread for KIP-281: >

Jenkins build is back to normal : kafka-trunk-jdk8 #2527

See

[jira] [Created] (KAFKA-6750) Add listener name to AuthenticationContext

Mickael Maison created KAFKA-6750: - Summary: Add listener name to AuthenticationContext Key: KAFKA-6750 URL: https://issues.apache.org/jira/browse/KAFKA-6750 Project: Kafka Issue Type:

[DISCUSS] KIP-281 ConsumerPerformance: Increase Polling Loop Timeout and Make It Reachable by the End User

Hello friends, I would like to start a discussion thread for KIP-281: https://cwiki.apache.org/confluence/display/KAFKA/KIP-281%3A+ConsumerPerformance%3A+Increase+Polling+Loop+Timeout+and+Make+It+Reachable+by+the+End+User JIRA: KAFKA-6743: ConsumerPerformance fails to consume all messages on

[jira] [Resolved] (KAFKA-6741) Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded

[ https://issues.apache.org/jira/browse/KAFKA-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-6741. --- Resolution: Fixed Assignee: Manikumar Fix Version/s: 1.2.0 > Transient test

Re: [DISCUSS] KIP-269: Substitution Within Configuration Values

Hi Ron, Now that login callback handlers are configurable, is this KIP still a pre-req for OAuth? I was wondering whether we still need the ability to add new substitutable types or whether it would be sufficient to add the built-in ones to read from file etc. On Thu, Mar 29, 2018 at 6:48 AM,

Re: [DISCUSS] KIP-257 - Configurable Quota Management

Thanks, Ismael. I have updated the KIP and the PR. On Wed, Apr 4, 2018 at 7:37 PM, Ismael Juma wrote: > Sounds good to me Rajini. Good catch spotting this before it's included in > a release. :) > > Ismael > > On Wed, Apr 4, 2018 at 11:13 AM, Rajini Sivaram

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 Original message From: Mickael Maison Date: 4/5/18 1:42 AM (GMT-08:00) To: dev Subject: Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets +1 (non-binding) Thanks for the KIP! On Thu, Apr 5, 2018 at

Build failed in Jenkins: kafka-trunk-jdk8 #2526

See Changes: [jason] MINOR: Don’t send the DeleteTopicsRequest for invalid topic names -- [...truncated 3.53 MB...] kafka.zookeeper.ZooKeeperClientTest >

[jira] [Created] (KAFKA-6749) TopologyTestDriver fails when topoloy under test uses EXACTLY_ONCE

Frederic Arno created KAFKA-6749: Summary: TopologyTestDriver fails when topoloy under test uses EXACTLY_ONCE Key: KAFKA-6749 URL: https://issues.apache.org/jira/browse/KAFKA-6749 Project: Kafka

Re: [VOTE] KIP-274: Kafka Streams Skipped Records Metrics

+1 On Thu, 5 Apr 2018 at 01:56 Matthias J. Sax wrote: > +1 (binding) on the latest proposal. > > > -Matthias > > > On 4/2/18 12:10 PM, Richard Yu wrote: > > +1 > > > > On Mon, Apr 2, 2018 at 8:42 AM, Guozhang Wang > wrote: > > > >> +1 (binding). > >>

[jira] [Created] (KAFKA-6748) Scheduler cannot be cancelled from Punctuator

Frederic Arno created KAFKA-6748: Summary: Scheduler cannot be cancelled from Punctuator Key: KAFKA-6748 URL: https://issues.apache.org/jira/browse/KAFKA-6748 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

Yes I can be there. On 04.04.2018 22:34, Jun Rao wrote: Hi, Jan, Dong, John, Guozhang, Perhaps it will be useful to have a KIP meeting to discuss this together as a group. Would Apr. 9 (Monday) at 9:00am PDT work? If so, I will send out an invite to the mailing list. Thanks, Jun On Wed,

Re: [DISCUSS] KIP-280: Enhanced log compaction

This is embarassingly hard to fix... going again... KIP-280:   https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction - Pull-4822:  https://github.com/apache/kafka/pull/4822 On Thursday, April 5, 2018, 11:03:22 AM GMT+2, Luís Cabral

Re: [DISCUSS] KIP-280: Enhanced log compaction

 Fixing the links:KIP-280:   https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822:   https://github.com/apache/kafka/pull/4822 On 2018/04/0508:44:00, Luís Cabral wrote: > Helloall,> > Starting adiscussion for this feature.>

[jira] [Resolved] (KAFKA-4292) KIP-86: Configurable SASL callback handlers

[ https://issues.apache.org/jira/browse/KAFKA-4292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-4292. --- Resolution: Fixed Reviewer: Jun Rao Fix Version/s: 1.2.0 > KIP-86:

[DISCUSS] KIP-280: Enhanced log compaction

Hello all, Starting a discussion for this feature. KIP-280   :  https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compactionPull-4822 :  https://github.com/apache/kafka/pull/4822 Kind Regards,Luís

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 (non-binding) Thanks for the KIP! On Thu, Apr 5, 2018 at 8:08 AM, Jason Gustafson wrote: > +1 Thanks Vahid! > > On Wed, Mar 28, 2018 at 7:27 PM, James Cheng wrote: > >> +1 (non-binding) >> >> Thanks for all the hard work on this, Vahid! >> >> -James

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 Thanks Vahid! On Wed, Mar 28, 2018 at 7:27 PM, James Cheng wrote: > +1 (non-binding) > > Thanks for all the hard work on this, Vahid! > > -James > > > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > > Hi all, > > > > As I

Re: [DISCUSS] KIP-276: Add StreamsConfig prefix for different consumers

I think this model is more confusing than it needs to be. We end up with 4 prefixes despite only have 3 types of consumers. We have prefixes for: "base", "main", "global", and "restore". However, we only instantiate consumers of type "main", "global", and "restore". Until now, we've only had two