Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

My previous response was talking about the new method in InternalTopologyBuilder. The exception just means there is no uniform extractor for all the sinks. On Mon, Jun 25, 2018 at 8:02 PM, Matthias J. Sax wrote: > Ted, > > Why? Each sink can have a different TopicNameExtractor. > > > -Matthias

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 (binding) On 6/25/18 3:00 PM, Guozhang Wang wrote: > +1 > > On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu wrote: > >> +1 >> >> On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: >> >>> Hello All, >>> >>> Thanks for the discussion on KIP-319. I'd now like to start the voting. >>> >>> As a

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

KAFKA-7080 is for this KIP. I meant to create a JIRA to add `segmentInterval` to `Materialized` and a JIRA to add `Materialized` to `KStream#join(KStream)`. Thx. -Matthias On 6/25/18 2:46 PM, John Roesler wrote: > Ah, it turns out I did create a ticket: it's KAFKA-7080: >

Re: [VOTE] 1.0.2 RC0

+1 (non-binding) Verified tests, quick start, producer/consumer perf tests. On Sat, Jun 23, 2018 at 2:25 AM Ted Yu wrote: > +1 > > Ran test suite. > > Checked signatures. > > On Fri, Jun 22, 2018 at 11:42 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > +1 (non-binding) > > > >

Build failed in Jenkins: kafka-trunk-jdk8 #2767

See Changes: [jason] KAFKA-6591; Move super user check before ACL matching (#4618) -- [...truncated 1.94 MB...] org.apache.kafka.common.acl.AclOperationTest > testName

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Ted, Why? Each sink can have a different TopicNameExtractor. -Matthias On 6/25/18 5:19 PM, Ted Yu wrote: > If there are different TopicNameExtractor classes from multiple sink nodes, > the new method should throw exception alerting user of such scenario. > > > On Mon, Jun 25, 2018 at 2:23

[jira] [Created] (KAFKA-7098) Improve accuracy of the log cleaner throttle rate

Dong Lin created KAFKA-7098: --- Summary: Improve accuracy of the log cleaner throttle rate Key: KAFKA-7098 URL: https://issues.apache.org/jira/browse/KAFKA-7098 Project: Kafka Issue Type:

Re: SASL Unit test failing

I ran the test on Linux as well. cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) Java version: 1.8.0_161, vendor: Oracle Corporation Java home: /jdk1.8.0_161/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.28.3.el7.x86_64", arch: "amd64",

Re: SASL Unit test failing

Here was the command I used: ./gradlew -Dtest.single=SaslAuthenticatorTest clients:test On Mon, Jun 25, 2018 at 5:39 PM, Ahmed A wrote: > I ran test with -i option as follows - "./gradlew -i test". The same set > of three tests failed. > > My environment: > $ java -version > java version

Re: SASL Unit test failing

I ran test with -i option as follows - "./gradlew -i test". The same set of three tests failed. My environment: $ java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) $ cat

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

If there are different TopicNameExtractor classes from multiple sink nodes, the new method should throw exception alerting user of such scenario. On Mon, Jun 25, 2018 at 2:23 PM, Bill Bejeck wrote: > Thanks for the KIP! > > Overall I'm +1 on the KIP. I have one question. > > The KIP states

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Good catch. I think the proposed change is to add that function in InternalTopologyBuilder#Sink class. Guozhang On Mon, Jun 25, 2018 at 2:23 PM, Bill Bejeck wrote: > Thanks for the KIP! > > Overall I'm +1 on the KIP. I have one question. > > The KIP states that the method

Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source

Flávio, thanks for creating this KIP. I think this "single-aggregation" use case is common enough that we should consider how to efficiently supports it: for example, for KSQL that's built on top of Streams, we've seen lots of query statements whose return is expected a single row indicating the

Re: [DISCUSS] KIP-323: Schedulable KTable as Graph source

This would be useful feature. In the Public Interfaces section, the new method lacks a closing parenthesis. In the Proposed Changes section, if the order of the 3 bullets can match the order of the parameters of the new method, it would be easier to read. For Rejected Alternatives #2, can you

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 On Mon, Jun 25, 2018 at 2:58 PM, Ted Yu wrote: > +1 > > On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: > > > Hello All, > > > > Thanks for the discussion on KIP-319. I'd now like to start the voting. > > > > As a reminder, KIP-319 proposes a fix to an issue I identified in > >

Re: [VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

+1 On Mon, Jun 25, 2018 at 2:56 PM, John Roesler wrote: > Hello All, > > Thanks for the discussion on KIP-319. I'd now like to start the voting. > > As a reminder, KIP-319 proposes a fix to an issue I identified in > KAFKA-7080. Specifically, the issue is that we're creating >

[VOTE] KIP-319: Replace numSegments to segmentInterval in Streams window configurations

Hello All, Thanks for the discussion on KIP-319. I'd now like to start the voting. As a reminder, KIP-319 proposes a fix to an issue I identified in KAFKA-7080. Specifically, the issue is that we're creating CachingWindowStore with the *number of segments* instead of the *segment size*. Here's

Re: [DISCUSS] KIP-291: Have separate queues for control requests and data requests

Hi Harsha, If I understand correctly, the replication quota mechanism proposed in KIP-73 can be helpful in that scenario. Have you tried it out? Thanks, Lucas On Sun, Jun 24, 2018 at 8:28 AM, Harsha wrote: > Hi Lucas, > One more question, any thoughts on making this configurable

[jira] [Resolved] (KAFKA-6978) Make Streams Window retention time strict

[ https://issues.apache.org/jira/browse/KAFKA-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Roesler resolved KAFKA-6978. - Resolution: Fixed Fix Version/s: 2.1.0 This feature was merged in 

Re: SASL Unit test failing

I ran the test alone which passed. Can you include -i on the command line to see if there is some clue from the output ? Here is my environment: Java version: 1.8.0_151, vendor: Oracle Corporation Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre Default locale:

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Ah, it turns out I did create a ticket: it's KAFKA-7080: https://issues.apache.org/jira/browse/KAFKA-7080 -John On Mon, Jun 25, 2018 at 4:44 PM John Roesler wrote: > Matthias, > > That's a good idea. I'm not sure why I didn't... > > Thanks, > -John > > On Mon, Jun 25, 2018 at 4:35 PM Matthias

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Matthias, That's a good idea. I'm not sure why I didn't... Thanks, -John On Mon, Jun 25, 2018 at 4:35 PM Matthias J. Sax wrote: > Ok. > > @John: can you create a JIRA to track this? I think KAFKA-4730 is > related, but actually an own ticket (that is blocked by not having > Materialized for

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Ok. @John: can you create a JIRA to track this? I think KAFKA-4730 is related, but actually an own ticket (that is blocked by not having Materialized for stream-stream joins). -Matthias On 6/25/18 2:10 PM, Bill Bejeck wrote: > I agree that it makes sense to have segmentInterval as a parameter

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

The scenario I had I mind was, that KS is started in one thread while a second thread has a reference to the object to issue queries. If a query is issue before the "main thread" started KS, and the "query thread" knows that it will eventually get started, it can retry. On the other hand, if KS

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Thanks for the KIP! Overall I'm +1 on the KIP. I have one question. The KIP states that the method "topicNameExtractor()" is added to the InternalTopologyBuilder.java. It could be that I'm missing something, but wow does this work if a user has provided different TopicNameExtractor instances

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

I agree that it makes sense to have segmentInterval as a parameter to a store, but I also agree with Guozhang's point about not moving as part of this KIP. Thanks, Bill On Mon, Jun 25, 2018 at 4:17 PM John Roesler wrote: > Thanks Matthias and Guozhang, > > About deprecating the "segments"

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Thanks Matthias and Guozhang, About deprecating the "segments" field instead of making it private. Yes, I just took another look at the code, and that is correct. I'll update the KIP. I do agree that in the long run, it makes more sense as a parameter to the store somehow than as a parameter to

Re: Request access to create KIP

Thank you! On Mon, Jun 25, 2018, 1:03 PM Jason Gustafson wrote: > Done. Thanks for contributing! > > -Jason > > On Mon, Jun 25, 2018 at 12:49 PM, Yishun Guan wrote: > > > Hi, could someone give me access to create KIP? Thanks! - Yishun > > > > On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote:

Re: Request access to create KIP

Done. Thanks for contributing! -Jason On Mon, Jun 25, 2018 at 12:49 PM, Yishun Guan wrote: > Hi, could someone give me access to create KIP? Thanks! - Yishun > > On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote: > > > Hi, my wiki id is gyishun. Thanks! - Yishun > > >

SASL Unit test failing

Hello, I did a fresh clone of the kafka src code, and the following SASL unit tests have been failing consistently: - testMechanismPluggability - testMechanismPluggability - testMultipleServerMechanisms All three tests have similar stack trace: at org.junit.Assert.assertTrue(Assert.java:52) at

Re: Request access to create KIP

Hi, could someone give me access to create KIP? Thanks! - Yishun On Mon, Jun 25, 2018, 10:44 AM Yishun Guan wrote: > Hi, my wiki id is gyishun. Thanks! - Yishun >

[jira] [Created] (KAFKA-7097) VerifiableProducer does not work properly with --message-create-time argument

Jasper Knulst created KAFKA-7097: Summary: VerifiableProducer does not work properly with --message-create-time argument Key: KAFKA-7097 URL: https://issues.apache.org/jira/browse/KAFKA-7097 Project:

Build failed in Jenkins: kafka-trunk-jdk8 #2766

See Changes: [mjsax] Minor: add exception to debug log for -- [...truncated 877.51 KB...] kafka.utils.SchedulerTest > testRestart STARTED kafka.utils.SchedulerTest >

Re: [VOTE] 2.0.0 RC0

+1 (non-binding) Heroku has run our usual set of upgrade and performance tests, and we haven't found any notable issues through that. On Sat, Jun 23, 2018 at 12:30 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 (non-binding) > > Built from source and ran quickstart successfully

Re: [DISCUSS] - KIP-314: KTable to GlobalKTable Bi-directional Join

Thanks for your help so far guys. While I do think that I have a fairly reasonable way forward for restructuring the topologies and threads, there is, unfortunately, what I believe is a fatal flaw that cannot be easily resolved. I have updated the page (

Request access to create KIP

Hi, my wiki id is gyishun. Thanks! - Yishun

[jira] [Created] (KAFKA-7096) Consumer should drop the data for unassigned topic partitions

Mayuresh Gharat created KAFKA-7096: -- Summary: Consumer should drop the data for unassigned topic partitions Key: KAFKA-7096 URL: https://issues.apache.org/jira/browse/KAFKA-7096 Project: Kafka

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

Thanks Bill! +1 On Mon, 25 Jun 2018 at 18:57 Ted Yu wrote: > +1 > > On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang wrote: > > > +1. > > > > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax > > wrote: > > > > > +1 (binding) > > > > > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > > > All, > > >

Re: [kafka-clients] Re: [VOTE] 1.1.1 RC1

+1 (non-binding) Ran tests, Verified quick start, producer/consumer perf tests On Sat, Jun 23, 2018 at 8:11 AM Dong Lin wrote: > Thank you for testing and voting the release! > > I noticed that the date for 1.1.1-rc1 is wrong. Please kindly test and > vote by Tuesday, June 26, 12 pm PT. > >

Re: [Discuss] KIP-321: Add method to get TopicNameExtractor in TopologyDescription

Yup I agree, generally speaking the `toString()` output is not recommended to be relied on programmatically in user's code, but we've observed convenience-beats-any-other-reasons again and again in development unfortunately. I think we should still not claiming it is part of the public APIs that

Re: [DISCUSS] KIP-319: Replace segments with segmentSize in WindowBytesStoreSupplier

Re `segmentInterval` parameter in Windows: currently it is used in two places, the windowed stream aggregation, and the stream-stream joins. For the former, we can potentially move the parameter from windowedBy() to Materialized, but for the latter we currently do not expose a Materialized object

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

Hey Manikumar, Thanks much for the KIP. It looks pretty good. Thanks, Dong On Thu, Jun 21, 2018 at 11:38 PM, Manikumar wrote: > Hi all, > > I have created a KIP to add support for dynamic update of > max.connections.per.ip/max.connections.per.ip.overrides configs > >

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

I'm wondering if StreamThreadNotStarted could be merged into StreamThreadNotRunning, because I think users' handling logic for the third case would be likely the same as the second. Do you have some scenarios where users may want to handle them differently? Guozhang On Sun, Jun 24, 2018 at 5:25

Re: [DISCUSS] KIP-308: Support dynamic update of max.connections.per.ip/max.connections.per.ip.overrides configs

Hey Manikumar, Thanks for the KIP. This seems useful. -Jason On Thu, Jun 21, 2018 at 11:38 PM, Manikumar wrote: > Hi all, > > I have created a KIP to add support for dynamic update of > max.connections.per.ip/max.connections.per.ip.overrides configs > >

[DISCUSS] KIP-323: Schedulable KTable as Graph source

Hey, guys, I've just started a KIP discussion here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-323%3A+Schedulable+KTable+as+Graph+source

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1 On Mon, Jun 25, 2018 at 9:45 AM, Guozhang Wang wrote: > +1. > > On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax > wrote: > > > +1 (binding) > > > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > > All, > > > I'd like to start a vote for this KIP now. > > > > > > Thanks, > > > Bill > > > > > >

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1. On Mon, Jun 25, 2018 at 8:12 AM, Matthias J. Sax wrote: > +1 (binding) > > On 6/25/18 6:11 AM, Bill Bejeck wrote: > > All, > > I'd like to start a vote for this KIP now. > > > > Thanks, > > Bill > > > > -- -- Guozhang

Re: [VOTE] KIP-280: Enhanced log compaction

+1 from me as well. On Mon, Jun 25, 2018 at 8:16 AM, Matthias J. Sax wrote: > +1 from my side for using `compaction.strategy` with values "offset", > "timestamp" and "header" and `compaction.strategy.header` > > -Matthias > > On 6/25/18 1:25 AM, Luís Cabral wrote: > > Hi, > > > > So, is

Re: Request permission to assign JIRA

Added. Thanks for contributing! -Jason On Mon, Jun 25, 2018 at 9:27 AM, lambdaliu(刘少波) wrote: > Hi Team, > > I am trying to claim a bug in Jira, Could you please help me gain the > required permissions. > my JIRA usernane is lambdaliu. > > thanks, > lambdaliu. >

Request permission to assign JIRA

Hi Team, I am trying to claim a bug in Jira, Could you please help me gain the required permissions. my JIRA usernane is lambdaliu. thanks, lambdaliu.

[DISCUSS] KIP-320: Allow fetchers to detect and handle log truncation

Hey All, I wrote up a KIP to handle one more edge case in the replication protocol and to support better handling of truncation in the consumer when unclean leader election is enabled. Let me know what you think.

[jira] [Created] (KAFKA-7095) Low traffic consumer is not consuming messages after the offsets is deleted by Kafka

Aldo Sinanaj created KAFKA-7095: --- Summary: Low traffic consumer is not consuming messages after the offsets is deleted by Kafka Key: KAFKA-7095 URL: https://issues.apache.org/jira/browse/KAFKA-7095

Re: [VOTE] KIP-280: Enhanced log compaction

+1 from my side for using `compaction.strategy` with values "offset", "timestamp" and "header" and `compaction.strategy.header` -Matthias On 6/25/18 1:25 AM, Luís Cabral wrote: > Hi, > > So, is everyone OK using the approach with 2 properties? > > E.g.: > > Scenario 1: >    

Re: [VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

+1 (binding) On 6/25/18 6:11 AM, Bill Bejeck wrote: > All, > I'd like to start a vote for this KIP now. > > Thanks, > Bill > signature.asc Description: OpenPGP digital signature

[VOTE] KIP-312: Add Overloaded StreamsBuilder Build Method to Accept java.util.Properties

All, I'd like to start a vote for this KIP now. Thanks, Bill

Build failed in Jenkins: kafka-2.0-jdk8 #56

See Changes: [rajinisivaram] MINOR: Fix timing issue in advertised listener update test (#5256) -- [...truncated 432.85 KB...] kafka.zookeeper.ZooKeeperClientTest >

[VOTE] KIP-293: Add new metrics for consumer/replication fetch requests

Hello, In the absence of additional feedback on this KIP I'd like to start a vote. To summarize, the KIP simply proposes to add a consumer metric to track the number of fetch requests made by (real) client consumers and not other replicating brokers. KIP link:

[jira] [Created] (KAFKA-7094) Variate should unify code style in one method, and use camel name

Matt Wang created KAFKA-7094: Summary: Variate should unify code style in one method, and use camel name Key: KAFKA-7094 URL: https://issues.apache.org/jira/browse/KAFKA-7094 Project: Kafka

Re: [VOTE] KIP-280: Enhanced log compaction

Hi, So, is everyone OK using the approach with 2 properties? E.g.: Scenario 1:     compaction.strategy: offset     :- Behaviour is the same as what currently exists, where the compaction is done only via the 'offset' Scenario 2:     compaction.strategy: timestamp     :- Similar to

Re: Using Kafka as Primary Datastore without compaction

As per my understanding, you should set the retention period to Long.MAX hours. This will ensure that your messages won't get compacted because retention period is huge. Regards, Aman On Mon, Jun 25, 2018 at 8:27 AM, Barathan Kulothongan wrote: > Hi There, I am currently reading the Kafka