Re: comment in addStandbyTasks() method of StreamThread has some typos

ok, I'll open an PR to fix this. 2017-06-17 0:59 GMT+08:00 Matthias J. Sax : > Thanks for reporting this! > > Would you like to open a MINOR PR to fix it? Don't think we need a Jira > for this. > > -Matthias > > On 6/16/17 9:26 AM, john cheng wrote: > >

Re: kafka-streams repeatedly rebalances on start up

Hi Tom, Thanks a lot for reporting this. We dug into it. It's easy to reproduce (thank a lot to describe a simple way to do that) and it seems to be a bug in Streams... I did open a JIRA: https://issues.apache.org/jira/browse/KAFKA-5464 For using Streams 0.10.2.1, there is nothing we can advice

Kafka-Spark Integration - build failing with sbt

I'm trying to compile kafka & Spark Streaming integration code i.e. reading from Kafka using Spark Streaming, and the sbt build is failing with error - [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found Scala version

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Thank you Vahid I appreciate you time. Arunkumar Pichaimuthu, PMP On Fri, 6/16/17, Vahid S Hashemian wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Friday, June

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Hi Arunkumar, I'm glad you were able to fix the issue. Also glad that the article was helpful. Regarding Kafka SSL configuration, I'm sending some links: - Kafka documentation: http://kafka.apache.org/documentation.html#security_ssl - Apache Kafka Security 101:

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Hi Vahid I deleted the dataDir and dataLogDir and restarted zookeeper,brokers, producers and consumer. Now it works All the messages produced are consumed from the producer. Thanks for all the help. The link you shared helped a lot. I am planning to setup SASL_SSL, I appreciate you advice

RE: IllegalStateException when putting to state store in Transformer implementation

Guozhang, Matthias Thanks for confirming, it was quite clear with current understanding that there was a mistake in the supplier implementation when seeing it. The Javadoc is indeed clear what to do, this was ultimately a failing to read that over the documentation provided on confluent /

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Hi Vahid I am working on the same use case ): . As per the document I was trying to set ACL's for topic which worked and now I am able to start my producer without error. Then I set ACL for the consumer and when I start my consumer it starts without issue. and also able to set second ACL for

Re: IllegalStateException when putting to state store in Transformer implementation

Adrian, I see. That would explain what you see, i.e. all tasks with their own "processor context" are accessing the same state store instance; hence for some task its processor context may not be updated yet while another task is accessing that state store, hence causing the issue. If you are

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Hi Arunkumar, Were you trying the same steps in the document when you got this error? Or you are working on a different use case? Also, I might have missed it in previous emails. What version of Kafka are you using? Thanks. --Vahid From: Arunkumar To:

Re: IllegalStateException when putting to state store in Transformer implementation

That is correct. You need to return a new instance on each call. I am not sure how to improve the docs though: 1) it's a supplier pattern (that should explain it) 2) also the `TransformerSupplier#get()` method JavaDoc says: > /** > * Return a new {@link Transformer} instance. > *

RE: IllegalStateException when putting to state store in Transformer implementation

Just to follow up, after making the change to .transform(() -> new PhaseTransformer<>(evaluator, storeName), transformer.getStoreName()) The problem appears to have gone away so I think my previous hypothesis was correct? Please let me know if this should have made no difference. May be

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

Hi Vahid Thank you for sharing link to set it up. It is really a very useful document. When I ran describe command for group I see this error bin/kafka-consumer-groups --bootstrap-server host:9097 --describe --group arun-group --command-config etc/kafka/producer.properties Note: This will

Re: Single Key Aggregation

I just double checked you example code from an email before. There you are using: stream.flatMap(...) .groupBy((k, v) -> k, Serdes.String(), Serdes.Integer()) .reduce((value1, value2) -> value1 + value2, In you last email, you say that you want to count on category that is contained

Re: Kafka Streams vs Spark Streaming : reduce by window

Thanks Michał! That is very good feedback. -Matthias On 6/16/17 2:38 AM, Michal Borowiecki wrote: > I wonder if it's a frequent enough use case that Kafka Streams should > consider providing this out of the box - this was asked for multiple > times, right? > > Personally, I agree totally with

Re: comment in addStandbyTasks() method of StreamThread has some typos

Thanks for reporting this! Would you like to open a MINOR PR to fix it? Don't think we need a Jira for this. -Matthias On 6/16/17 9:26 AM, john cheng wrote: > https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L1345 >

Re: Kafka 0.11 transactions API question

As Michał said. It's not designed for this use case. Kafka's transaction, are not the same thing as DB transactions and if you break it down, it allows for atomic (multi-partition) writes, but no 2-phase commit. Also, a transaction is "owned" by a single thread (ie, producer) and cannot be

Re: Kafka Streams vs Spark Streaming : reduce by window

I think the question is when do you actually *want* processing time semantics? There are definitely times when its safe to assume the two are close enough that a little lossiness doesn't matter much but it is pretty hard to make assumptions about when the processing time is and has been hard for

Re: Kafka Streams vs Spark Streaming : reduce by window

I think the question is when do you actually *want* processing time semantics? There are definitely times when its safe to assume the two are close enough that a little lossiness doesn't matter much but it is pretty hard to make assumptions about when the processing time is and has been hard for

comment in addStandbyTasks() method of StreamThread has some typos

https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L1345 This Line: log.info("{} Adding assigned standby tasks {}", logPrefix, partitionAssignor .activeTasks()); The parameter is active task, but the info content is

Re: Kafka 0.11 transactions API question

But isn't it a low hanging fruit at this moment? Isn't that just an API limitation and wouldn't the backend for transactions support it with only minor changes to the API (do not fail automatically dangling transactions on Producer restart)? Flushing is already there so that _should_ handle the

Re: Kafka 0.11 transactions API question

I don't think KIP-98 is as ambitious as to provide support for distributed transactions (2 phase commit). It would be great if I was wrong though :P Cheers, Michał On 16/06/17 14:21, Piotr Nowojski wrote: Hi, I'm looking into Kafka's transactions API as proposed in KIP-98. I've read both

Kafka 0.11 transactions API question

Hi, I'm looking into Kafka's transactions API as proposed in KIP-98. I've read both this KIP-98 document and I looked into the code that is on the master branch. I would like to use it to implement some two phase commit mechanism on top of the Kafka's transactions, that would allow me to tie

Re: Kafka Streams vs Spark Streaming : reduce by window

I wonder if it's a frequent enough use case that Kafka Streams should consider providing this out of the box - this was asked for multiple times, right? Personally, I agree totally with the philosophy of "no final aggregation", as expressed by Eno's post, but IMO that is predicated totally

RE: IllegalStateException when putting to state store in Transformer implementation

Hi Guozhang It's just occurred to me that the transformer is added to the topology like this: PhaseTransformer transformer = new PhaseTransformer<>(evaluator, storeName); ... .transform(() -> transformer, transformer.getStoreName()) Thus meaning that the same transformer is used

Re: Slow Consumer Group Startup

Hi Bryan, So this must be something else since KIP-134 is not in 0.10.2.1, but in the new release 0.11 that hasn't come out yet. Eno > On 14 Jun 2017, at 21:35, Bryan Baugher wrote: > > It does seem like we are in a similar situation described in the KIP ( >

Re: NPE on startup with a low-level API based application

Hey Frank, I think I spotted the issue and submitted a patch. Here's a link to the JIRA: https://issues.apache.org/jira/browse/KAFKA-5456. I expect we'll get the fix into 0.11.0.0. Thanks for finding this! -Jason On Thu, Jun 15, 2017 at 11:53 PM, Frank Lyaruu wrote: > Yes,

Re: NPE on startup with a low-level API based application

Yes, compression was on (lz4), key and value sizes fluctuate, key sizes are small <10 bytes, value sizes fluctuate also, but nothing crazy, up to about 100kb. I did some stepping through the code and at some point I saw a branch that used a different path depending on protocol version (something

Re: Single Key Aggregation

Ok.. Let me try explain it again. So, Lets say my source processor has a different key, now the value that it contains lets say contains an identifier type: which basically denotes category and I am counting on different categories. A specific case would be I do a filter and outputs only a