Re: Kafka streams: Getting a state store associated to a processor

2017-02-14 Thread Michael Noll
> By the way - do I understand correctly that when a state store is persistent, it is logged by default? Yes. > So enableLogging(Map) only is a way to provide default configuration to the default logging? Yes. That is, any configs that should be applied to the state store's changelog topic. >

Re: Kafka streams: Getting a state store associated to a processor

2017-02-14 Thread Adam Warski
Hello, > Can you try this out with 0.10.2 branch or current trunk? With 0.10.2 this works fine! The state-store changelog is created with 3 partitions if the source topic has 3 partitions. I checked with client from https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.1

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Michael Noll
Adam, also a FYI: The upcoming 0.10.2 version of the Streams API will be backwards compatible with 0.10.1 clusters, so you can keep your brokers on 0.10.1.1 and still use the latest Streams API version (including the one from trunk, as Matthias mentioned). -Michael On Mon, Feb 13, 2017 at 1:04

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Matthias J. Sax
Can you try this out with 0.10.2 branch or current trunk? We put some fixed like you suggested already. Would be nice to get feedback if those fixed resolve the issue for you. Some more comments inline. -Matthias On 2/13/17 12:27 PM, Adam Warski wrote: > Following this answer, I checked that th

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Adam Warski
Following this answer, I checked that the auto-created "app1-store1-changelog” topic had 1 partition - which caused the problem. Creating this topic upfront with 3 partitions (which matches the stream source partition count) fixes the problem. However, I think this should be handled somehow diff

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Adam Warski
> If you increase the number of partitions in the topic "topic1" after the > state store is created, you'd need to manually increase the number of > partitions in the "app1-store1-changelog" topic as well. Or remove the > topic and let KS recreate it next run. But, either way, hopefully you > do

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Adam Warski
> If you increase the number of partitions in the topic "topic1" after the > state store is created, you'd need to manually increase the number of > partitions in the "app1-store1-changelog" topic as well. Or remove the > topic and let KS recreate it next run. But, either way, hopefully you > do

Re: Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Mathieu Fenniak
Hi Adam, If you increase the number of partitions in the topic "topic1" after the state store is created, you'd need to manually increase the number of partitions in the "app1-store1-changelog" topic as well. Or remove the topic and let KS recreate it next run. But, either way, hopefully you don

Kafka streams: Getting a state store associated to a processor

2017-02-13 Thread Adam Warski
Hello, I have a simple example (or so it would seem) of a stream processor which uses a persistent state store. Testing on one local Kafka (0.10.1.1) node, this starts up without problems for a topic with 1 partition. However, if I create a topic with 3 partitions I’m getting the following exce