Re: Kafka Streams multi-node

2016-07-26 Thread Matthias J. Sax
See inline. -Matthias On 07/26/2016 11:10 PM, Davood Rafiei wrote: > Thanks David and Matthias for reply. To make sure that I understand > correctly: > - Each stream application is limited to only one node. In that node all > stream execution DAG is processed. An application is not limited to a

Re: Kafka Streams multi-node

2016-07-26 Thread Davood Rafiei
Thanks David and Matthias for reply. To make sure that I understand correctly: - Each stream application is limited to only one node. In that node all stream execution DAG is processed. - If we want to parallelize our application, we start new instance of streams application, which will be similar

Re: Kafka Streams multi-node

2016-07-26 Thread Matthias J. Sax
David's answer is correct. Just start the same application multiple times on different nodes and the library does the rest for you. Just one addition: as Kafka Streams is for standard application development, there is no need to run the application on the same nodes as your brokers are running (ie

Re: Kafka Streams multi-node

2016-07-26 Thread David Garcia
http://docs.confluent.io/3.0.0/streams/architecture.html#parallelism-model you shouldn’t have to do anything. Simply starting a new thread will “rebalance” your streaming job. The job coordinates with tasks through kafka itself. On 7/26/16, 12:42 PM, "Davood Rafiei" wrote: Hi,

Kafka Streams multi-node

2016-07-26 Thread Davood Rafiei
Hi, I am newbie in Kafka and Kafka-Streams. I read documentation to get information how it works in multi-node environment. As a result I want to run streams library on cluster that consists of more than one node. >From what I understood, I try to resolve the following conflicts: - Streams is a st