Re: Optimizing Kafka Stream

2014-06-02 Thread Chi Hoang
in the topology. For example if I said .parallelismHint(4), then I would have 4 different partitions. Is this not the case ? Is there a set number of partitions my topology has that I need to increase in order to have higher parallelism ? Thanks On Sat, May 31, 2014 at 11:50 AM, Chi Hoang c

Re: Optimizing Kafka Stream

2014-05-31 Thread Chi Hoang
Raphael, You can try tuning your parallelism (and num workers). For Kafka 0.7, your spout parallelism could max out at: # brokers x # partitions (for the topic). If you have 4 Kafka brokers, and your topic has 5 partitions, then you could set the spout parallelism to 20 to maximize the