Re: Kafka Producer Performance

2016-08-02 Thread Sergey Zhemzhitsky
Hi Daniel, Thanks a lot for the suggestion! I'll give it a try. Now if with batch consumers everything is pretty clear, with event-driven consumers - it is not. For example, amqp and jms implementations can prefetch a number of messages, but cannot batch those ones out of the box. So to make came

Re: Kafka Producer Performance

2016-08-01 Thread Daniel Kulp
For one of my clients, I ended up not using the splitter in Camel and instead us a custom processor that would create an Iterator. This will work with updates to camel-kafka that are included in 2.17.3. For my tests, using the camel splitter like you have would get about 5K-10K msg/sec. With

Kafka Producer Performance

2016-08-01 Thread Sergey Zhemzhitsky
Hi Camel Gurus, I've faced with some performance issues of camel-kafka component during migrating it from 2.17.0 then to 2.17.1 and then to 2.17.2. The camel route is pretty simple and looks like this from("file:/var/lib/app/input") .split().simple("\n").streaming() .to("direct:kafka