which producer type (sync/async) you are using? Async producer will improve the throughput dramatically.
On Mon, Jan 11, 2016 at 12:09 AM, Vineet Mishra <clearmido...@gmail.com> wrote: > Hi John, > > Increasing the Kafka partition should be helpful. > > Additionally what is the KafkaBolt parallelism you are keeping? > > Cheers! > VM > On Jan 10, 2016 11:33 PM, "John Yost" <hokiege...@gmail.com> wrote: > >> Hi Everyone, >> >> I am attempting to use the KafkaBolt for output from my Storm topology. >> Whereas the throughput from the KafkaSpout is screaming fast, output to >> Kafka via the KafkaBolt is comparatively really slow. I've tried outputting >> with request.required.acks set to 0,1,-1 which does not affect throughput >> significantly. >> >> I am thinking that one issue is that the KafkaSpout is reading off of a >> topic with 20 partitions, whereas the KafkaBolt is writing to a topic with >> 10 partitions. I am going to expand the write topic to twenty partitions >> tomorrow and retry as I think that increasing the producer group size via a >> greater number of partitions should help. >> >> Any other ideas/guidance would be great--thanks! >> >> --John >> > -- Regards, Abhishek Agarwal
