Awesome, thanks Abhishek! Yeah, was wondering if async would help, will try that.
--John On Mon, Jan 11, 2016 at 12:51 AM, Abhishek Agarwal <abhishc...@gmail.com> wrote: > which producer type (sync/async) you are using? Async producer will > improve the throughput dramatically. > > On Mon, Jan 11, 2016 at 12:09 AM, Vineet Mishra <clearmido...@gmail.com> > wrote: > >> Hi John, >> >> Increasing the Kafka partition should be helpful. >> >> Additionally what is the KafkaBolt parallelism you are keeping? >> >> Cheers! >> VM >> On Jan 10, 2016 11:33 PM, "John Yost" <hokiege...@gmail.com> wrote: >> >>> Hi Everyone, >>> >>> I am attempting to use the KafkaBolt for output from my Storm topology. >>> Whereas the throughput from the KafkaSpout is screaming fast, output to >>> Kafka via the KafkaBolt is comparatively really slow. I've tried outputting >>> with request.required.acks set to 0,1,-1 which does not affect throughput >>> significantly. >>> >>> I am thinking that one issue is that the KafkaSpout is reading off of a >>> topic with 20 partitions, whereas the KafkaBolt is writing to a topic with >>> 10 partitions. I am going to expand the write topic to twenty partitions >>> tomorrow and retry as I think that increasing the producer group size via a >>> greater number of partitions should help. >>> >>> Any other ideas/guidance would be great--thanks! >>> >>> --John >>> >> > > > -- > Regards, > Abhishek Agarwal > >
