Hi John, Increasing the Kafka partition should be helpful.
Additionally what is the KafkaBolt parallelism you are keeping? Cheers! VM On Jan 10, 2016 11:33 PM, "John Yost" <hokiege...@gmail.com> wrote: > Hi Everyone, > > I am attempting to use the KafkaBolt for output from my Storm topology. > Whereas the throughput from the KafkaSpout is screaming fast, output to > Kafka via the KafkaBolt is comparatively really slow. I've tried outputting > with request.required.acks set to 0,1,-1 which does not affect throughput > significantly. > > I am thinking that one issue is that the KafkaSpout is reading off of a > topic with 20 partitions, whereas the KafkaBolt is writing to a topic with > 10 partitions. I am going to expand the write topic to twenty partitions > tomorrow and retry as I think that increasing the producer group size via a > greater number of partitions should help. > > Any other ideas/guidance would be great--thanks! > > --John >
