Hi Nikos, My topic have 6 partition so my kafka-spouts parallelism hint also 6.
My worker is 10 KafkaSpout executor is 6 Only with Kafka-spout without bolt I got 1,00,000/s but excepted 10,00,000/s Regards, -Anandh Kumar On Mon, Apr 18, 2016 at 8:53 PM, Nikos R. Katsipoulakis < nick.kat...@gmail.com> wrote: > Hello again, > > I am not using kafka-spouts and I did not run any benchmarks on my setup. > Can I ask what is your parallelism hint on the kafka-spouts? Because, it > might be the case that 7000 t/s might be the ceiling for your parallelism > hint and you may have to increase that in order to achieve higher > throughput. > > Cheers, > Nikos > > On Mon, Apr 18, 2016 at 1:56 AM, Anandh Kumar <anandhz...@gmail.com> > wrote: > >> Nikos, >> >> Thanks for your reply. >> >> I tried many value from low to high of max.spout.pending but not getting >> throughput more than 7000/s. >> >> I tried turning internal buffer but no improvement. >> >> Did you tried any benchmark test on storm kafkaspout ? how much you got? >> given the code so that I can run same test in my environment. >> >> Regards, >> -Anandh Kumar >> >> On Sat, Apr 16, 2016 at 8:44 PM, Nikos R. Katsipoulakis < >> nick.kat...@gmail.com> wrote: >> >>> Hello Anandh, >>> >>> Why do you keep the max.spout.pending this low? I suggest increasing it >>> to a higher value, because this might be the cause of your low throughput. >>> In fact, it makes sense if you consider the following scenario: >>> >>> The KafkaSpout sends out 250 messages that try to fill out the Netty >>> buffers before they are sent to the next node. When those are send, the >>> Spout stops sending any additional tuples, until the ones "on-flight" are >>> fully ACKed. Meanwhile, the Netty buffers are not full, therefore some time >>> is spent up until they are sent out. Therefore, your throughput is >>> implicitly "throttled" >>> >>> If I were you, I would keep the default values on the buffers (or even >>> increase them a little - common practice for high throughput) and also try >>> with a bigger value on max.spout.pending. >>> >>> Cheers, >>> Nikos >>> >>> On Sat, Apr 16, 2016 at 11:06 AM, John Yost <hokiege...@gmail.com> >>> wrote: >>> >>>> Hi Anandh, >>>> >>>> I recommend profiling your workers with something like jvisualvm to see >>>> where your Bolt executors are spending their time. >>>> >>>> --John >>>> >>>> On Sat, Apr 16, 2016 at 10:01 AM, Anandh Kumar <anandhz...@gmail.com> >>>> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I benchmark the storm 0.10.0 with KafkaSpout and simple bolt not >>>>> getting throughput more than 7000 per seconds. >>>>> >>>>> Right now getting following throughput. >>>>> >>>>> Ack thread: 4 >>>>> max.spout.pending: 250 >>>>> >>>>> With KafkaSpout with simple bolt >>>>> >>>>> Throughput - 7000/s message >>>>> >>>>> ====================================================== >>>>> >>>>> Ack thread: 4 >>>>> max.spout.pending: 250 >>>>> >>>>> With KafkaSpout with* no bolt * >>>>> >>>>> Throughput - 100000/s message >>>>> >>>>> ==================================================== >>>>> >>>>> Why with bolt I am getting very less throughput? >>>>> >>>>> Not getting thoughput more than 7000/s. Tried to tune all the buffer >>>>> parameters( send,receive buffer) not use. >>>>> >>>>> Please help me. >>>>> >>>>> Regards, >>>>> -Anandh Kumar >>>>> >>>> >>>> >>> >>> >>> -- >>> Nikos R. Katsipoulakis, >>> Department of Computer Science >>> University of Pittsburgh >>> >> >> > > > -- > Nikos R. Katsipoulakis, > Department of Computer Science > University of Pittsburgh >
