Thanks Srikanth. I played with 1-2 tasks per executor, it helps a bit but not by much. I need to fieldshuffle, so parallelism won't solve my problem.
I profiled the python bolt, and it takes order of 200ms per tuple, which is in line with the execute latency. But process latency is in 20s of minutes! Played with worker and executor buffer sizes but it didn't help either. It seems that tuples are waiting somewhere in a queue but I'm not able to reduce that wait. Really low max spout pending does help but anything above 256 kills the performance. One more data point: I have a large json encoded object(str) in my tuple. Thinking of dumping it in a db and referencing it in the tuple. Open to suggestions. Thanks. Sid On Apr 27, 2015 4:10 PM, "Srikanth" <srikanth...@gmail.com> wrote: > Why have you configured 32 tasks on 36 executors? Set noof task to at > least 36. > Looks like your Python bolt takes some time to process a tuple. You may > need to tune that or give it more threads. > If you are not maxing out on resource usage, set noof task to 72 and see > if that helps. > > Srikanth > > On Fri, Apr 24, 2015 at 2:16 PM, Sid <jack8danie...@gmail.com> wrote: > >> My storm topology has python bolts using multilang support. >> >> Kafka_spout(java) -> format_filter_bolt -> process_bolt >> >> My storm cluster has 3 32 core EC2 instances. format_filter_bolt has 1 >> executor and 1 task, process_bolt has 36 executor and 32 tasks. I have max >> spout pending = 250. >> >> I observe that format_filter_bolt has a execute latency of 0.018 and >> process latency of 7.585, while process_bolt has an execute latency of >> 0.008 and process latency 94664.242! >> >> These numbers look strange already. Comparing the two bolts, one with >> lower execute latency and handling fewer tuples has magnitude times larger >> process latency. >> >> Looking for suggestions on what I can do improve the latencies without >> throwing more computation at it. Thanks >> > >
