Github user roshannaik commented on the issue: https://github.com/apache/storm/pull/2241 @HeartSaVioR the current master clocks little over 3 mill/sec for the ConstSpoutNullBolt Topo without ACKing. On the latency end, with ACK enabled i saw numbers in multiple *seconds* with batchSz=1 for master and touching as low ~30 microseconds with the new patch. More analysis? I thought I might have spent too much time providing measurements of various kinds, both in the design docs and here. :-) Actually I have only presented a few pruned down numbers that I felt were interesting, in which case I was a bit meticulous to record the numbers. Given the scope of this patch, meticulously recording and cataloging all the perf tests that I have done (both high level profiler data & low level hardware perf counters) and performing the many more tests that I could have done, will take forever. I felt, ROI was -ve to rigorously pursue some of those measurements any further. If you feel some other critical measurements are necessary, bring it up. Batching is a small aspect of this patch. I had done some quick tests to determine a default that appears to be a modest sweetspot for high throughput low latency topos, but not enough to provide a rigorous analysis. As batch size is very dependent on topology components, deployment size and ack-mode, IMO the search for a more generally optimal batch size is not important right now. I think we will have more publish worthy analysis once these changes stabilize and get more testing. On a related note, about latency and ACK-mode, during my recent measurements, I noticed an issue in ACK mode, both in current master and optimized version, perf degrades with time (for certain batch sizes)... Something I had not noticed on the old version of master on which I had based my changes for a long time. I will share these observations soon. This needs to be looked into before we have an official release.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---