I deploy with LocalCluster mode. I get several thousands messages processed a second but expect my botteneck is not at the storm layer.
It would be interesting to see a full explanation of the differences in LocalCluster and a real deployment. I expect quite a few people would benefit from having storm run in process in actual deployments. -Rits On Fri, Jan 10, 2014 at 9:46 AM, Michael Rose <mich...@fullcontact.com>wrote: > Post your code. Even Dev mode is far faster for us. > On Jan 10, 2014 8:44 AM, "Klausen Schaefersinho" < > klaus.schaef...@gmail.com> wrote: > >> > I've benched storm at 1.8 million tuples per second on a big (24 core) >> box using local or shuffle grouping between a spout and bolt. >> Production or development mode? >> >> > you're only seeing 10 events per second make sure you don't have any >> sleeps >> Yeah I checked for sleeps etc. and stripped down my code. Now my bolts >> do nothing and the spout just creates random data... >> >> >> >> >> On Fri, Jan 10, 2014 at 4:37 PM, Nathan Leung <ncle...@gmail.com> wrote: >> >>> I've benched storm at 1.8 million tuples per second on a big (24 core) >>> box using local or shuffle grouping between a spout and bolt. If you're >>> only seeing 10 events per second make sure you don't have any sleeps >>> (whether in your code or elsewhere e.g. a library or triggered due to lack >>> of data in the storm spout). Also note that the default sleep period in the >>> spout when there is no data (in storm 0.9) is I believe 1ms so even if >>> you're hitting this condition you should see much more than 10 events per >>> second. >>> On Jan 10, 2014 10:19 AM, "Jon Logan" <jmlo...@buffalo.edu> wrote: >>> >>>> Why would you benchmark local mode? It's intended for debugging and >>>> development purposes, not actual production use............... >>>> >>>> >>>> per the website, Storm itself is benchmarked at > 1 million tuples per >>>> second per node. >>>> >>>> >>>> On Fri, Jan 10, 2014 at 8:40 AM, Klausen Schaefersinho < >>>> klaus.schaef...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> how is the performance of a real cluster compared to a local >>>>> development cluster? I was trying to benchmark Storm now and found that >>>>> it >>>>> only manages to consume 10 events per second in a simple topology which >>>>> only consists out of i spout and one bolt where the spout creates random >>>>> objects and the bolt acknowledge the tuple. >>>>> >>>>> 10 events terrible slow, and I am not sure if this is related to the >>>>> develop mode. >>>>> >>>>> Cheers, >>>>> >>>>> Klaus >>>>> >>>> >>>> >>
