Sadly storm is a very talky so reducing the amount of network traffic is going to be difficult. I don't think anyone has really tried this before so you are going to be in new territory.
There are a few things you might want to look at. 1) all of the different daemons in storm tend to poll periodically for updates. Almost all of the polling intervals are configurable so you would want to drop them down. They probably are not making that much network traffic, but it could be an easy win. 2) acking does use up a lot of messages. They are not large, but it is still a lot of messages. If you can live with acking off it should reduce your network load a lot. 3) Try to use the localOrShuffle grouping where ever possible, or even better just combine your bolts together into a bigger bolts and avoid the transfer all together. LocalOrShuffle is a kind a brittle feature when scheduling does not go perfectly your way and we are working to make it much better through https://issues.apache.org/jira/browse/STORM-2684 but that code is not done yet. Beyond those I'm not sure what else you would want to do. You would need to do some profiling to see what is taking up the memory, network, and CPU. - Bobby On Wed, Sep 13, 2017 at 10:36 PM Gayashan Amarasinghe < gayashan.amarasin...@gmail.com> wrote: > Hi Bobby, > > Thank you very much for the details. I'm not fixed on a specific version > yet, would prefer the latest. Is it possible to run with minimum network > usage, apart from receiving and delivering tuples, such as shutting down > ackers, running with no supervision, pros and cons of that, etc. Do you > have any study or details on that or can you point me to where I should > look at to get some idea? > > Thank you. > > Best regards, > /Gayashan > > On Tue, Sep 12, 2017 at 12:26 AM, Bobby Evans <ev...@oath.com.invalid> > wrote: > > > A lot of that depends on the version of storm you have, and it is not a > use > > case that we have played around with too much. > > > > Just looking at a few of our production clusters the CPU usage is very > > small (but trying to determine what the CPU usage on a raspberry pi arm > > chip when all you have is the usage on a server CPU is hard to do). The > > memory usage, however can be somewhat high, especially for a raspberry > pi. > > We are setting the heap size for the supervisor on our clusters to 256MB, > > but the heap size on the logviewer is 768MB. But we have not tuned these > > at all, so it could easily be a lot smaller and still work. > > > > - Bobby > > > > > > > > On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe < > > gayashan.amarasin...@gmail.com> wrote: > > > > > Hi devs, > > > > > > What are the bare minimum requirements to run a Storm worker on a low > > > resource environment such as a raspberry pi node where bandwidth > > > consumption and power consumption could be critical? > > > > > > Thank you. > > > > > > Best regards, > > > /Gayashan > > > > > >
