For scaling the storm setup rebalance is already there but there are some issues with it to run in production.
1 - How to know the amount of resource required for scaling up or down. There is already a lot of research done in that context. 2 - Another is messages are lost on rebalancing a running DAG (if you are not using ACKing). And with Acking overhead of message re-processing is very high. On Fri, Jul 14, 2017 at 2:24 AM, Shashank Prasad <[email protected]> wrote: > Hi Folks, > > We have been using storm for sometime now and it servers our purpose very > well. The nature of our business is such that, we process high volume of > data during only certain hours of the day, the rest of the time it receives > very low volume of data and sometimes it completely sits idle, receiving no > data at all. Since our infrastructure is on AWS, we are paying by the > hour. > > Ideally what i would like is automatic scaling and downsizing of storm > supervisors depending on the traffic it receives. I am wondering if anyone > of you guys have experienced something similar and have implemented an > auto-scaling de-scaling method. If so, how are you going about it? what > applications are you using to achieve it? > > Thank you for you time! > > -shashank > -- Thanks & Regards, Anshu Shukla
