Note: be careful if you topology has state, Storm topologies may be stateful in which case autoscaling (up or down) could lead to data inconsistencies
You can perform this using Hortonworks Cloudbreak to perform auto scaling based on configurable thresholds; you can also try to put something together using custom deployment scripts and auto scaling groups. This is also doable in dockerized environments (compose/swarm or kubernetes) with careful configuration and image building. On Jul 13, 2017 3:10 PM, "Shashank Prasad" <[email protected]> wrote: > Hi Folks, > > We have been using storm for sometime now and it servers our purpose very > well. The nature of our business is such that, we process high volume of > data during only certain hours of the day, the rest of the time it receives > very low volume of data and sometimes it completely sits idle, receiving no > data at all. Since our infrastructure is on AWS, we are paying by the > hour. > > Ideally what i would like is automatic scaling and downsizing of storm > supervisors depending on the traffic it receives. I am wondering if anyone > of you guys have experienced something similar and have implemented an > auto-scaling de-scaling method. If so, how are you going about it? what > applications are you using to achieve it? > > Thank you for you time! > > -shashank >
