+1 Scaling up & down can also take some time (multiple days even) if the nodes in the cluster hold a lot of data. The process has a performance impact on the cluster, as the rest of the cluster must stream data to the new nodes, or stream data away from the decommissioning nodes, also usually leading to increased compaction activity. When adding/removing multiple nodes in parallel this performance impact is increased. If the cluster is already under very heavy load, or has other performance-related issues, then an ASG automatically adding multiple nodes could be enough to bring your application down or make it unusable. It's usually also a good idea to run a repair afterwards.
You'd also need to handle the case where the ASG decided to remove nodes that were still joining the cluster. Definitely not impossible, but not an easy task. On Tue, 17 Oct 2017 at 16:40 Jeff Jirsa <jji...@gmail.com> wrote: > Just to expand on this: > > In an asg environment, you could have an autoscaling event to expand or > shrink the cluster and multiple nodes terminate at the same time. Your > userdata or your AMI would have to know how to find the cluster, know how > many instances were down and what the target size was, check to see which > of those were down from app level issues versus down from instance level > issues, vs not joined the cluster yet, figure out which of the down > instances it is meant to replace, and then rejoin the cluster before > another replica fails. And even without down instances, simultaneous > bootstrap and decom has serious implications for consistency guarantees. > > It’s not impossible, but it’s far from trivial, and figuring out how to > grab an appropriate EIP is about the easiest part of it. If you remove the > scale in/our component it gets a lot easier, but then you likely don’t > really need an ASG at all. > > > > -- > Jeff Jirsa > > > On Oct 16, 2017, at 10:06 PM, Justin Cameron <jus...@instaclustr.com> > wrote: > > I wouldn't recommend putting your Cassandra nodes in an ASG. Scaling > production Cassandra clusters can be a tricky process that should be done > manually. There are a lot of things that can go wrong if you aren't > watching/controlling the process. > > Terraform is a good option for automating complex AWS deployments. It also > allows you to scale your cluster incrementally once it's been deployed. > CloudFormation is another option. There are also fully managed services > that will take care of it all for you, such as Instaclustr (full disclosure > - I work for Instaclustr). > > > On Tue, 17 Oct 2017 at 15:45 cass savy <casss...@gmail.com> wrote: > >> How can we solve EIP allocation using APIs for multiple nodes that are >> in one ASG? What are the config management tools that you are referring to? >> >> On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves <k...@instaclustr.com> >> wrote: >> >>> AWS API's provide the functionality to allocate and associate elastic >>> IPs to instances. Generally the API's aren't pretty but they work. What >>> issues are you having? If it's a configuration problem there are a variety >>> of config management tools that you can use to populate the yaml/env files >>> with the correct IP addresses. >>> >> >> -- > > > *Justin Cameron*Senior Software Engineer > > > <https://www.instaclustr.com/> > > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > -- *Justin Cameron*Senior Software Engineer <https://www.instaclustr.com/> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA). This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
