Hi, A couple of things to check for why this isn't "just working": - are you passing `--balance-similar-node-groups` flag to Cluster Autoscaler? - are the groups identical? In particular, please verify if the nodes have the same set of labels (except for host name and zone) and taints. Failure domain tags you mentioned may be preventing the balancing if they are Kubernetes node labels (instance labels or annotations should be OK.)
> But > I think as a workarround (or might even be better for some scenarios), you can > just have one ASG (and on IG in kops, that matches that very same ASG). This setup isn't supported and may cause problems in scale-up if you have any zonal resources (e.g. PersistentVolume.) Regards, Aleksandra On Wed, May 9, 2018 at 9:16 PM, Rodrigo Campos <rodr...@sdfg.com.ar> wrote: > On Wed, May 09, 2018 at 07:35:23AM -0700, a...@unbotify.com wrote: > > Hi all, > > > > I'm building a K8s cluster with KOps. With 3 IGs, one in each AZ. Nodes > are automatically tagged with failure-domain tags - e.g.``` > failure-domain.beta.kubernetes.io/region=us-east-1,failure-domain.beta. > kubernetes.io/zone=us-east-1a``` and so for each AZ. > > I'v added the cluster-autoscaler with auto discovery based on: > > https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/ > cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml > > > > then created a few hundreds of dummy containers just to see it scale > out, but once it started scaling up the underlying ASG, it was completely > uneven (one ASG went to maximum and one other was added a couple of nodes, > while the third did nothing). Pods were distributed accordingly (this is a > new cluster so nothing else runs there). > > I haven't used the cluster-autoscaler, so sorry if this doesn't really > help. But > I think as a workarround (or might even be better for some scenarios), you > can > just have one ASG (and on IG in kops, that matches that very same ASG). > > But, the trick is to make that ASG/IG have several subnets, one in every > AZ you > want. Then, the ASG (is documented by aws, not just by chance) will spread > the > nodes it creates in the different AZs. And it will do it evenly. And if > one AZ > fails, then it will move the nodes to another. And then if it recovers, it > will > move the nodes again so it's even. > > We are using that ASG feature (with kops) and it works fine. > > And I guess that if you have only one ASG, cluster-autoscaler will only > increase > the nodes numbers and AWS will do the spreding things as capacity is > available > and that stuff. > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
