Re: [I] Unable to auto-scale Kubernetes cluster [cloudstack-kubernetes-provider]
weizhouapache commented on issue #52: URL: https://github.com/apache/cloudstack-kubernetes-provider/issues/52#issuecomment-2124110440 I notice @saffronjam used a regular user to deploy the CKS cluster, would it be related to the issue ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@cloudstack.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] Unable to auto-scale Kubernetes cluster [cloudstack-kubernetes-provider]
kiranchavala commented on issue #52:
URL:
https://github.com/apache/cloudstack-kubernetes-provider/issues/52#issuecomment-1968230660
Hi @saffronjam
The autoscaling feature works fine on a k8s cluster deployed by CKS.
Please find the steps that i have followed
After you enable autoscaling on the cluster
Make sure the autoscaling pod is deployed in the cluster
```
kubectl get pods -A
NAMESPACE NAME
READY STATUSRESTARTS AGE
kube-systemcluster-autoscaler-8d8894d6c-q8r4h 1/1
Running 0 19m
```
Before scaling
```
➜ ~ k get nodes -A
NAME STATUS ROLES AGE VERSION
gh-control-18debd77e18 Readycontrol-plane 10h v1.28.4
gh-node-18debd8440c Ready 10h v1.28.4
```
Deploy a application
`kubectl create deployment hello-node
--image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- /agnhost netexec
--http-port=80`
```
➜ ~ k get pods -A
NAMESPACE NAME
READY STATUSRESTARTS AGE
defaulthello-node-7c6c5fb9d8-bgd69 1/1
Running 0 10h
```
Scale the application
`kubectl scale --replicas=150 deployment/hello-node`
logs from the autoscaler pod
```
I0228 04:51:46.798087 1 reflector.go:536]
/home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:356:
Watch close - *v1.StatefulSet total 9 items received
I0228 04:51:51.244004 1 static_autoscaler.go:235] Starting main loop
I0228 04:51:51.244382 1 client.go:169] NewAPIRequest API request
URL:http://10.0.34.2:8080/client/api?apiKey=***=listKubernetesClusters=14b42c5d-e7e6-4c41-b638-5facb98b0a93=json=***
I0228 04:51:51.279721 1 client.go:175] NewAPIRequest response status
code:200
I0228 04:51:51.280798 1 cloudstack_manager.go:88] Got cluster :
&{14b42c5d-e7e6-4c41-b638-5facb98b0a93 gh 2 3 1 1 [0xc0013bfad0 0xc0013bfb00]
map[gh-control-18debd77e18:0xc0013bfad0 gh-node-18debd8440c:0xc0013bfb00]}
W0228 04:51:51.292009 1 clusterstate.go:590] Failed to get nodegroup
for dc95f481-15a3-4629-bb78-055fbe4a7139: Unable to find node
dc95f481-15a3-4629-bb78-055fbe4a7139 in cluster
W0228 04:51:51.292052 1 clusterstate.go:590] Failed to get nodegroup
for facdd040-53fe-4984-8654-c186a7cdde9b: Unable to find node
facdd040-53fe-4984-8654-c186a7cdde9b in cluster
I0228 04:51:51.292095 1 static_autoscaler.go:341] 2 unregistered nodes
present
I0228 04:51:51.292105 1 static_autoscaler.go:624] Removing
unregistered node dc95f481-15a3-4629-bb78-055fbe4a7139
W0228 04:51:51.292126 1 static_autoscaler.go:627] Failed to get node
group for dc95f481-15a3-4629-bb78-055fbe4a7139: Unable to find node
dc95f481-15a3-4629-bb78-055fbe4a7139 in cluster
W0228 04:51:51.292137 1 static_autoscaler.go:346] Failed to remove
unregistered nodes: Unable to find node dc95f481-15a3-4629-bb78-055fbe4a7139 in
cluster
I0228 04:51:51.292569 1 filter_out_schedulable.go:65] Filtering out
schedulables
I0228 04:51:51.292590 1 filter_out_schedulable.go:137] Filtered out 0
pods using hints
I0228 04:51:51.624523 1 filter_out_schedulable.go:175] 44 pods were
kept as unschedulable based on caching
I0228 04:51:51.624568 1 filter_out_schedulable.go:176] 0 pods marked
as unschedulable can be scheduled.
I0228 04:51:51.624667 1 filter_out_schedulable.go:87] No schedulable
pods
I0228 04:51:51.870314 1 static_autoscaler.go:480] Calculating unneeded
nodes
I0228 04:51:51.870353 1 pre_filtering_processor.go:66] Skipping
gh-control-18debd77e18 - node group min size reached
I0228 04:51:51.870361 1 pre_filtering_processor.go:66] Skipping
gh-node-18debd8440c - node group min size reached
I0228 04:51:51.870413 1 static_autoscaler.go:534] Scale down status:
unneededOnly=false lastScaleUpTime=2024-02-27 17:39:33.032517071 + UTC
m=-3594.093061754 lastScaleDownDeleteTime=2024-02-27 17:39:33.032517071 +
UTC m=-3594.093061754 lastScaleDownFailTime=2024-02-27 17:39:33.032517071 +
UTC m=-3594.093061754 scaleDownForbidden=false isDeleteInProgress=false
scaleDownInCooldown=false
I0228 04:52:22.258545 1 scale_up.go:468] Best option to resize:
14b42c5d-e7e6-4c41-b638-5facb98b0a93
I0228 04:52:22.258602 1 scale_up.go:472] Estimated 1 nodes needed in
14b42c5d-e7e6-4c41-b638-5facb98b0a93
I0228 04:52:22.266675 1 scale_up.go:595] Final scale-up plan:
