mw-0 opened a new issue, #14244:
URL: https://github.com/apache/cloudstack/issues/14244
### problem
Since 4.23, create-kubernetes-binaries-iso.sh packages the Headlamp
dashboard as headlamp.yaml
instead of dashboard.yaml. The cluster create path (k8s-control-node.yml)
handles both files,
but upgrade-kubernetes.sh still hardcodes:
/opt/bin/kubectl apply -f ${BINARIES_DIR}/dashboard.yaml
The script runs with bash -e, so on the first control node it exits non-zero
after kubeadm
upgrade apply and the kubelet swap have already succeeded. The worker retries
(cloud.kubernetes.cluster.upgrade.retries) and then fails the job, leaving
the control node
on the new version but cordoned, the workers on the old version, the cluster
in Alert,
and the DB still recording the old version.
Management server log:
SSH execution of command sudo ./upgrade-kubernetes.sh<rand>.sh 1.36.0 true
false false true
has an error status code in return. Result output: Installing binaries
from /mnt/k8sdisk/
Failed to upgrade Kubernetes cluster : single, unable to upgrade
Kubernetes node on VM : single-control-...
Running the script by hand on the control node shows:
error: the path "/mnt/k8sdisk//dashboard.yaml" does not exist
ISO contents:
10-kubeadm.conf autoscaler.yaml cni cri-tools docker headlamp.yaml k8s
kubelet.service
manifest.yaml network.yaml provider.yaml snapshot-crds.yaml
### versions
ACS 4.23.0.0 (also present on main at 0a5bf30)
Kubernetes upgrade 1.35.8 -> 1.36.0, ISO built with the 4.23
create-kubernetes-binaries-iso.sh
Hypervisor: kvm Network: isolated..
### The steps to reproduce the bug
1. Build and register a Kubernetes version ISO with the 4.23
create-kubernetes-binaries-iso.sh (contains headlamp.yaml)
2. Deploy a CKS cluster on the previous minor version
3. Call upgradeKubernetesCluster to the new version
4. Job fails on the first control node; kubectl get nodes shows control on
new version, SchedulingDisabled
### What to do about it?
Make upgrade-kubernetes.sh use the same fallback as k8s-control-node.yml:
apply headlamp.yaml
if present, else dashboard.yaml, else warn. I'll open a PR. Workaround: patch
script/upgrade-kubernetes.sh inside the management server jar and restart.
I already have tested a fix and will put in a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
