GitHub user ahmadamirahmadi1401 added a comment to the discussion: Upgrading
K8s Cluster Failed
Yes, but I did not make any changes to the file and this is the default
behavior when upgrading.
This happens for other versions as well. For example, the following error was
obtained for upgrading a Kubernetes cluster from version 1.30.5 to version
1.31.8. And in the end, it failed to unmount the ISO and got an alert.
2025-06-03 18:40:17,001 ERROR [c.c.u.s.SshHelper]
(API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01) SSH
execution of command sudo ./upgrade-kubernetes.sh13249113815295454536.sh 1.31.8
true false false has an error status code in return. Result output: Installing
binaries from /mnt/k8sdisk/
unpacking quay.io/apalia/cloudstack-csi-driver:0.0.2
(sha256:6f38051b27964da06af1cbcf1c759b4eb5266bdd02033f07c3d2e04630e2893f)...done
unpacking ghcr.io/leaseweb/cloudstack-csi-driver:0.8.1
(sha256:7a6cf3ba95be182ee8c991dced373f56faacbfac9a719056586d98af3aa932c5)...done
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest
(sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.1.0
(sha256:10c058968e7d2f8e55da5976336a42d88b057e75880f000d3120aa2d75649e97)...done
unpacking registry.k8s.io/coredns/coredns:v1.11.3
(sha256:9caabbf6238b189a65d0d6e6ac138de60d6a1c419e5a341fbbb7c78382559c6e)...done
unpacking k8s.gcr.io/sig-storage/csi-attacher:v3.0.2
(sha256:6f80b12657a7e0a5c683b24e806c4bbbe33a43e39b041fe9b7514d665d478ea4)...done
unpacking registry.k8s.io/sig-storage/csi-attacher:v4.6.1
(sha256:b4d611100ece2f9bc980d1cb19c2285b8868da261e3b1ee8f45448ab5512ab94)...done
unpacking k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1
(sha256:e07f914c32f0505e4c470a62a40ee43f84cbf8dc46ff861f31b14457ccbad108)...done
unpacking registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
(sha256:f25af73ee708ff9c82595ae99493cdef9295bd96953366cddf36305f82555dac)...done
unpacking k8s.gcr.io/sig-storage/csi-provisioner:v2.0.4
(sha256:bec571992d40203edcd056ac0b0d97003887ee5e4be144c41932d18639673b03)...done
unpacking registry.k8s.io/sig-storage/csi-provisioner:v5.0.1
(sha256:405a14e1aa702f7ea133cea459e8395fe40a6125c088c55569e696d48e1bd385)...done
unpacking registry.k8s.io/sig-storage/csi-resizer:v1.11.1
(sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0
(sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.15-0
(sha256:a6dc63e6e8cfa0307d7851762fa6b629afb18f28d8aa3fab5a6e91b4af60026a)...done
unpacking registry.k8s.io/kube-apiserver:v1.31.8
(sha256:30090db6a7d53799163ce82dae9e8ddb645fd47db93f2ec9da0cc787fd825625)...done
unpacking registry.k8s.io/kube-controller-manager:v1.31.8
(sha256:29eaddc64792a689df48506e78bbc641d063ac8bb92d2e66ae2ad05977420747)...done
unpacking registry.k8s.io/kube-proxy:v1.31.8
(sha256:dd0c9a37670f209947b1ed880f06a2e93e1d41da78c037f52f94b13858769838)...done
unpacking registry.k8s.io/kube-scheduler:v1.31.8
(sha256:22994a2632e81059720480b9f6bdeb133b08d58492d0b36dfd6e9768b159b22a)...done
unpacking registry.k8s.io/sig-storage/livenessprobe:v2.12.0
(sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8
(sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.10
(sha256:ee6521f290b2168b6e0935a181d4cff9be1ac3f505666ef0e3c98fae8199917a)...done
unpacking registry.mainx.com/weaveworks/weave-kube:latest
(sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking registry.mainx.com/weaveworks/weave-npc:latest
(sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n
kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n
kube-system get cm kubeadm-config -o yaml'
[upgrade] Running cluster health checks
====================
2025-06-03 18:43:04,106 ERROR [c.c.k.c.a.KubernetesClusterActionWorker]
(API-Job-Executor-34:ctx-a226f7b8 job-32868 ctx-5a7f130c) (logid:f244ee01)
Failed to upgrade Kubernetes cluster :
e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade
Kubernetes node on VM :
e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
com.cloud.utils.exception.CloudRuntimeException: Failed to upgrade Kubernetes
cluster : e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable
to upgrade Kubernetes node on VM :
e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.logTransitStateDetachIsoAndThrow(KubernetesClusterActionWorker.java:266)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeKubernetesClusterNodes(KubernetesClusterUpgradeWorker.java:127)
at
com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterUpgradeWorker.upgradeCluster(KubernetesClusterUpgradeWorker.java:177)
at
com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.upgradeKubernetesCluster(KubernetesClusterManagerImpl.java:1622)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy466.upgradeKubernetesCluster(Unknown Source)
at
org.apache.cloudstack.api.command.user.kubernetes.cluster.UpgradeKubernetesClusterCmd.execute(UpgradeKubernetesClusterCmd.java:114)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
=============================
2025-06-03 18:43:04,685 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-34:ctx-a226f7b8 job-32868) (logid:f244ee01) Complete async
job-32868, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to upgrade Kubernetes cluster :
e1faffb3e614e6c2fba74296962386b7-01jwvgydkaj5jket7z5qmqnsfe, unable to upgrade
Kubernetes node on VM :
e1faffb3e614e6c2fba74296962386b7-01jwvgy-control-197370ff089"}
@weizhouapache
GitHub link:
https://github.com/apache/cloudstack/discussions/11006#discussioncomment-13434094
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
