Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
Wei, It did work before so a routing change at our core must have messed it up. I assume the routing issue was the actual issue here. Everything else was just ancillary. Thanks for all the help, the clusters are working now! -Wally On Thu, Feb 15, 2024 at 10:07 AM Wei ZHOU wrote: > Hi, > > As

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wei ZHOU
Hi, As I understand, 1. After upgrading, you need to patch the system vms or recreate them. Not a bug I think. 2. a minor issue which does not impact the provisioning and operation of CKS cluster. 3. Looks like a network misconfiguration, but did it work before ? -Wei On Thu, 15 Feb 2024 at

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
As a quick add-on. After running those commands and getting the kubectl commands working the Error in the management log is tail -f /var/log/cloudstack/management/management-server.log | grep ERROR 2024-02-15 14:09:41,124 ERROR [c.c.k.c.a.KubernetesClusterActionWorker]

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
Thanks Wei ZHOU! That fixed the kubectl command issue but the cluster still just sits at Create Kubernetes cluster k8s-cluster-1 in progress Maybe this is just a UI issue? Unfortunately If I stop the k8s cluster after it errors out it just stays in the error state. 1. Click Stop Kubernetes

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wei ZHOU
Hi, Please run the following commands as root: mkdir -p /root/.kube cp -i /etc/kubernetes/admin.conf /root/.kube/config After then the kubectl commands should work -Wei On Thu, 15 Feb 2024 at 13:53, Wally B wrote: > What command do you suggest I run? > > kubeconfig returns command not found

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
What command do you suggest I run? kubeconfig returns command not found on your PR I see kubeadm join is being called out as well but I wanted to verify what you wanted me to test first. On Thu, Feb 15, 2024 at 2:41 AM Wei ZHOU wrote: > Hi Wally, > > I think the cluster is working fine. >

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wei ZHOU
Hi Wally, I think the cluster is working fine. The kubeconfig is missing in extra nodes. I have just created a PR for it: https://github.com/apache/cloudstack/pull/8658 You can run the command on the control nodes which should fix the problem. -Wei On Thu, 15 Feb 2024 at 09:31, Wally B wrote:

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
3 Nodes Control 1 -- No Errors kubectl get nodes NAMESTATUS ROLES AGE VERSION pz-dev-k8s-ncus-1-control-18dabdb141b Readycontrol-plane 2m6s v1.28.4 pz-dev-k8s-ncus-1-control-18dabdb6ad6 Readycontrol-plane 107s v1.28.4

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wei ZHOU
Can you try with 3 control nodes ? -Wei On Thu, 15 Feb 2024 at 09:13, Wally B wrote: > - zone type : > Core > - network type: > Advanced > Isolated Network inside a Redundant VPC (same results in just an > Isolated network without VPC) > - number of control nodes: >

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wally B
- zone type : Core - network type: Advanced Isolated Network inside a Redundant VPC (same results in just an Isolated network without VPC) - number of control nodes: 2 Control Nodes (HA Cluster) We were able to deploy k8s in the past, not sure what changed.

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-15 Thread Wei ZHOU
Hi, can you share - zone type - network type - number of control nodes -Wei On Thu, 15 Feb 2024 at 08:52, Wally B wrote: > So > > Recreating the Sec Storage VM Fixed the Cert issue and I was able to > install K8s 1.28.4 Binaries. --- THANKS Wei ZHOU ! > > > Im still getting > > [FAILED]

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Wally B
So Recreating the Sec Storage VM Fixed the Cert issue and I was able to install K8s 1.28.4 Binaries. --- THANKS Wei ZHOU ! Im still getting [FAILED] Failed to start Execute cloud user/final scripts. on 1 control and 1 worker. *Control 1 -- pz-dev-k8s-ncus-1-control-18dabaf66c1 --:*

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Wei ZHOU
Destroy ssvm and retry when new ssvm is Up ? -Wei 在 2024年2月15日星期四,Wally B 写道: > Super Weird. I have two other versions added successfully but now when I > try to add an ISO/version I get the following on the management host. This > is the first time I've tried adding a K8s version since

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Wally B
Super Weird. I have two other versions added successfully but now when I try to add an ISO/version I get the following on the management host. This is the first time I've tried adding a K8s version since 4.18.0 tail -f /var/log/cloudstack/management/management-server.log | grep ERROR 2024-02-15

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Wei ZHOU
Can you try 1.27.8 or 1.28.4 on https://download.cloudstack.org/cks/ ? -Wei 在 2024年2月15日星期四,Wally B 写道: > Hello Everyone! > > We are currently attempting to deploy k8s clusters and are running into > issues with the deployment. > > > Current CS Environment: > > CloudStack Verison: 4.19.0

Re: Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Kiran Chavala
9m12s 3. Please provide the following logs to investigate which step its failing cat /var/log/daemon.log cat /var/log/messages Regards Kiran From: Wally B Date: Thursday, 15 February 2024 at 4:35 AM To: users@cloudstack.apache.org Subject: Kubernetes Clusters Failing to Start 4.19.0

Kubernetes Clusters Failing to Start 4.19.0

2024-02-14 Thread Wally B
Hello Everyone! We are currently attempting to deploy k8s clusters and are running into issues with the deployment. Current CS Environment: CloudStack Verison: 4.19.0 (Same issue before we upgraded from 4.18.1). Hypervisor Type: Ubuntu 20.04.03 KVM Attempted K8s Bins: 1.23.3, 1.27.3