3 Nodes
Control 1 -- No Errors
kubectl get nodes
NAME STATUS ROLES AGE
VERSION
pz-dev-k8s-ncus-00001-control-18dabdb141b Ready control-plane 2m6s
v1.28.4
pz-dev-k8s-ncus-00001-control-18dabdb6ad6 Ready control-plane 107s
v1.28.4
pz-dev-k8s-ncus-00001-control-18dabdbc0a8 Ready control-plane 108s
v1.28.4
pz-dev-k8s-ncus-00001-node-18dabdc1644 Ready <none> 115s
v1.28.4
pz-dev-k8s-ncus-00001-node-18dabdc6c16 Ready <none> 115s
v1.28.4
kubectl get pods --all-namespaces
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system coredns-5dd5756b68-g84vk
1/1 Running 0 2m46s
kube-system coredns-5dd5756b68-kf92x
1/1 Running 0 2m46s
kube-system etcd-pz-dev-k8s-ncus-00001-control-18dabdb141b
1/1 Running 0 2m50s
kube-system etcd-pz-dev-k8s-ncus-00001-control-18dabdb6ad6
1/1 Running 0 2m16s
kube-system etcd-pz-dev-k8s-ncus-00001-control-18dabdbc0a8
1/1 Running 0 2m37s
kube-system
kube-apiserver-pz-dev-k8s-ncus-00001-control-18dabdb141b 1/1
Running 0 2m52s
kube-system
kube-apiserver-pz-dev-k8s-ncus-00001-control-18dabdb6ad6 1/1
Running 1 (2m16s ago) 2m15s
kube-system
kube-apiserver-pz-dev-k8s-ncus-00001-control-18dabdbc0a8 1/1
Running 0 2m37s
kube-system
kube-controller-manager-pz-dev-k8s-ncus-00001-control-18dabdb141b 1/1
Running 1 (2m25s ago) 2m51s
kube-system
kube-controller-manager-pz-dev-k8s-ncus-00001-control-18dabdb6ad6 1/1
Running 0 2m18s
kube-system
kube-controller-manager-pz-dev-k8s-ncus-00001-control-18dabdbc0a8 1/1
Running 0 2m37s
kube-system kube-proxy-445qx
1/1 Running 0 2m37s
kube-system kube-proxy-8swdg
1/1 Running 0 2m2s
kube-system kube-proxy-bl9rx
1/1 Running 0 2m47s
kube-system kube-proxy-pv8gj
1/1 Running 0 2m43s
kube-system kube-proxy-v7cw2
1/1 Running 0 2m43s
kube-system
kube-scheduler-pz-dev-k8s-ncus-00001-control-18dabdb141b 1/1
Running 1 (2m22s ago) 2m50s
kube-system
kube-scheduler-pz-dev-k8s-ncus-00001-control-18dabdb6ad6 1/1
Running 0 2m15s
kube-system
kube-scheduler-pz-dev-k8s-ncus-00001-control-18dabdbc0a8 1/1
Running 0 2m37s
kube-system weave-net-8dvl5
2/2 Running 0 2m37s
kube-system weave-net-c54bz
2/2 Running 0 2m43s
kube-system weave-net-lv8l4
2/2 Running 1 (2m42s ago) 2m47s
kube-system weave-net-vg6td
2/2 Running 0 2m2s
kube-system weave-net-vq9s4
2/2 Running 0 2m43s
kubernetes-dashboard dashboard-metrics-scraper-5657497c4c-4k886
1/1 Running 0 2m46s
kubernetes-dashboard kubernetes-dashboard-5b749d9495-jpbxl
1/1 Running 1 (2m22s ago) 2m46s
Control 2: Errors at the CLI
Failed to start Execute cloud user/final scripts.
kubectl get nodes
E0215 08:27:07.797825 2772 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:07.798759 2772 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:07.801039 2772 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:07.801977 2772 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:07.804029 2772 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify
the right host or port?
kubectl get pods --all-namespaces
E0215 08:29:41.818452 2811 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:41.819935 2811 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:41.820883 2811 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:41.822680 2811 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:41.823571 2811 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify
the right host or port?
Ping Google: Success
Ping Control Node 1: Success
Control 3: Errors at the CLI
Failed to start Execute cloud user/final scripts.
Failed to start deploy-kube-system.service.
kubectl get nodes
E0215 08:27:15.057313 2697 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:15.058538 2697 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:15.059260 2697 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:15.061599 2697 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:27:15.062029 2697 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify
the right host or port?
kubectl get pods --all-namespaces
E0215 08:29:57.108716 2736 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:57.109533 2736 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:57.111372 2736 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:57.112074 2736 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
E0215 08:29:57.113956 2736 memcache.go:265] couldn't get current server
API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp
127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify
the right host or port?
Ping Google: Success
Ping Control Node 1: Success
On Thu, Feb 15, 2024 at 2:17 AM Wei ZHOU <[email protected]> wrote:
> Can you try with 3 control nodes ?
>
> -Wei
>
> On Thu, 15 Feb 2024 at 09:13, Wally B <[email protected]> wrote:
>
> > - zone type :
> > Core
> > - network type:
> > Advanced
> > Isolated Network inside a Redundant VPC (same results in just an
> > Isolated network without VPC)
> > - number of control nodes:
> > 2 Control Nodes (HA Cluster)
> >
> > We were able to deploy k8s in the past, not sure what changed.
> >
> > Thanks!
> > -Wally
> >
> > On Thu, Feb 15, 2024 at 2:04 AM Wei ZHOU <[email protected]> wrote:
> >
> > > Hi,
> > >
> > > can you share
> > > - zone type
> > > - network type
> > > - number of control nodes
> > >
> > >
> > > -Wei
> > >
> > > On Thu, 15 Feb 2024 at 08:52, Wally B <[email protected]> wrote:
> > >
> > > > So
> > > >
> > > > Recreating the Sec Storage VM Fixed the Cert issue and I was able to
> > > > install K8s 1.28.4 Binaries. --- THANKS Wei ZHOU !
> > > >
> > > >
> > > > Im still getting
> > > >
> > > > [FAILED] Failed to start Execute cloud user/final scripts.
> > > >
> > > > on 1 control and 1 worker.
> > > >
> > > > *Control 1 -- pz-dev-k8s-ncus-00001-control-18dabaf66c1 -- :* No
> > > > errors at the CLI
> > > >
> > > > kubectl get nodes
> > > > NAME STATUS ROLES
> > AGE
> > > > VERSION
> > > > pz-dev-k8s-ncus-00001-control-18dabaf0edb Ready control-plane
> > 5m2s
> > > > v1.28.4
> > > > pz-dev-k8s-ncus-00001-control-18dabaf66c1 Ready control-plane
> > > 4m44s
> > > > v1.28.4
> > > > pz-dev-k8s-ncus-00001-node-18dabafb0bd Ready <none>
> > > 4m47s
> > > > v1.28.4
> > > > pz-dev-k8s-ncus-00001-node-18dabb006bc Ready <none>
> > > 4m47s
> > > > v1.28.4
> > > >
> > > >
> > > > kubectl get pods --all-namespaces
> > > > NAMESPACE NAME
> > > > READY STATUS RESTARTS AGE
> > > > kube-system coredns-5dd5756b68-295gb
> > > > 1/1 Running 0 5m32s
> > > > kube-system coredns-5dd5756b68-cdwvw
> > > > 1/1 Running 0 5m33s
> > > > kube-system etcd-pz-dev-k8s-ncus-00001-control-18dabaf0edb
> > > > 1/1 Running 0 5m36s
> > > > kube-system etcd-pz-dev-k8s-ncus-00001-control-18dabaf66c1
> > > > 1/1 Running 0 5m23s
> > > > kube-system
> > > > kube-apiserver-pz-dev-k8s-ncus-00001-control-18dabaf0edb
> > 1/1
> > > > Running 0 5m36s
> > > > kube-system
> > > > kube-apiserver-pz-dev-k8s-ncus-00001-control-18dabaf66c1
> > 1/1
> > > > Running 0 5m23s
> > > > kube-system
> > > > kube-controller-manager-pz-dev-k8s-ncus-00001-control-18dabaf0edb
> > 1/1
> > > > Running 1 (5m13s ago) 5m36s
> > > > kube-system
> > > > kube-controller-manager-pz-dev-k8s-ncus-00001-control-18dabaf66c1
> > 1/1
> > > > Running 0 5m23s
> > > > kube-system kube-proxy-2m8zb
> > > > 1/1 Running 0 5m26s
> > > > kube-system kube-proxy-cwpjg
> > > > 1/1 Running 0 5m33s
> > > > kube-system kube-proxy-l2vbf
> > > > 1/1 Running 0 5m26s
> > > > kube-system kube-proxy-qhlqt
> > > > 1/1 Running 0 5m23s
> > > > kube-system
> > > > kube-scheduler-pz-dev-k8s-ncus-00001-control-18dabaf0edb
> > 1/1
> > > > Running 1 (5m8s ago) 5m36s
> > > > kube-system
> > > > kube-scheduler-pz-dev-k8s-ncus-00001-control-18dabaf66c1
> > 1/1
> > > > Running 0 5m23s
> > > > kube-system weave-net-5cs26
> > > > 2/2 Running 1 (5m9s ago) 5m26s
> > > > kube-system weave-net-9zqrw
> > > > 2/2 Running 1 (5m28s ago) 5m33s
> > > > kube-system weave-net-fcwtr
> > > > 2/2 Running 0 5m23s
> > > > kube-system weave-net-lh2dh
> > > > 2/2 Running 1 (4m41s ago) 5m26s
> > > > kubernetes-dashboard dashboard-metrics-scraper-5657497c4c-r284t
> > > > 1/1 Running 0 5m32s
> > > > kubernetes-dashboard kubernetes-dashboard-5b749d9495-vtwdd
> > > > 1/1 Running 0 5m32s
> > > >
> > > >
> > > >
> > > > *Control 2 --- pz-dev-k8s-ncus-00001-control-18dabaf66c1 :*
> > [FAILED]
> > > > Failed to start Execute cloud user/final scripts.
> > > >
> > > > kubectl get nodes
> > > > E0215 07:38:33.314561 2643 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:38:33.316751 2643 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:38:33.317754 2643 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:38:33.319181 2643 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:38:33.319975 2643 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > The connection to the server localhost:8080 was refused - did you
> > specify
> > > > the right host or port?
> > > >
> > > >
> > > > kubectl get pods --all-namespaces
> > > > E0215 07:42:23.786704 2700 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:42:23.787455 2700 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:42:23.789529 2700 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:42:23.790051 2700 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > E0215 07:42:23.791742 2700 memcache.go:265] couldn't get current
> > > server
> > > > API group list: Get "http://localhost:8080/api?timeout=32s": dial
> tcp
> > > > 127.0.0.1:8080: connect: connection refused
> > > > The connection to the server localhost:8080 was refused - did you
> > specify
> > > > the right host or port?
> > > >
> > > >
> > > > */var/log/daemon.log*
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1KuIx0jI4TuAXPgACY3rJQz3L2B8AjeqOL0Fm5r4YF5M/edit?usp=sharing
> > > >
> > > > */var/log/messages*
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/15xet6kxI9rdgi4RkIHqtn-Wywph4h1Coyt_cyrJYkv4/edit?usp=sharing
> > > >
> > > > On Thu, Feb 15, 2024 at 1:21 AM Wei ZHOU <[email protected]>
> > wrote:
> > > >
> > > > > Destroy ssvm and retry when new ssvm is Up ?
> > > > >
> > > > > -Wei
> > > > >
> > > > > 在 2024年2月15日星期四，Wally B <[email protected]> 写道：
> > > > >
> > > > > > Super Weird. I have two other versions added successfully but now
> > > when
> > > > I
> > > > > > try to add an ISO/version I get the following on the management
> > host.
> > > > > This
> > > > > > is the first time I've tried adding a K8s version since 4.18.0
> > > > > >
> > > > > >
> > > > > > tail -f /var/log/cloudstack/management/management-server.log |
> grep
> > > > ERROR
> > > > > >
> > > > > > 2024-02-15 06:26:18,900 DEBUG [c.c.a.t.Request]
> > > > > > (AgentManager-Handler-5:null) (logid:) Seq
> 48-6373437897659383816:
> > > > > > Processing: { Ans: , MgmtId: 15643723020152, via: 48, Ver: v1,
> > > Flags:
> > > > > 10,
> > > > > > [{"com.cloud.agent.api.storage.DownloadAnswer":{"
> > > > > > jobId":"39d72d08-ab48-47dd-b09a-eee3ed816f4d","
> > > > > > downloadPct":"0","errorString":"PKIX
> > > > > > path building failed:
> > > > > > sun.security.provider.certpath.SunCertPathBuilderException:
> unable
> > to
> > > > > find
> > > > > > valid certification path to requested
> > > > > > target","downloadStatus":"DOWNLOAD_ERROR","downloadPath"
> > > > > > :"/mnt/SecStorage/73075a0a-38a1-3631-8170-8887c04f6073/
> > > > > > template/tmpl/1/223/dnld9180711723601784047tmp_","
> > > > > > installPath":"template/tmpl/1/223","templateSize":"(0
> > > > > > bytes) 0","templatePhySicalSize":"(0 bytes)
> > > > > > 0","checkSum":"4dfb9d8be2191bc8bc4b89d78795a5
> > > > > > b","result":"true","details":"PKIX
> > > > > > path building failed:
> > > > > > sun.security.provider.certpath.SunCertPathBuilderException:
> unable
> > to
> > > > > find
> > > > > > valid certification path to requested
> > > > > > target","wait":"0","bypassHostMaintenance":"false"}}] }
> > > > > >
> > > > > > 2024-02-15 06:26:18,937 ERROR
> [o.a.c.s.i.BaseImageStoreDriverImpl]
> > > > > > (RemoteHostEndPoint-5:ctx-55063062) (logid:e21177cb) Failed to
> > > register
> > > > > > template: b6e79c5a-38d4-4cf5-8606-e6f209b6b4c2 with error: PKIX
> > path
> > > > > > building failed:
> > > > > > sun.security.provider.certpath.SunCertPathBuilderException:
> unable
> > to
> > > > > find
> > > > > > valid certification path to requested target
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 14, 2024 at 11:27 PM Wei ZHOU <[email protected]
> >
> > > > wrote:
> > > > > >
> > > > > > > Can you try 1.27.8 or 1.28.4 on
> > > https://download.cloudstack.org/cks/
> > > > ?
> > > > > > >
> > > > > > >
> > > > > > > -Wei
> > > > > > >
> > > > > > > 在 2024年2月15日星期四，Wally B <[email protected]> 写道：
> > > > > > >
> > > > > > > > Hello Everyone!
> > > > > > > >
> > > > > > > > We are currently attempting to deploy k8s clusters and are
> > > running
> > > > > into
> > > > > > > > issues with the deployment.
> > > > > > > >
> > > > > > > >
> > > > > > > > Current CS Environment:
> > > > > > > >
> > > > > > > > CloudStack Verison: 4.19.0 (Same issue before we upgraded
> from
> > > > > 4.18.1).
> > > > > > > > Hypervisor Type: Ubuntu 20.04.03 KVM
> > > > > > > > Attempted K8s Bins: 1.23.3, 1.27.3
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ======== ISSUE =========
> > > > > > > >
> > > > > > > > For some reason when we attempt the cluster provisioning all
> of
> > > the
> > > > > VMs
> > > > > > > > start up, SSH Keys are installed, but then at least 1,
> > sometimes
> > > 2
> > > > of
> > > > > > the
> > > > > > > > VMs (control and/or worker) we get:
> > > > > > > >
> > > > > > > > [FAILED] Failed to start deploy-kube-system.service.
> > > > > > > > [FAILED] Failed to start Execute cloud user/final scripts.
> > > > > > > >
> > > > > > > > The Cloudstack UI just says:
> > > > > > > > Create Kubernetes cluster test-cluster in progress
> > > > > > > > for about an hour (I assume this is the 3600 second timeout)
> > and
> > > > then
> > > > > > > > fails.
> > > > > > > >
> > > > > > > > In the users event log it stays on:
> > > > > > > > INFO KUBERNETES.CLUSTER.CREATE
> > > > > > > > Scheduled
> > > > > > > > Creating Kubernetes cluster. Cluster Id: XXX
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I can ssh into the VMs with their assigned private keys. I
> > > > attempted
> > > > > to
> > > > > > > run
> > > > > > > > the deploy-kube-system script but it just says already
> > > provisioned!
> > > > > I'm
> > > > > > > not
> > > > > > > > sure how I would Execute cloud user/final scripts. If I
> attempt
> > > to
> > > > > stop
> > > > > > > the
> > > > > > > > cluster and start it again nothing seems to change.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Any help would be appreciated, I can provide any details as
> > they
> > > > are
> > > > > > > > needed!
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > Wally
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
