jdhirst opened a new issue, #11691: URL: https://github.com/apache/cloudstack/issues/11691
### problem I have been recently testing out the CKS enhancements in Cloudstack and have been wondering if anyone happens to have an example CNI configuration for Cilium? Additionally, the Calico example in the documentation doesnt appear to work (at least with the 1.33 prebuilt ISO I was using) Calico example here: https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-service.html ISO I used: https://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso (I have also tested by building a custom kubernetes image following the documentation with the same results) I am also using the default CloudStack 4.21 system VM instead of a custom node image. When using the custom CNI config, I get the following error in the control node's cloud-init-output.log: ``` 2025-09-20 06:44:36,835 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token found character '%' that cannot start any token in "<unicode string>", line 431, column 4: {% if registry is defined %} ^" 2025-09-20 06:44:36,849 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 431 column 4: "while scanning for the next token found character '%' that cannot start any token in "<unicode string>", line 431, column 4: {% if registry is defined %} ``` I pulled the userdata files from the control node in question and base64 decoded them: [userdata_0.txt](https://github.com/user-attachments/files/22439017/userdata_0.txt) [userdata_1.txt](https://github.com/user-attachments/files/22439016/userdata_1.txt) I can see this if registry is defined thing, but am not sure whether that means something is not being templated correctly or not: ``` {% if registry is defined %} - path: /opt/bin/setup-containerd permissions: '0755' owner: root:root content: | #!/bin/bash -e export registryConfig="\\ [plugins.\"io.containerd.grpc.v1.cri\".registry.mirrors.\"{{registry.url.endpoint}}\"]\n \\ endpoint = [\"{{registry.url}}\"]" export registryCredentials="\\ [plugins.\"io.containerd.grpc.v1.cri\".registry.configs.\"{{registry.url.endpoint}}\".auth]\n\tusername = \"{{registry.username}}\" \n\tpassword = \"{{registry.password}}\" \n\tidentitytoken = \"{{registry.token}}\"" echo "creating config file for containerd" containerd config default > /etc/containerd/config.toml sed -i '/\[plugins."io.containerd.grpc.v1.cri".registry\]/a '"${registryCredentials}"'' /etc/containerd/config.toml sed -i '/\[plugins."io.containerd.grpc.v1.cri".registry.mirrors\]/a '"${registryConfig}"'' /etc/containerd/config.toml echo "Restarting containerd service" systemctl daemon-reload systemctl restart containerd {% endif %} ``` ### versions cloudstack-management 4.21.0.0-1 cloudstack-agent 4.21.0.0-1 ### The steps to reproduce the bug 1. Either create a new kubernetes binaries ISO or use [this one](https://download.cloudstack.org/cks/setup-v1.33.1-calico-x86_64.iso) 2. Import the example calico CNI configuration from the documentation: ``` #cloud-config - for i in {1..3}; do curl https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml -o /home/cloud/calico.yaml && break || sleep 5; done - until [ -f /home/cloud/success ]; do sleep 5; done - echo "Kubectl apply file" - for i in {1..3}; do sudo /opt/bin/kubectl create -f /home/cloud/calico.yaml && break || sleep 5; done - export PATH=$PATH:/home/cloud - | cat << 'EOF' > /home/cloud/create-configs.sh #!/bin/bash cat << 'EOL' > /home/cloud/bgp-config.yaml apiVersion: crd.projectcalico.org/v1 kind: BGPConfiguration metadata: name: default spec: logSeverityScreen: Debug asNumber: {{ AS_NUMBER }} EOL cat << 'EOL' > /home/cloud/bgp-peer.yaml apiVersion: crd.projectcalico.org/v1 kind: BGPPeer metadata: name: bgp-peer-example spec: peerIP: {{ ds.meta_data.peer_ip_address }} asNumber: {{ ds.meta_data.peer_as_number }} EOL EOF - chmod +x /home/cloud/create-configs.sh - /home/cloud/create-configs.sh - for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-config.yaml && break || sleep 5; done - for i in {1..3}; do sudo /opt/bin/kubectl apply -f /home/cloud/bgp-peer.yaml && break || sleep 5; done ``` 3. Create a new cluster and define the custom CNI configuration. It will be stuck in Starting state forever and the control node will fail to render the cloud init scripts properly. To login to the node, use the `debian` user instead of `cloud` since it hasn't yet rebooted into the new user. ... ### What to do about it? Please provide a working example of a custom CNI configuration for CloudStack 4.21.0. Either calico or cilium (preferred since I am hoping to build clusters with cilium). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
