Public bug reported: [Environment]
Kubernetes 1.16.17 Containerd 1.3.3 Ubuntu Bionic [Affected Releases] containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x [Description] Reported upstream: https://github.com/containerd/containerd/issues/4108 The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841 Caused a regression. The following endpoint description works with containerd 1.2.X without defining a protocol scheme. (/etc/containerd/config.toml). [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"] endpoint = ["niedbalski-bastion.cloud.sts:5000"] This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't works using the same registry mirror definition. The pod definition is: apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: niedbalski-bastion.cloud.sts:5000/busybox:latest command: - sleep - "3600" imagePullSecrets: - name: regcred restartPolicy: Always New pods fail with the following error: " failed to do request: Head niedbalski- bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4 Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" [Steps to reproduce] Configure a private docker repository repository Modify the containerd registry mirror config as follows: ** http://paste.ubuntu.com/p/yP63WMkVT6/ Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/) Status of the scheduled pod should be ImagePullBackOff and the before mentioned error should be raised. [Possible workaround and solution] As a workaround change the endpoint to support the scheme (https://) Provide a fallback mechanism for URL parsing validation to fallback to http or https. I suspect that this change introduced on 1.3.3 through 0b29c9c) may be the offending commit. ** Affects: containerd (Ubuntu) Importance: Undecided Status: New ** Affects: containerd (Ubuntu Bionic) Importance: Undecided Status: New ** Affects: containerd (Ubuntu Eoan) Importance: Undecided Status: New ** Affects: containerd (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Bionic) Importance: Undecided Status: New ** Description changed: [Environment] Kubernetes 1.16.17 Containerd 1.3.3 Ubuntu Bionic + + [Affected Releases] + + containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x + containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x + containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x + [Description] Reported upstream: https://github.com/containerd/containerd/issues/4108 The bump of to version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841 Caused a regression. The following endpoint description works with containerd 1.2.X without defining a protocol scheme. (/etc/containerd/config.toml). - [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"] - endpoint = ["niedbalski-bastion.cloud.sts:5000"] + [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"] + endpoint = ["niedbalski-bastion.cloud.sts:5000"] This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't works using the same registry mirror definition. The pod definition is: apiVersion: v1 kind: Pod metadata: - name: busybox - namespace: default + name: busybox + namespace: default spec: - containers: - - name: busybox - image: niedbalski-bastion.cloud.sts:5000/busybox:latest - command: - - sleep - - "3600" - imagePullSecrets: - - name: regcred - restartPolicy: Always + containers: + - name: busybox + image: niedbalski-bastion.cloud.sts:5000/busybox:latest + command: + - sleep + - "3600" + imagePullSecrets: + - name: regcred + restartPolicy: Always New pods fail with the following error: " failed to do request: Head niedbalski- bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4 Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" [Steps to reproduce] Configure a private docker repository repository Modify the containerd registry mirror config as follows: ** http://paste.ubuntu.com/p/yP63WMkVT6/ Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/) Status of the scheduled pod should be ImagePullBackOff and the before mentioned error should be raised. [Possible workaround and solution] As a workaround change the endpoint to support the scheme (https://) Provide a fallback mechanism for URL parsing validation to fallback to http or https. I suspect that this change introduced on 1.3.3 through 0b29c9c) may be the offending commit. ** Also affects: containerd (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: containerd (Ubuntu Eoan) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1867398 Title: [Regression] unsupported protocol scheme To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
