** Description changed: [Description] Kubernetes 1.16.17 Containerd 1.3.3 Ubuntu Bionic [Affected Releases] containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x [Impact] Reported upstream: https://github.com/containerd/containerd/issues/4108 - The bump of to version 1.3.3 through [0] - https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841 + User Impact: - Caused a regression. + Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841 + a regression was introduced. - The following endpoint description works with containerd 1.2.X without defining - a protocol scheme. (/etc/containerd/config.toml). + The following endpoint description stopped working when scheduling pods + with k8s 1.16-1.17 isn't longer working. [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"] endpoint = ["niedbalski-bastion.cloud.sts:5000"] - This stopped working on 1.3.X , scheduling pods with k8s 1.16-1.17 doesn't - works using the same registry mirror definition. - The pod definition is: + + As an example, A pod defined as following: apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: niedbalski-bastion.cloud.sts:5000/busybox:latest command: - sleep - "3600" imagePullSecrets: - name: regcred restartPolicy: Always - New pods fail with the following error: + + Will fail with the following error: " failed to do request: Head niedbalski- bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4 Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" [Test Case] - Configure a private docker repository repository + 1) Configure a private docker repository repository - Modify the containerd registry mirror config as follows: + 2) Modify the containerd registry mirror config as follows: ** http://paste.ubuntu.com/p/yP63WMkVT6/ - Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/) + 3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/) Status of the scheduled pod should be ImagePullBackOff and the before mentioned error should be raised. + [Possible workaround and solution] As a workaround change the endpoint to support the scheme (https://) Provide a fallback mechanism for URL parsing validation to fallback to http or https. I suspect that this change introduced on 1.3.3 through 0b29c9c) may be the offending commit. [Regression Potential] - ** Not identified yet any regression potential, this functionality fixes - an existing regression introduced in the latest update. + ** The change proposed on the SRU takes in consideration both cases + 1) a endpoint without a schema 2) a endpoint with a schema. + + 1) worked in 1.2.6 as explained in the "Impact section" and stopped + being supported with the current Bionic version 1.3.3, 2) Should work + on both cases. + + In neither case this should break existing endpoint definitions + now new deployments of containerd. [Other Info] ** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue.
** Description changed: [Description] Kubernetes 1.16.17 Containerd 1.3.3 Ubuntu Bionic [Affected Releases] containerd | 1.3.3-0ubuntu1~18.04.1 | bionic-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1~19.10.1 | eoan-updates/universe | source, amd64, arm64, armhf, i386, ppc64el, s390x containerd | 1.3.3-0ubuntu1 | focal | source, amd64, arm64, armhf, ppc64el, s390x [Impact] Reported upstream: https://github.com/containerd/containerd/issues/4108 User Impact: Since the Ubuntu bionic-updates bump of the version 1.3.3 through [0] https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841 a regression was introduced. The following endpoint description stopped working when scheduling pods with k8s 1.16-1.17 isn't longer working. [plugins."io.containerd.grpc.v1.cri".registry.mirrors."niedbalski-bastion.cloud.sts:5000"] endpoint = ["niedbalski-bastion.cloud.sts:5000"] - - As an example, A pod defined as following: + As an example, creating a k8s pod defined as following: apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: niedbalski-bastion.cloud.sts:5000/busybox:latest command: - sleep - "3600" imagePullSecrets: - name: regcred restartPolicy: Always - Will fail with the following error: + Will fail in the current Bionic-updates version with the following + error: " failed to do request: Head niedbalski- bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Normal Scheduled default-scheduler Successfully assigned default/busybox to juju-3a79d2-00268738-4 Normal Pulling 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Failed to pull image "niedbalski-bastion.cloud.sts:5000/busybox:latest": rpc error: code = Unknown desc = failed to pull and unpack image "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to resolve reference "niedbalski-bastion.cloud.sts:5000/busybox:latest": failed to do request: Head niedbalski-bastion.cloud.sts:///v2/busybox/manifests/latest: unsupported protocol scheme "niedbalski-bastion.cloud.sts" Warning Failed 8m39s (x4 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ErrImagePull Warning Failed 8m27s (x6 over 10m) kubelet, juju-3a79d2-00268738-4 Error: ImagePullBackOff Normal BackOff 4m56s (x21 over 10m) kubelet, juju-3a79d2-00268738-4 Back-off pulling image "niedbalski-bastion.cloud.sts:5000/busybox:latest" [Test Case] 1) Configure a private docker repository repository 2) Modify the containerd registry mirror config as follows: ** http://paste.ubuntu.com/p/yP63WMkVT6/ 3) Execute the following pod (http://paste.ubuntu.com/p/BVYQFMfCmk/) Status of the scheduled pod should be ImagePullBackOff and the before mentioned error should be raised. - [Possible workaround and solution] As a workaround change the endpoint to support the scheme (https://) Provide a fallback mechanism for URL parsing validation to fallback to http or https. I suspect that this change introduced on 1.3.3 through 0b29c9c) may be the offending commit. [Regression Potential] ** The change proposed on the SRU takes in consideration both cases - 1) a endpoint without a schema 2) a endpoint with a schema. + 1) a endpoint without a schema 2) a endpoint with a schema. 1) worked in 1.2.6 as explained in the "Impact section" and stopped being supported with the current Bionic version 1.3.3, 2) Should work on both cases. In neither case this should break existing endpoint definitions now new deployments of containerd. [Other Info] ** This commit upstream https://github.com/containerd/containerd/commit/a022c218194c05449ad69b69c48fc6cac9d6f0b3 addresses the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1867398 Title: [Regression] unsupported protocol scheme To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1867398/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
