[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~18.04.1 --- runc (1.0.0~rc95-0ubuntu1~18.04.1) bionic; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:11:34 -0300 ** Changed in: runc (Ubuntu Bionic) Status: Fix Committed => Fix Released ** Changed in: runc (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~20.04.1 --- runc (1.0.0~rc95-0ubuntu1~20.04.1) focal; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:06:57 -0300 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~21.04.1 --- runc (1.0.0~rc95-0ubuntu1~21.04.1) hirsute; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 10:53:22 -0300 ** Changed in: runc (Ubuntu Hirsute) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1~20.10.1 --- runc (1.0.0~rc95-0ubuntu1~20.10.1) groovy; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:01:11 -0300 ** Changed in: runc (Ubuntu Groovy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I did the verification of the packages in -proposed in all releases: Hirsute --- ubuntu@hirsute:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog runc (1.0.0~rc95-0ubuntu1~21.04.1) hirsute; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 10:53:22 -0300 ubuntu@hirsute:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null ... autopkgtest [16:38:08]: summary basic-smoke PASS command1 PASS Groovy -- ubuntu@groovy:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog runc (1.0.0~rc95-0ubuntu1~20.10.1) groovy; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:01:11 -0300 ubuntu@groovy:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null ... autopkgtest [16:39:17]: summary basic-smoke PASS command1 PASS Focal - ubuntu@focal:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog runc (1.0.0~rc95-0ubuntu1~20.04.1) focal; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:06:57 -0300 ubuntu@focal:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null ... autopkgtest [16:41:19]: summary basic-smoke PASS command1 PASS Bionic -- ubuntu@bionic:~/runc/runc-1.0.0-rc95$ head -n 12 debian/changelog runc (1.0.0~rc95-0ubuntu1~18.04.1) bionic; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 11:11:34 -0300 ubuntu@bionic:~/runc/runc-1.0.0-rc95$ sudo autopkgtest . -- null ... autopkgtest [16:42:02]: summary basic-smoke PASS command1 PASS ** Tags removed: verification-needed verification-needed-bionic verification-needed-groovy verification-needed-hirsute ** Tags added: verification-done verification-done-bionic verification-done-groovy verification-done-hirsute -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Regarding the golang-github-containers-common/0.33.4+ds1-1 regression in Hirsute, it was reported here in LP #1931258 as a LTO optimization issue. We could SRU this package adding the export DEB_BUILD_MAINT_OPTIONS=optimize=-lto to d/rules but I believe this does not worth a SRU. If the SRU team thinks this is important I can prepare an update with the content mentioned above (I confirmed it fixes the issue). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
The containerd regressions in Focal and Bionic were already sorted out, the tests are passing now. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I've been trying to find a fix for the golang-github-containers- common/0.33.4+ds1-1 regression in Hirsute but without success so far. However, the issue seems to be unrelated to this runc SRU, it is a compilation issue in one of its components: # github.com/containers/storage/pkg/devicemapper ../../../go-build3064824172/b263/_cgo_export.c:21:13: error: variable ‘_cgoexp_ebd00b0772db_StorageDevmapperLogCallback’ redeclared as function 21 | extern void _cgoexp_ebd00b0772db_StorageDevmapperLogCallback(void *); | ^ ../../../go-build3064824172/b263/cgo-generated-wrappers:1:5: note: previously declared here lto1: fatal error: errors during merging of translation units compilation terminated. Due to that, I'd ask the SRU team to ignore this regression for now and not block this SRU just because of it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I confirmed the golang-github-docker-go-connections regression in Hirsute and I filed another SRU bug to fix it: https://bugs.launchpad.net/ubuntu/+source/golang-github-docker-go- connections/+bug/1930891 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Hi Brian, I tested the package from focal-proposed and everything is working as expected. Tested version: root@node0:~# runc -v runc version 1.0.0~rc95-0ubuntu1~20.04.1 spec: 1.0.2-dev go: go1.13.8 libseccomp: 2.5.1 Test: Start 300 replicas of nginx on a single kuberentes node. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Hello Colin, or anyone else affected, Accepted runc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: runc (Ubuntu Focal) Status: Confirmed => Fix Committed ** Tags added: verification-needed-focal ** Changed in: runc (Ubuntu Bionic) Status: Confirmed => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Hello Colin, or anyone else affected, Accepted runc into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc95-0ubuntu1~21.04.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: runc (Ubuntu Hirsute) Status: Confirmed => Fix Committed ** Tags added: verification-needed verification-needed-hirsute ** Changed in: runc (Ubuntu Groovy) Status: Confirmed => Fix Committed ** Tags added: verification-needed-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
This bug was fixed in the package runc - 1.0.0~rc95-0ubuntu1 --- runc (1.0.0~rc95-0ubuntu1) impish; urgency=medium * New upstream release. - Several regressions were found in 1.0.0-rc93 by upstream and fixed in this new release. + Ensure the scratch pipe is read during ExportBPF (LP: #1927219). - Drop patches applied by upstream: + d/patches/CVE-2021-30465/*.patch + d/patches/fix-patchpbf-test-on-32-bit.patch * d/rules: set VERSION variable when building runc (LP: #1929106). -- Lucas Kanashiro Thu, 20 May 2021 10:40:14 -0300 ** Changed in: runc (Ubuntu) Status: Triaged => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Changed in: runc (Ubuntu Bionic) Importance: Undecided => High ** Changed in: runc (Ubuntu Focal) Importance: Undecided => High ** Changed in: runc (Ubuntu Groovy) Importance: Undecided => High ** Changed in: runc (Ubuntu Hirsute) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Tags added: sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Thanks for the testing Colin! FYI I already uploaded the fix to Impish (development release) and as soon it lands in the release pocket I can start the SRUs. At the moment it is blocked by one regression: https://people.canonical.com/~ubuntu-archive/proposed- migration/impish/update_excuses.html#runc I'll be working on that today to sort it out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
@Lucas, I just tested with runc 1.0.0~rc95-0ubuntu1~20.04.1~ppa1 from your PPA, everything is working ok (300 containers tested). I don't know if the containerd update is planned, but I ran the test with containerd 1.3.3-0ubuntu2.3 from the Ubuntu repo and 1.5.2-0ubuntu1~20.04.1~ppa1 form your PPA, both were running ok. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: runc (Ubuntu Hirsute) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: runc (Ubuntu Groovy) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: runc (Ubuntu Bionic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: runc (Ubuntu Focal) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
As Colin stated, this version of runc is causing a serious regression. As such it should be marked as High. ** Changed in: runc (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Tags added: seg -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Changed in: runc (Ubuntu) Assignee: (unassigned) => Lucas Kanashiro (lucaskanashiro) ** Changed in: runc (Ubuntu Focal) Assignee: (unassigned) => Lucas Kanashiro (lucaskanashiro) ** Changed in: runc (Ubuntu Groovy) Assignee: (unassigned) => Lucas Kanashiro (lucaskanashiro) ** Changed in: runc (Ubuntu Bionic) Assignee: (unassigned) => Lucas Kanashiro (lucaskanashiro) ** Changed in: runc (Ubuntu Hirsute) Assignee: (unassigned) => Lucas Kanashiro (lucaskanashiro) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Description changed: + [Impact] + + Several regressions were identified by upstream in version 1.0.0-rc93 + and were fixed in version 1.0.0-rc94: + + https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc94 + + "This release fixes several regressions found in v1.0.0-rc93. We + recommend users update as soon as possible." + + And in version 1.0.0-rc95 we also have the fix for CVE-2021-30465. + + [Test Plan] + + Per https://wiki.ubuntu.com/DockerUpdates, our test case is the + autopkgtests + + [Where problems could occur] + + As usual, we deliver most benefit to our users by delivering an upstream + experience. A risk of regressions is part of that. + + [Original Message] + When upgrading runc to the latest version from focal-updates, Kubernetes, using containerd, fails to start new containers after a certain amount of container have been reached a certain amount (more than 100 ~ 150 containers). With the previous version of runc, I was able to run more than 340 containers on a single server without any issue. I got those logs in containerd (journalctl -u containerd) ``` May 05 00:48:17 node6 containerd[1368656]: time="2021-05-05T00:48:17.260349619Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" May 05 00:48:21 node6 containerd[1368656]: time="2021-05-05T00:48:21.807947009Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown" May 05 00:48:23 node6 containerd[1368656]: time="2021-05-05T00:48:23.810434820Z" level=error msg="get state for 71005cc34ac16f6976a7aebd72faa734792639b003c5dec67f666449cf6eb8d7" error="context deadline exceeded: unknown" May 05 00:48:25 node6 containerd[1368656]: time="2021-05-05T00:48:25.813453574Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown" May 05 00:48:27 node6 containerd[1368656]: time="2021-05-05T00:48:27.816285268Z" level=error msg="get state for c6497bbf82f668f8b8b534204cd2efaa30a97f884da3dd5fca4f51489298f855" error="context deadline exceeded: unknown" May 05 00:48:29 node6 containerd[1368656]: time="2021-05-05T00:48:29.823052797Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown" May 05 00:48:31 node6 containerd[1368656]: time="2021-05-05T00:48:31.826412096Z" level=error msg="get state for f90f054966ab6b78304797d420a1f948c85e5b6e425037417f8966c08cd9a0f1" error="context deadline exceeded: unknown" ``` This version of runc triggered the problem: ``` runc (1.0.0~rc93-0ubuntu1~20.04.1) focal; urgency=medium - * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322, - LP: #1916485). + * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322, + LP: #1916485). - -- Lucas Kanashiro Tue, 16 Mar 2021 15:34:35 -0300 + -- Lucas Kanashiro Tue, 16 Mar 2021 15:34:35 -0300 ``` ``` # runc -v runc version spec: 1.0.2-dev go: go1.13.8 libseccomp: 2.5.1 ``` - Reverting to the previous version of runc solved the problem, and I was able to run more than 340 pods / containers without any error. ``` apt-get install runc=1.0.0~rc10-0ubuntu1 # runc -v runc version spec: 1.0.1-dev ``` ProblemType: Bug DistroRelease: Ubuntu 20.04 Package: runc 1.0.0~rc93-0ubuntu1~20.04.1 ProcVersionSignature: Ubuntu 5.4.0-72.80-generic 5.4.101 Uname: Linux 5.4.0-72-generic x86_64 ApportVersion: 2.20.11-0ubuntu27.16 Architecture: amd64 CasperMD5CheckResult: skip Date: Wed May 5 12:06:30 2021 SourcePackage: runc UpgradeStatus: No upgrade log present (probably fresh install) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
** Also affects: runc (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: runc (Ubuntu Hirsute) Importance: Undecided Status: New ** Also affects: runc (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: runc (Ubuntu Focal) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
@Colin I updated my PPA with version 1.0-rc95 which also contains the fix for CVE-2021-30465. Could you test this new version and make sure your regression is fixed? ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-30465 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I'm marking this bug as Triaged: I didn't try to reproduce it locally, but I think it's clear enough we have a bug here. Also tagging this as a regression. ** Changed in: runc (Ubuntu) Status: Incomplete => Triaged ** Tags added: regression-update server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Thanks for raising this issue. The VERSION file is not missing but we are not using the upstream Makefile to build the source, that's why the version is not set. I fixed this issue and uploaded the fix to my PPA (it might take some minutes to get it published), now if you update the runc package you will have the correct version when running 'runc --version'. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
The missing version seems to be caused by a different build environment (missing VERSION file at the root of the source tree). https://github.com/opencontainers/runc/blob/master/main.go#L20 https://github.com/opencontainers/runc/blob/master/Makefile#L15 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
The version published in the PPA fixes the issue. ``` root@node0:~# runc -v runc version spec: 1.0.2-dev go: go1.13.8 libseccomp: 2.5.1 ``` Fore some reason, the runc version published in the PPA don't show the actual runc version with `runc -v`, just like the previous version (1.0-rc93) published in the focal-update repository. The patches applied to the orig source tree only relate to test fixes for the build environment. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I updated the runc package to version 1.0-rc94 and it is available in this PPA for testing: https://launchpad.net/~lucaskanashiro/+archive/ubuntu/container-stack Let me know if this indeed fix your issue. If yes, I'll update runc in the development release (a.k.a. impish) and then start the SRU process (I'll need you to perform the same tests at this stage). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Of course, I could try a PPA or an attached deb file, anything that suits you best, everything is running on a test system anyway. Thanks for your help! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Thanks for the investigation! It seems that a bunch of regressions were found in 1.0.0~rc93 as we can see in the 1.0.0~rc94 release notes: "This release fixes several regressions found in v1.0.0-rc93. We recommend users update as soon as possible." https://github.com/opencontainers/runc/releases/tag/v1.0.0-rc94 I'll be preparing a PPA with this new runc version and ask you to test it if possible. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I tested the pre-built binary of the latest version of runc (by just replacing /usr/sbin/runc) and I was able to go to 400 pods on a single test node. This new version was released today. ``` root@node0:~# runc -v runc version 1.0.0-rc94 spec: 1.0.2-dev go: go1.14.15 libseccomp: 2.5.1 ``` https://github.com/opencontainers/runc/releases I tried the pre-built binary of runc 1.0.0-rc93, and I was stuck to 164 pods again with "code = DeadlineExceeded desc = context deadline exceeded" error: ``` root@node0:~# runc -v runc version 1.0.0-rc93 spec: 1.0.2-dev go: go1.14.14 libseccomp: 2.5.1 ``` The changelog for 1.0.0-rc94 contains this line that could match: runc init: fix a hang caused by deadlock in seccomp/ebpf loading code (regression in rc93, #2871) The version 1.0.0-rc92 was working fine too. ``` root@node0:~# runc -v runc version 1.0.0-rc92 spec: 1.0.2-dev ``` Please upgrade to version 1.0.0-rc94 or downgrade to 1.0.0-rc92. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
In order to test weave's container network impact, on the limit, I
started 200 "sleep 7200" pods with the busybox image, with "hostNetwork:
true" to disable container networking.
This time, I was able to run 200 replicas, but I was unable to go to
over 200 pods, I tried the grow the replicas count to 300 and just got
100 pods in a ContainerCreating state, and the same logs.
```
piVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 200
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
hostNetwork: true
containers:
- name: nginx
image: busybox
command: ['sleep', '7200']
```
```
May 08 00:12:34 node0 containerd[399]: time="2021-05-08T00:12:34.406607997Z"
level=warning msg="unknown status" status=0
May 08 00:12:36 node0 containerd[399]: time="2021-05-08T00:12:36.409698273Z"
level=error msg="get state for
0c20bb658e597c7ef66089e910e525e4da03f4145f0dcee645202ae7bb18778c"
error="context deadline exceeded: unknown"
May 08 00:12:36 node0 containerd[399]: time="2021-05-08T00:12:36.410297996Z"
level=warning msg="unknown status" status=0
May 08 00:12:38 node0 containerd[399]: time="2021-05-08T00:12:38.413845203Z"
level=error msg="get state for
04f407c1f0b373163b0f646351e9533b4219e4f33f7a0ddc6725314dce8a21e3"
error="context deadline exceeded: unknown"
May 08 00:12:38 node0 containerd[399]: time="2021-05-08T00:12:38.413877394Z"
level=warning msg="unknown status" status=0
May 08 00:12:40 node0 containerd[399]: time="2021-05-08T00:12:40.416039250Z"
level=error msg="get state for
e7a50022e2fb9806899f3d088b3d30562f012a0d550c7be19520b7bb8e8a6563"
error="context deadline exceeded: unknown"
May 08 00:12:40 node0 containerd[399]: time="2021-05-08T00:12:40.416476408Z"
level=warning msg="unknown status" status=0
```
```
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.486466380Z"
level=info msg="starting signal loop" namespace=k8s.io
path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/9e325f009e5a4456eea7f9037ef5e2513e8dd4634b0b4427b57faa8b367bc37e
pid=132736
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.518910596Z"
level=info msg="starting signal loop" namespace=k8s.io
path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/dbfa2e9ebef9cc375080b05d32b19b5465f504cf102cea9c6b0ed1c07324bb9f
pid=132762
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.526959046Z"
level=info msg="starting signal loop" namespace=k8s.io
path=/run/containerd/io.containerd.runtime.v2.task/k8s.io/86a0e34b828552812d1e23da8445c20977c4d55383f3a4b30772297ed320da21
pid=132792
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.555110616Z"
level=info msg="shim disconnected"
id=769c0f791b28650f343595fa1b2aef3605001ccbebcacbec8180a44a44a8565e
May 08 00:14:43 node0 containerd[399]: time="2021-05-08T00:14:43.568631491Z"
level=error msg="RunPodSandbox for
{Name:nginx-deployment-5c4997c6fd-jk4nd,Uid:6588e20a-f4e4-46fa-9e0b-03a57fe4282b,Namespace:default,Attempt:0,}
failed, error" error="failed to start sandbox container task
\"769c0f791b28650f343595fa1b2aef3605001ccbebcacbec8180a44a44a8565e\": context
deadline exceeded: unknown"
```
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927219
Title:
context deadline exceeded: unknown in containerd with latest runc
version
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
I was able to reproduce the behaviour under a test VM. I used a simple example nginx deployment with a high number of replicas. With the older version of runc, I was able to run 200 replicas successfully and even 250. With the newer version of runc, I was only able to run 164 replicas out of 200 (the last 36 pods were stuck in a ContainerCreating state. There were also 2 system pods on the node (weave + kube-proxy), so a total of 166 containers. ``` nginx-deployment-66b6c48dd5-2hdgd 0/1 ContainerCreating 0 11m nginx-deployment-66b6c48dd5-2ql8r 0/1 ContainerCreating 0 11m nginx-deployment-66b6c48dd5-4z54r 0/1 ContainerCreating 0 11m nginx-deployment-66b6c48dd5-5vvzl 0/1 ContainerCreating 0 11m nginx-deployment-66b6c48dd5-6xh9g 0/1 ContainerCreating 0 11m ... ``` I saw the same logs as what I saw on the production server for containerd, and nothing more in dmesg / kern.log. The load on the node was ok: ``` root@node0:~# vmstat -S M 5 procs ---memory-- ---swap-- -io -system-- --cpu- r b swpd free buff cache si sobibo in cs us sy id wa st 0 0 0 2305 7099700 140 390 1242 2560 5 4 92 0 0 1 0 0 2305 7099800 0 0 2312 4529 1 1 98 0 0 0 0 0 2303 7099800 0 3 5055 11301 4 4 92 0 0 0 0 0 2303 7099800 0 0 2171 4232 1 1 98 0 0 0 0 0 2302 7099800 0 0 5512 11719 4 4 92 0 0 2 0 0 2302 7099800 0 0 2182 4261 1 1 98 0 0 1 0 0 2302 7099800 0 0 3822 7883 3 1 96 0 0 0 0 0 2302 7099800 0 0 3724 7891 3 3 95 0 0 0 0 0 2303 7099800 0 0 2729 5351 3 1 96 0 0 0 0 0 2300 7099800 0 4 4674 9863 3 3 94 0 0 0 0 0 2300 7099800 0 0 2476 4930 2 1 97 0 0 1 0 0 2305 7099700 065 5219 10770 4 4 91 0 0 0 0 0 2430 7099400 0 2267 4918 9713 6 5 89 0 0 0 0 0 2558 7199000 0 2033 7205 15041 7 7 85 1 0 0 0 0 2589 7198900 0 1364 5006 10203 6 5 88 1 0 0 0 0 2429 7199300 0 1554 9502 20112 12 13 74 1 0 0 0 0 2325 7199700 0 1043 9185 19772 11 11 78 1 0 0 0 0 2303 7199800 0 1810 3723 7264 4 4 91 0 0 2 0 0 2318 7199800 0 0 3181 6494 3 2 96 0 0 0 0 0 2317 7199800 023 2789 5525 2 1 97 0 0 3 0 0 2307 71999002684 5406 11430 5 4 90 0 0 0 0 0 2307 7199900 031 2631 5201 2 1 97 0 0 0 0 0 2306 7199900 0 5 2687 5532 2 1 97 0 0 1 0 0 2306 7199900 0 221 4689 9833 3 3 94 0 0 1 0 0 2305 7199900 022 3072 6339 2 2 96 0 0 0 0 0 2306 7199900 0 4 2482 4812 2 1 98 0 0 0 0 0 2307 7199900 046 3528 7051 3 1 96 0 0 0 0 0 2306 7199900 040 4363 9302 3 3 94 0 0 0 0 0 2306 7199900 033 2537 5363 2 1 97 0 0 2 0 0 2306 7199900 035 4215 8648 3 3 94 0 0 0 0 0 2306 7199900 0 6 3303 6824 3 2 95 0 0 1 0 0 2306 7199900 0 3 2122 4117 2 1 97 0 0 1 0 0 2305 7199900 0 0 5339 11477 4 3 93 0 0 0 0 0 2304 7199900 0 0 2193 4326 1 1 98 0 0 0 0 0 2304 7199900 0 0 5424 11640 5 4 92 0 0 4 0 0 2304 7199900 0 0 1859 3685 2 1 98 0 0 2 0 0 2302 7199900 0 0 5260 11252 4 4 92 0 0 3 0 0 2302 7199900 0 0 2044 4116 1 1 98 0 0 0 0 0 2301 7199900 0 0 4949 11084 4 3 93 0 0 2 0 0 2301 7199900 0 0 2252 4349 2 1 98 0 0 0 0 0 2299 7199900 0 0 5329 11410 4 3 92 0 0 2 0 0 2299 7199900 0 0 2077 4186 2 1 97 0 0 root@node0:~# free -m totalusedfree shared buff/cache available Mem: 594525772296 2010713297 Swap: 0 0 0 root@node0:~# uptime 23:41:26 up 20 min, 1 user, load average: 0.48, 0.75, 0.53 ``` After a while, I went down to 150 replicas, which were running
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Hi, Here is the kern.log extract of the time containers were trying to start. I only see logs about network interfaces, and the log is full of it. I don't see those logs anymore after the runc version downgrade. It looks like just a consequence of the runner's start failure. About the load, I don't think it was very high, the server is quite powerful (256GB of RAM, 16 core / 32 threads). I experienced the problem on a production system during what should have been a quick server reinstall during late maintenance hours, so I might be mistaken about the load at that time. I'll try to reproduce the issue on a test environment this weekend to get real load figures with both versions of runc & allow further testing. ** Attachment added: "kern.log" https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+attachment/5495606/+files/kern_extract.log -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1927219 Title: context deadline exceeded: unknown in containerd with latest runc version To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1927219] Re: context deadline exceeded: unknown in containerd with latest runc version
Hello and thanks for this bug report. I had a look a the runc bug
reports and changes to see if I could easily spot a relevant change, but
I couldn't. I don't have a good hypothesis of what could be wrong, but
I'd start checking if the newer runc is causing a higher memory or CPU
usage, causing components to timeout ("context deadline exceeded") or
OOMs.
Could you please:
- Attach the kernel log (dmesg) to this bug, captured after
hitting those containerd errors?
- Check the system load and memory usage when using the two
different versions of runc. This could be done for example
via something like `vmstat -S M 5`.
If no useful clues are found, then I think we'll have to bisect.
** Changed in: runc (Ubuntu)
Status: New => Incomplete
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1927219
Title:
context deadline exceeded: unknown in containerd with latest runc
version
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/runc/+bug/1927219/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
