Created a new bug report:
https://bugs.launchpad.net/ubuntu/+source/linux-aws-5.13/+bug/1978475
Hopefully @electricdaemon and other affected users can help to provide a
reproducible test.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Sebastian Neumann (basti-megamorf+ubuntu-com) - please start a new bug
report so that we can address your specific problem. It may or may not
be related to the patch that fixed this kernel crash.
** Changed in: linux-aws-5.13 (Ubuntu Focal)
Status: Fix Committed => Fix Released
** Changed
I can confirm that the problem is indeed not fully fixed.
@electricdaemon said:
> Test kernel posted fixes crash but has another bug with unkillable
stuck defunct docker-proxy service causing more issues. Bug is not
solved. Tested on Linux AWS Lightsail instance.
And that's the problem that I'm
Fixed kernel works like a charm.
@matthew-nocturnal you have to change the default GRUB that loads, so it
is on /etc/default/grub. There you change the DEFAULT_GRUB with another
one, as has been pointed out in the previous messages. But now you can
simply run apt update / upgrade and it should
How did people fix this on aws instances that have no serial console
access? assuming the disk was mounted and grub.cfg was edited. what did
you change in the grub.cfg?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in
All of the updated 5.13 kernels have now made it to the archive and into
both the focal-updates and focal-security pockets. That list of kernels
is:
linux-aws-5.13 - 5.13.0-1029.32~20.04.1
linux-azure-5.13 - 5.13.0-1029.34~20.04.1
linux-gcp-5.13 - 5.13.0-1031.37~20.04.1
linux-oracle-5.13 -
In Oracle Cloud you can start a cloud shell console connection then
force reboot the instance and in the console press esc, in the Grub menu
select the previous kernel. I was able to boot with kernel
5.13.0-1030-oracle without panics.
--
You received this bug notification because you are a
The ability to do 'sudo grub-reboot' depends on the use-case. In my
case, the docker jobs were started via crontab, and the machine didn't
crash completely, so I was able to login.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Not sure if it will work on GCP but in Azure you hold escape to get into
Grub while the system is booting.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker
i'm intrigued, how do you 'sudo grub-reboot' when the machine is
crashed?
And if anyone knows how to get the grub boot menu to respond to the
keyboard over the serial console on GCP that'd be great, as it would
have having to attach the disk to another instance to change the boot
kernel or
I had limited success with "grub-reboot 2", but the following worked
fine for me on an AWS EC2 running Ubuntu 20.04.2
sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux
5.13.0-1025-aws"
--
You received this bug notification because you are a member of Kernel
Packages, which is
For those who can't update, because the machine starts docker at startup
and so crashes before you can get a shell open to upgrade to 1031,
here's my method (on gcp)
stop and edit machine to detach disk
attach to another machine boot that and mount somewhere
edit /boot/grub/grub.cfg and add
Updated kernels are in flight. The updated kernel packages and versions
are:
linux-aws-5.13- 5.13.0-1029.32~20.04.1
linux-azure-5.13 - 5.13.0-1029.34~20.04.1
linux-gcp-5.13- 5.13.0-1031.37~20.04.1
linux-oracle-5.13 - 5.13.0-1034.40~20.04.1
The azure and gcp kernels are already in
On GCP the fix hit apt. So the easiest way to fix now is simply `sudo
apt update && sudo apt upgrade`
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker
For Debian and Ubuntu, I used "sudo grub-reboot 2", which forces grub
menu 2's kernel on next reboot. Once rebooted, use "dpkg-l | grep 1028"
and apt remove each package relate to kernel 1028. Apt will also update
grub for you.
Don't forget to uninstall or "break" unattended-upgrades, which is
@n1sni - wanted to extend my thanks as well, but on ubuntu 20.04 that
settings was not present in /etc/default/grub, so i had to uninstall
1028 and install 1025. After adding that setting, and reloading and
rebooting the change didn't take place, hence the reinstall. Going to
make an AMI, until
I would like to echo earlier comments, I think that all affected kernel
packages should be pulled from the APT repositories, I also think that
all cloud images built with the bad kernel should be pulled too.
--
You received this bug notification because you are a member of Kernel
Packages, which
@n1sni - thank you for your post.
With 5.13.0-1028-aws I could only run hello-world without killing the
host.
Reverting back to 5.13.0-1025-aws from 5.13.0-1028-aws I can now run our
build containers without problems.
--
You received this bug notification because you are a member of Kernel
So this was painful for us. AWS hosted server running Ubuntu 20.04.4
LTS. Just for others, here are the steps we took thanks to the other
comments here:
We had to force shut down the machine and wait (aws console). Then we
got this script running, and started the machine back up:
while true; do
Sorry for my ignorance of the software development procedure for Ubuntu,
but now that this fix has been committed, how long until it is available
via apt on the normal release channels?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
@cz172638 - we are aware that the 5.15 Focal backport kernels have this
issue as well. However, since 5.15 is not the default -edge kernel yet,
it will have to wait until the next SRU cycle due for release 20-June,
2022.
** Changed in: linux-oracle-5.13 (Ubuntu Focal)
Importance: Undecided =>
Bug confirmed on Oracle Cloud running Ubuntu 20.04.4 Kernel
5.13.0-1033-oracle.
Workaround with ssh by Gerard(g-kok) works. thx
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
linux-intel-iotg-5.15:
5.15.0-1003 good
5.15.0-1008 bad
also reproducer (using podman) is smaller:
podman run --rm -it alpine:3.16 ls
which knocks down system
versus
podman run --rm -it busybox ls
which doesn't
--
You received this bug notification because you are a member of Kernel
I can confirm going back to 5.13.0-1027-gcp from 5.13.0-1030-gcp fixed
it for me.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes
We can confirm on: 5.13.0-1028-azure
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on linux-aws
@timg-tpi Yes, in https://bugs.launchpad.net/bugs/1977973 I found
5.13.0-1027-gcp to work fine.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container
@electricdaemon - please start a new bug report with sufficient detail
that someone can diagnose the problem. Is this a regression from
previous versions ?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
@electricdaemon - please start a new bug report with sufficient detail
that someone can diagnose the problem. Is this a regression from
previous versions ?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
below kernels on azure have this issue. please hold on the new images which
contain these kernel releases. thanks.
focal/linux-azure-5.13: 5.13.0-1026-azure
focal/linux-azure-5.15: 5.15.0-1008-azure
--
You received this bug notification because you are a member of Kernel
Packages, which is
Test kernel posted fixes crash but has another bug with unkillable stuck
defunct docker-proxy service causing more issues. Bug is not solved.
Tested on Linux AWS Lightsail instance.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-intel-iotg-5.15 (Ubuntu Focal)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-intel-iotg-5.15 (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
** Also affects: linux-intel-iotg-5.15 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-azure-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container
also present in linux-image-5.15.0-1008-intel-iotg:
##
[ 1666.319425] [ cut here ]
[ 1666.319433] kernel BUG at include/linux/fs.h:3082!
[ 1666.319443] invalid opcode: [#3] SMP NOPTI
[ 1666.319449] CPU: 0 PID: 17586 Comm:
hit same problem using podman in rootless on linux-image-5.15.0-1008-intel-iotg:
##
[ 1666.319425] [ cut here ]
[ 1666.319433] kernel BUG at include/linux/fs.h:3082!
[ 1666.319443] invalid opcode: [#3]
Is it possible to take this kernel back / away from repository before
more system are get broken?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1967924 seems
related.
"This patch is touching overlayfs, so we may see potential regressions
in overlayfs." We did indeed... :)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Just wondering, could we get a "run docker container" test as part of
the QA process going forward before new kernels are released?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
Has anyone had luck downgrading to a prior kernel and getting things in
a stable place?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation
Also seeing this on AWS with t4g instances. Kernel panic:
[ 12.489272] kernel BUG at include/linux/fs.h:3104!
[ 12.490111] Internal error: Oops - BUG: 0 [#1] SMP
[ 12.490923] Modules linked in: veth xt_nat xt_tcpudp xt_conntrack
xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user
I believe I'm getting a similar issue on Azure with a linux & Docker after
updates last night.
Trying to bring up the docker network interface.
[ 37.662249] kernel BUG at include/linux/fs.h:3103!
[ 37.665024] invalid opcode: [#1] SMP PTI
[ 37.667710] CPU: 1 PID: 3383 Comm: id Not
** Tags added: indeed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on linux-aws
5.13.0.1028.31~20.04.22
Status
What are the chances we can remove the the affected kernels from the
archives so more people don't get bit by this.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-oracle-5.13 (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-azure-5.13 (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-oracle-5.13 (Ubuntu Focal)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-aws-5.13 (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
The fix commit is impish/linux 6a6dd081d512c812a937503d5949e4479340accb
("UBUNTU: SAUCE: overlayfs: prevent dereferencing struct file in
ovl_vm_prfile_set()")
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws-5.13 in Ubuntu.
** Package changed: linux-gcp (Ubuntu) => linux-gcp-5.13 (Ubuntu)
** Also affects: linux-azure-5.13 (Ubuntu)
Importance: Undecided
Status: New
** Changed in: linux-azure-5.13 (Ubuntu Focal)
Importance: Undecided => High
** Changed in: linux-azure-5.13 (Ubuntu Focal)
Status:
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-gcp-5.13 (Ubuntu Focal)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-gcp in Ubuntu.
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-aws-5.13 (Ubuntu Focal)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
** Package changed: linux-aws (Ubuntu) => linux-aws-5.13 (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-gcp in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on
** Also affects: linux-aws (Ubuntu Focal)
Importance: Undecided
Status: New
** Also affects: linux-gcp (Ubuntu Focal)
Importance: Undecided
Status: New
** Also affects: linux-oracle-5.13 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug
Just tested this 5.13.0-1029.32~lp1977919.1 kernel and confirmed that it
fixes the issue (doesn't crash when running the same docker container
that would crash in the -1028 kernel)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws
Please try this test kernel at https://kernel.ubuntu.com/~rtg/focal-
docker-crash-lp1977919/5.13.0-1029.32~lp1977919.1/
wget
https://kernel.ubuntu.com/~rtg/focal-docker-crash-lp1977919/5.13.0-1029.32~lp1977919.1/amd64/linux-image-unsigned-5.13.0-1029-aws_5.13.0-1029.32~lp1977919.1_amd64.deb
wget
Work on this issue continues. We have identified the following impacted
kernels and versions:
focal linux-aws-5.13 5.13.0-1028.31~20.04.1
focal linux-azure-5.13 5.13.0-1028.33~20.04.1
focal linux-gcp-5.13 5.13.0-1030.36~20.04.1
focal linux-oracle-5.13 5.13.0-1033.39~20.04.1
--
You received
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-gcp (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
I have similar lernel panics launching docker-ce instances on the google
cloud platform after recent ubuntu update, 20.04 LTS. 22.04 is
unaffected.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
This happened to two of our instances in AWS. In the hope that this is
helpful to anyone: in an attempt to avoid having to mount the root
volumes on another instances, we disabled docker and containerd in the
small timeframe between SSH becoming accessible and the kernel panic, by
running
** Package changed: linux-meta-gcp-5.13 (Ubuntu) => linux-gcp (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-gcp (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
** Also affects: linux-meta-gcp-5.13 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation
This broke a lot of our servers running Docker, which I all had to
restore by adding the root volume to a different instance and then
changing /boot/grub/grub.cfg in order to boot 5.13.0-1025-aws again.
So another "I can confirm this" from me.
--
You received this bug notification because you
We are actively working on the issue. This also affects more than the
`linux-aws` kernel, as we've been able to reproduce on 5.13 versions of:
linux-oracle
linux-azure
linux-gcp
linux-aws
This appears to be confined to the latest 5.13 kernel update. We will
provide more updates shortly on all
Can confirm, one of my AWS EC2 instance running Ubuntu 20.04 is dying
during reboot after installing the update. Also running docker on this
instance.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
Can't even build the image:
kernel:[ 221.374595] Kernel panic - not syncing: Fatal exception in interrupt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker
`docker run -it ubuntu bash` is enough to cause it to crash.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on linux-aws
I think it has something to do with docker network / volumes. Because
with the container watchtower which doen'st use any open network ports
or volumes don't make the system crashing.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Confirmed on AWS AMI ubuntu/images/hvm-ssd/ubuntu-
focal-20.04-amd64-server-20211129. We fixed it by reverting to kernel
GNU/Linux 5.13.0-1025-aws x86_64, forcing GRUB to load it instead of a
corrupted one.
P.S.: We faced loop rebooting and unkillable docker process. After the
kernel downgrade,
I believe I've got the same issue on Azure 5.13.0-1028-azure.
** Attachment added: "Trace"
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1977919/+attachment/5595607/+files/kernel-issue-5.13.0-1028-azure.txt
--
You received this bug notification because you are a member of Kernel
Perhaps this is obvious, but same thing happens when using containerd
directly, without docker as intermediate.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker
Also seeing this on AWS Ubuntu 20.04 after an update to linux-image-
aws/focal-updates 5.13.0.1028.31~20.04.22
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker
Google Compute Engine seems to be affected as well for Ubuntu 20.04.
Using kernel 5.13.0-1030-gcp #36~20.04.1-Ubuntu
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
I'm also having this issue but on Oracle Cloud (linux-oracle
v5.13.0-1033.39)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel
Also can confirm - very easy to reproduce.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1977919
Title:
Docker container creation causes kernel oops on linux-aws
Can confirm it. A restore from last week 03.06.2022 on one node fixed
it.
Another node i reinstalled Ubuntu 20.04 and i had stoöö the same issue.
Here i fixed it with a reinstall of Ubuntu 22.04
--
You received this bug notification because you are a member of Kernel
Packages, which is
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux-aws (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
77 matches
Mail list logo
