Hi Chris,
I have some good news to share. Thanks to your detailed comments, I can
reproduce the issue easily in my lab. Here is my reproducer:
Start a fresh VM, either Jammy or Kinetic, just needs to use the Ubuntu
-generic kernel.
1. Edit /etc/default/grub and append apparmor=0 to GRUB_CMDLINE_LINUX_DEFAULT
2. sudo update-grub
3. sudo apt update
4. sudo apt install auditd
5. Append the following to /etc/audit/rules.d/audit.rules:
-a exit,always -F arch=b64 -S execve
-a exit,always -F arch=b32 -S execve
6. sudo reboot
7. sudo apt install stress-ng
8. stress-ng --exec $(nproc)
9. Check the following for memory leaks:
$ watch "sudo cat /proc/meminfo | grep SUnreclaim"
$ watch "sudo cat /proc/slabinfo | grep kmalloc-2k"
$ sudo slabtop
At this point SUnreclaim will grow rapidly, at a rate of 3mb or so per
second. If you leave it for a few minutes, it will consume hundreds of
megabytes.
I have been doing some testing, and the Jammy 5.15.0-46-generic and
Kinetic 5.19.0-15-generic kernels are affected.
I tried 5.15.0-25-generic as well, and it had the same issue.
I tried mainline 5.15 and 5.19 from the Ubuntu mainline repo, but they
did not reproduce the issue at all.
Interesting. It currently looks like a custom Ubuntu SAUCE patch to
either apparmor or audit is causing the memory leak. I'm going to start
investigating this more deeply.
For now, I think that you should run with apparmor=1 on the kernel
command line as a workaround while we root cause and get this fixed.
I'll keep you updated on what I find.
Thanks,
Matthew
** Also affects: linux (Ubuntu Kinetic)
Importance: Undecided
Status: Confirmed
** Also affects: linux (Ubuntu Jammy)
Importance: Undecided
Status: New
** Changed in: linux (Ubuntu Jammy)
Status: New => In Progress
** Changed in: linux (Ubuntu Kinetic)
Status: Confirmed => In Progress
** Changed in: linux (Ubuntu Jammy)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Kinetic)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Jammy)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Changed in: linux (Ubuntu Kinetic)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Tags added: jammy kinetic seg
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1987430
Title:
Ubuntu 22.04 kernel 5.15.0-46-generic leaks kernel memory in
kmalloc-2k slabs
Status in linux package in Ubuntu:
In Progress
Status in linux source package in Jammy:
In Progress
Status in linux source package in Kinetic:
In Progress
Bug description:
Since updating to kernel 5.15.0-46-generic (package version
5.15.0-46.49), all of our Ubuntu 22.04 LTS servers are leaking kernel
memory; our first server with 8 GB of RAM just fatally OOMed, causing
us to detect this. Inspection of OOM reports, /proc/meminfo, and
/proc/slabinfo says that it's mostly going to unreclaimable kmalloc-2k
slabs:
Aug 23 12:51:11 cluster kernel: [361299.864757] Unreclaimable slab
info:
Aug 23 12:51:11 cluster kernel: [361299.864757] Name
Used Total
[...]
Aug 23 12:51:11 cluster kernel: [361299.864924] kmalloc-2k
6676584KB 6676596KB
Most of our machines appear to be leaking slab memory at a rate of
around 20 to 40 Mbytes/hour, with some machines leaking much faster;
the champions are leaking kernel memory at 145 Mbytes/hour and 237
Mbytes/hour.
We aren't running any proprietary kernel modules and our only unusual
kernel configuration is that we've disabled AppArmor with 'apparmor=0'
on the kernel command line.
/proc/version_signature:
Ubuntu 5.15.0-46.49-generic 5.15.39
Full kernel command line from the Dell R240 system that fatally OOMd:
BOOT_IMAGE=/boot/vmlinuz-5.15.0-46-generic
root=UUID=3165564f-a2dd-4b39-935b-114f3e23ff54 ro console=ttyS0,115200
console=tty0 apparmor=0
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1987430/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
