Bug#1072831: getting memory info fails when running under lxc
On Mon 24 Jun 2024, Craig Small wrote: > On Sun, 9 Jun 2024 at 01:03, Paul Slootman wrote: > > I am running a number of virtual systems under lxc via libvirt. > > This means these systems share the host kernel (not like qemu where a > > whole virtual machine is emulated). > Hi Paul, > I did the following (as root) > > lxc-create --name debtest2 --template download -- --dist debian > --release bookworm --arch amd64 > sudo lxc-start --name debtest2 > lxc-attach --name debtest2 I'm running the lxc containers under control of libvirt, I suspect the difference is how libvirt implements the container vs. plain lxc. With libvirt the controlling process is /usr/lib/libvirt/libvirt_lxc , with lxc I see /usr/bin/lxc-start . Regards, Paul
Bug#1072831: getting memory info fails when running under lxc
On Sun, 9 Jun 2024 at 01:03, Paul Slootman wrote: > I am running a number of virtual systems under lxc via libvirt. > This means these systems share the host kernel (not like qemu where a > whole virtual machine is emulated). Hi Paul, I did the following (as root) lxc-create --name debtest2 --template download -- --dist debian --release bookworm --arch amd64 sudo lxc-start --name debtest2 lxc-attach --name debtest2 root@debtest2:/# free totalusedfree shared buff/cache available Mem:32712728 1392032694744 76414032698808 Swap:1000444 0 1000444 root@debtest2:/# free -V free from procps-ng 4.0.2 I'm not seeing this issue at all.
Bug#1072831: getting memory info fails when running under lxc
Control: tags -1 fixed-upstream On Tue, 11 Jun 2024 at 21:16, Paul Slootman wrote: > I see I missed the case lseek() fails with another errno. > Updated patch attached. Thanks Paul, This was applied upstream at https://gitlab.com/procps-ng/procps/-/commit/104b3ce3df67092eeb868ba5e019cb895ebdf32d We're hoping to get a new procps release soon. - Craig
Bug#1072831: getting memory info fails when running under lxc
On 6/12/24 9:35 AM, Jim Warner wrote: For what it's worth, I am unable to duplicate the meminfo problem in a newly created debian-bookworm based lxc container. I'm using the snap version of lxd identified as: $ lxc --version 5.21.1 LTS The top, ps and free programs (all identified as from procps-ng 4.0.2) work just fine. Sorry, I should have also mentioned that the host is ubuntu 24.04.
Bug#1072831: getting memory info fails when running under lxc
Hi Guys, For what it's worth, I am unable to duplicate the meminfo problem in a newly created debian-bookworm based lxc container. I'm using the snap version of lxd identified as: $ lxc --version 5.21.1 LTS The top, ps and free programs (all identified as from procps-ng 4.0.2) work just fine. Regards, Jim
Bug#1072831: getting memory info fails when running under lxc
On Tue 11 Jun 2024, Paul Slootman wrote:
> This works for me. Patch attached.
I see I missed the case lseek() fails with another errno.
Updated patch attached.
Paul
--- library/meminfo.c.orig 2023-07-11 11:09:18.436786212 +0200
+++ library/meminfo.c 2024-06-11 13:11:12.878627527 +0200
@@ -646,12 +646,20 @@
// clear out the soon to be 'current' values
memset(>hist.new, 0, sizeof(struct meminfo_data));
-if (-1 == info->meminfo_fd
-&& (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY
-return 1;
-
-if (lseek(info->meminfo_fd, 0L, SEEK_SET) == -1)
-return 1;
+if (-1 == info->meminfo_fd) {
+ if (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY)))
+ return 1;
+}
+else {
+ if (lseek(info->meminfo_fd, 0L, SEEK_SET) == -1)
+ if (ESPIPE == errno) {
+ close(info->meminfo_fd);
+ if (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY)))
+ return 1;
+ }
+ else
+ return 1;
+}
for (;;) {
if ((size = read(info->meminfo_fd, buf, sizeof(buf)-1)) < 0) {
Bug#1072831: getting memory info fails when running under lxc
tags 1072831 patch
thanks
On Tue 11 Jun 2024, Craig Small wrote:
> Could you check to see if in the container that lxcfs has overwritten
> the /proc/meminfo file? They sometimes do this for /proc/uptime. They
> might have messed one of the lines up and choked procps; I'm thinking
> like a tab/space at the end of the line or something that looks right
> to a human but not a computer.
The /proc/meminfo in the lxc container is basically the same as that of
the host, except the total memory reflects the limit imposed on the
container. So
-MemTotal:7914164 kB$
+MemTotal:2097152 kB$
and the Slab / SReclaimable / SUnreclaim lines show 0 kB.
> I think we'll need to either run a strace on free or a debugger to see
> where exactly the issue is.
I've been doing this.
Apparently the /proc/meminfo inside the lxc container is not seekable,
errno is ESPIPE.
The code does some seeks to reset to the beginning in preparation for
rereading the file.
I've changed the code to not do an lseek() just after opening the file
(as we should be at the start of the file then anyway), and if the file
is already open, try lseek() and if that returns errno == ESPIPE, then
close and reopen the file.
This works for me. Patch attached.
I don't know whether configure needs to test for existence of ESPIPE;
I do believe it's POSIX.
Paul
--- library/meminfo.c.orig 2023-07-11 11:09:18.436786212 +0200
+++ library/meminfo.c 2024-06-11 10:41:41.643438234 +0200
@@ -646,12 +646,18 @@
// clear out the soon to be 'current' values
memset(>hist.new, 0, sizeof(struct meminfo_data));
-if (-1 == info->meminfo_fd
-&& (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY
-return 1;
-
-if (lseek(info->meminfo_fd, 0L, SEEK_SET) == -1)
-return 1;
+if (-1 == info->meminfo_fd) {
+ if (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY)))
+ return 1;
+}
+else {
+ if (lseek(info->meminfo_fd, 0L, SEEK_SET) == -1)
+ if (ESPIPE == errno) {
+ close(info->meminfo_fd);
+ if (-1 == (info->meminfo_fd = open(MEMINFO_FILE, O_RDONLY)))
+ return 1;
+ }
+}
for (;;) {
if ((size = read(info->meminfo_fd, buf, sizeof(buf)-1)) < 0) {
Bug#1072831: getting memory info fails when running under lxc
On Sun, 9 Jun 2024 at 01:03, Paul Slootman wrote: > # free > free: Unable to create meminfo structure That's procps_meminfo_new() failing but /proc/meminfo exists. The function: checks the parameters allocates some memory for the structure runs meminfo_make_hash_failed(), which sets up the hashtable runs meminfo_read failed which does some opens, seeks and reads. Could you check to see if in the container that lxcfs has overwritten the /proc/meminfo file? They sometimes do this for /proc/uptime. They might have messed one of the lines up and choked procps; I'm thinking like a tab/space at the end of the line or something that looks right to a human but not a computer. I think we'll need to either run a strace on free or a debugger to see where exactly the issue is. I'm not particularly worried about ps. This is a library meminfo issue so fixing free should fix ps too. procps after 3.3.17 has a *lot* of changes, but I didn't think parsing the meminfo file was one of those. - Craig
Bug#1072831: getting memory info fails when running under lxc
Package: procps Version: 2:4.0.2-3 Severity: normal I am running a number of virtual systems under lxc via libvirt. This means these systems share the host kernel (not like qemu where a whole virtual machine is emulated). I upgraded one to bookworm today, and when running 'ps faxu' or 'free' I get an error: Unable to create meminfo structure I downgraded procps to 2:3.3.17-5 (including libprocps8 2:3.3.17-5) and then it worked again. Working 'free' output: # free totalusedfree shared buff/cache available Mem: 2097152 64600 19323529028 100200 1932352 Swap: 10338296 795392 9542904 Not working 'free' output: # free free: Unable to create meminfo structure Contents of /proc/meminfo : MemTotal:2097152 kB MemFree: 1930792 kB MemAvailable:1930792 kB Buffers: 0 kB Cached: 100672 kB SwapCached:60812 kB Active: 119140 kB Inactive: 38800 kB Active(anon): 64480 kB Inactive(anon): 84 kB Active(file): 54660 kB Inactive(file):38716 kB Unevictable: 660 kB Mlocked: 43200 kB SwapTotal: 10338296 kB SwapFree:9542904 kB Zswap: 0 kB Zswapped: 0 kB Dirty: 344 kB Writeback: 0 kB AnonPages: 3726656 kB Mapped: 455856 kB Shmem: 9028 kB KReclaimable: 142356 kB Slab: 0 kB SReclaimable: 0 kB SUnreclaim:0 kB KernelStack:8576 kB PageTables:19560 kB SecPageTables:48 kB NFS_Unstable: 0 kB Bounce:0 kB WritebackTmp: 0 kB CommitLimit:14295376 kB Committed_AS:7452164 kB VmallocTotal: 34359738367 kB VmallocUsed: 42352 kB VmallocChunk: 0 kB Percpu: 3216 kB HardwareCorrupted: 0 kB AnonHugePages: 3117056 kB ShmemHugePages:0 kB ShmemPmdMapped:0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free:0 HugePages_Rsvd:0 HugePages_Surp:0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 179704 kB DirectMap2M: 6940672 kB DirectMap1G: 3145728 kB I'm willing to help debug on my system if needed. Thanks, Paul -- System Information: Debian Release: 12.5 APT prefers stable APT policy: (800, 'stable'), (500, 'stable-updates'), (500, 'stable-security'), (500, 'unstable'), (500, 'oldstable'), (100, 'bookworm-fasttrack'), (100, 'bookworm-backports-staging') Architecture: amd64 (x86_64) Kernel: Linux 6.1.0-9-amd64 (SMP w/4 CPU threads; PREEMPT) Locale: LANG=en_IE.UTF-8, LC_CTYPE=en_IE.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages procps depends on: ii init-system-helpers 1.65.2 ii libc62.36-9+deb12u4 ii libncursesw6 6.4-4 ii libproc2-0 2:4.0.2-3 ii libtinfo66.4-4 Versions of packages procps recommends: ii psmisc 23.6-1 procps suggests no packages. -- no debconf information
