Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Control: retitle -1 systemd user instance: Attaching egress BPF program to cgroup failed with Invalid argument > Currently, to trigger this issue, I need task-xfce-desktop (on every arch.). > I believe that this can be reproduced without task-xfce-desktop, > and will try to find a reproduction procedure with CUI in a weekend... A minimal reproduction procedure of 986905 is as follows: 1. Install dbus-user-session, pulseaudio and libpam-systemd. 2. Add "DefaultIPAccounting=yes" to /etc/systemd/user.conf (reload systemd if necessary) 3. Login as a non-root user of UID >= 1000. 4. journalctl -b -g BPF as root. Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed,Re: Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
> So you enabled "DefaultIPAccounting=yes" for the systemd --user > instances (i.e. in user.conf) and only those systemd --user instances > trigger that error? Yes (on amd64, arm64 and armhf). I do not see the error in the systemd PID 1 except on an armhf kernel with CONFIG_BPF_JIT_ALWAYS_ON=y. But we do not need to consider the issue on the armhf kernel here. > If you remove "DefaultIPAccounting=yes" from user.conf, are the error > messages gone? Yes. This seems something like "Permission denied" or "Operation not permitted", but the error is "Invalid argument", which seems strange. Currently, to trigger this issue, I need task-xfce-desktop (on every arch.). I believe that this can be reproduced without task-xfce-desktop, and will try to find a reproduction procedure with CUI in a weekend... Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Am 15.04.21 um 19:46 schrieb Michael Biebl: Am 15.04.21 um 08:57 schrieb Ryutaroh Matsumoto: * In the VM as root echo "DefaultIPAccounting=yes" >>/etc/systemd/user.conf in your initial email you said /etc/systemd/system.conf? So you enabled "DefaultIPAccounting=yes" for the systemd --user instances (i.e. in user.conf) and only those systemd --user instances trigger that error? If you remove "DefaultIPAccounting=yes" from user.conf, are the error messages gone? Michael OpenPGP_signature Description: OpenPGP digital signature
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Am 15.04.21 um 08:57 schrieb Ryutaroh Matsumoto: * In the VM as root echo "DefaultIPAccounting=yes" >>/etc/systemd/user.conf in your initial email you said /etc/systemd/system.conf? OpenPGP_signature Description: OpenPGP digital signature
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Hi Michael, I failed to send my email to you: > This reported symptom (986905) is also observed with Debian amd64&arm64, > outside of a VM, with task-xfce-desktop installed and > DefaultIPAccounting=yes in /etc/systemd/user.conf. We do not need armhf to reproduce 986905. > The systemd system instance (PID==1) also gives > "Attaching egress BPF program to cgroup failed", > and its reason is "Cannot allocate memory". (What is it???) "Cannot allocate memory" was not observed with Debian linux-image-armmp-lpae, but CONFIG_BPF_JIT_ALWAYS_ON=y caused "Cannot allocate memory" on armhf. CONFIG_BPF_JIT_ALWAYS_ON=n in Debian kernel. "Cannot allocate memory" might be a bug in systemd upstream, but Debian does not suffer from it. Sorry for my misinformation. Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
My reports have been wrong. This reported symptom is also observed with Debian amd64&arm64, outside of a VM, with task-xfce-desktop installed and DefaultIPAccounting=yes in /etc/systemd/user.conf. In https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986905#5 all the error messages are from systemd user instance (whose PID != 1), and the error is "Invalid argument". But on linux-image-armmp-lpae running a barebone hardware (my RPi4B 8GB), the symptom has difference appearance: The systemd system instance (PID==1) also gives "Attaching egress BPF program to cgroup failed", and its reason is "Cannot allocate memory". (What is it???) A full output of "journalctl -b -g BPF" on barebone hardware is as below, which cannot be reproduced in a VM (afaik): -- Journal begins at Tue 2021-04-13 20:01:59 JST, ends at Wed 2021-04-14 12:58:58 JST. -- Apr 14 12:57:48 raspi4b-armhf systemd[1]: -.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-getty.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-getty.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-modprobe.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-modprobe.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-postfix.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-postfix.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-serial\x2dgetty.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-serial\x2dgetty.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-systemd\x2dfsck.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-systemd\x2dfsck.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system-wpa_supplicant.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-wpa_supplicant.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: user.slice: Attaching egress BPF program to cgroup /sys/fs/cgroup/user.slice failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: dev-hugepages.mount: Attaching egress BPF program to cgroup /sys/fs/cgroup/dev-hugepages.mount failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: dev-mqueue.mount: Attaching egress BPF program to cgroup /sys/fs/cgroup/dev-mqueue.mount failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: sys-kernel-debug.mount: Attaching egress BPF program to cgroup /sys/fs/cgroup/sys-kernel-debug.mount failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: sys-kernel-tracing.mount: Attaching egress BPF program to cgroup /sys/fs/cgroup/sys-kernel-tracing.mount failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: fake-hwclock.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/fake-hwclock.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: keyboard-setup.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/keyboard-setup.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: kmod-static-nodes.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/kmod-static-nodes.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: modprobe@configfs.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-modprobe.slice/modprobe@configfs.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: modprobe@drm.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-modprobe.slice/modprobe@drm.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: modprobe@fuse.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/system-modprobe.slice/modprobe@fuse.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: systemd-modules-load.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/systemd-modules-load.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: systemd-remount-fs.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/systemd-remount-fs.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: systemd-udev-trigger.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/systemd-udev-trigger.service failed: Cannot allocate memory Apr 14 12:57:48 raspi4b-armhf systemd[1]: system.s
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Hi, a minimal procedure to reproduce this in armhf VM is as follows: > Download d-i Alpha 3 for armhf. > (The latest weekly build of d-i did not boot...) * Start installer in VM with non-expert CUI mode. * Install as you like with "standard packages" with Xfce desktop (Instllation is painfully slow on VM...) * In the VM as root echo "DefaultIPAccounting=yes" >>/etc/systemd/user.conf * Reboot the VM * Make sure lightdm.service is running normally. A video card must be with the VM. * journalctl -b -g BPF I guess that task-gnome-desktop also produces this symptom, but Gnome is less likely for armhf installations than Xfce. Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Control: tags -1 - moreinfo > Do you have instructions how I can reproduce this (on a amd64 box)? I removed "moreinfo" tag. If the following is not enough, please attach the tag again. (Assuming "virt-manager" is acceptable) In the following, lighdm display manager is somehow needed to reproduce the reported symptom. "systemctl set-target multi-user.target" prevents the symptom from appearing... >> Is it related with CONFIG_BPF_JIT_DEFAULT_ON??? > Hm, I don't think so. The above looks ok. > Can you raise this upstream please (maybe with a reference to #7054). > I don't have an armhf setup to easily reproduce this myself. > I'd attach an strace log to the upstream bug report as well. I am weakly doubting if this is an upstream issue, could you confirm again that this should be reported upstream...? This seems caused by subtle interactions among Debian packages... *** start of reproduction steps *** (Verified on Ubuntu 20.04 amd64 notebook) # apt-get --install-recommends install virt-manager qemu-system-arm qemu-system-aarch64 qemu-efi-arm qemu-efi-aarch64 ipxe-qemu A qemu hard disk image of size 5GB with no password for root is placed at http://153.240.174.134:64193/debian11-armhf.qcow2 (alternative instructions to produce an equivalent image is given below). Start the virt-manager. In the virt-manager: * Create a new VM. * Select "arm" as "Architecture", with "virt" machine type, 2 CPUs and memory = 5120 MB (smaller maybe OK). * Check "customize configuration before install" * By pushing "Add Hardware", add (they are needed for lightdm to start) - "virtio video" in the subcategory video. - "virtio keyboard" and "virtio tablet" in the subcategory input * Boot into the installed disk image. * Log in as root with no password, verify lightdm.service has started and "journalctl -b -g BPF". *** steps to produce debian11-armhf.qcow2 *** Download d-i Alpha 3 for armhf. (The latest weekly build of d-i did not boot...) * Install as you like with "standard packages" with NO desktop. * linux-image-armmp-lpae should be chosen as the kernel. * In the VM as root run https://github.com/emojifreak/debian-rpi-image-script/blob/main/root-setup.sh I am unsure if just installing task-xfce-desktop reproduces the reported symptom... Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed, Re: Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Am 15.04.21 um 02:27 schrieb Ryutaroh Matsumoto: Can you attach the output of grep BPF /boot/config-$(uname -r) root@qemu-bullseye-armhf:~# uname -a Linux qemu-bullseye-armhf 5.10.0-5-armmp-lpae #1 SMP Debian 5.10.26-1 (2021-03-27) armv7l GNU/Linux root@qemu-bullseye-armhf:~# fgrep -nH BPF /boot/config-5.10.0-5-armmp-lpae /boot/config-5.10.0-5-armmp-lpae:161:CONFIG_CGROUP_BPF=y /boot/config-5.10.0-5-armmp-lpae:189:CONFIG_BPF=y /boot/config-5.10.0-5-armmp-lpae:216:CONFIG_BPF_LSM=y /boot/config-5.10.0-5-armmp-lpae:217:CONFIG_BPF_SYSCALL=y /boot/config-5.10.0-5-armmp-lpae:218:# CONFIG_BPF_JIT_ALWAYS_ON is not set /boot/config-5.10.0-5-armmp-lpae:219:# CONFIG_BPF_PRELOAD is not set /boot/config-5.10.0-5-armmp-lpae:1195:CONFIG_IPV6_SEG6_BPF=y /boot/config-5.10.0-5-armmp-lpae:1335:CONFIG_NETFILTER_XT_MATCH_BPF=m /boot/config-5.10.0-5-armmp-lpae:1554:# CONFIG_BPFILTER is not set /boot/config-5.10.0-5-armmp-lpae:1714:CONFIG_NET_CLS_BPF=m /boot/config-5.10.0-5-armmp-lpae:1741:CONFIG_NET_ACT_BPF=m /boot/config-5.10.0-5-armmp-lpae:1796:CONFIG_BPF_JIT=y /boot/config-5.10.0-5-armmp-lpae:1797:CONFIG_BPF_STREAM_PARSER=y /boot/config-5.10.0-5-armmp-lpae:2013:CONFIG_LWTUNNEL_BPF=y /boot/config-5.10.0-5-armmp-lpae:2021:CONFIG_HAVE_EBPF_JIT=y /boot/config-5.10.0-5-armmp-lpae:7784:# CONFIG_NBPFAXI_DMA is not set /boot/config-5.10.0-5-armmp-lpae:10100:CONFIG_BPF_EVENTS=y /boot/config-5.10.0-5-armmp-lpae:10173:CONFIG_TEST_BPF=m root@qemu-bullseye-armhf:~# In addition, there is little difference between arm64 and armhf in linux kernel config as follows. Is it related with CONFIG_BPF_JIT_DEFAULT_ON??? Hm, I don't think so. The above looks ok. Can you raise this upstream please (maybe with a reference to #7054). I don't have an armhf setup to easily reproduce this myself. I'd attach an strace log to the upstream bug report as well. Regards, Michael OpenPGP_signature Description: OpenPGP digital signature
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed,Re: Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
> Can you attach the output of > grep BPF /boot/config-$(uname -r) root@qemu-bullseye-armhf:~# uname -a Linux qemu-bullseye-armhf 5.10.0-5-armmp-lpae #1 SMP Debian 5.10.26-1 (2021-03-27) armv7l GNU/Linux root@qemu-bullseye-armhf:~# fgrep -nH BPF /boot/config-5.10.0-5-armmp-lpae /boot/config-5.10.0-5-armmp-lpae:161:CONFIG_CGROUP_BPF=y /boot/config-5.10.0-5-armmp-lpae:189:CONFIG_BPF=y /boot/config-5.10.0-5-armmp-lpae:216:CONFIG_BPF_LSM=y /boot/config-5.10.0-5-armmp-lpae:217:CONFIG_BPF_SYSCALL=y /boot/config-5.10.0-5-armmp-lpae:218:# CONFIG_BPF_JIT_ALWAYS_ON is not set /boot/config-5.10.0-5-armmp-lpae:219:# CONFIG_BPF_PRELOAD is not set /boot/config-5.10.0-5-armmp-lpae:1195:CONFIG_IPV6_SEG6_BPF=y /boot/config-5.10.0-5-armmp-lpae:1335:CONFIG_NETFILTER_XT_MATCH_BPF=m /boot/config-5.10.0-5-armmp-lpae:1554:# CONFIG_BPFILTER is not set /boot/config-5.10.0-5-armmp-lpae:1714:CONFIG_NET_CLS_BPF=m /boot/config-5.10.0-5-armmp-lpae:1741:CONFIG_NET_ACT_BPF=m /boot/config-5.10.0-5-armmp-lpae:1796:CONFIG_BPF_JIT=y /boot/config-5.10.0-5-armmp-lpae:1797:CONFIG_BPF_STREAM_PARSER=y /boot/config-5.10.0-5-armmp-lpae:2013:CONFIG_LWTUNNEL_BPF=y /boot/config-5.10.0-5-armmp-lpae:2021:CONFIG_HAVE_EBPF_JIT=y /boot/config-5.10.0-5-armmp-lpae:7784:# CONFIG_NBPFAXI_DMA is not set /boot/config-5.10.0-5-armmp-lpae:10100:CONFIG_BPF_EVENTS=y /boot/config-5.10.0-5-armmp-lpae:10173:CONFIG_TEST_BPF=m root@qemu-bullseye-armhf:~# In addition, there is little difference between arm64 and armhf in linux kernel config as follows. Is it related with CONFIG_BPF_JIT_DEFAULT_ON??? I have not seen the reported issue on arm64 (nor amd64). ryutaroh@ryutaroh-CFSZ6-1L:/var/tmp/linux-config$ fgrep BPF usr/src/linux-config-5.10/config.armhf_none_armmp-lpae >bpf-armhf.txt ryutaroh@ryutaroh-CFSZ6-1L:/var/tmp/linux-config$ fgrep BPF usr/src/linux-config-5.10/config.arm64_none_arm64 >bpf-arm64.txt ryutaroh@ryutaroh-CFSZ6-1L:/var/tmp/linux-config$ diff -u bpf-arm64.txt bpf-armhf.txt --- bpf-arm64.txt 2021-04-15 09:24:59.596968170 +0900 +++ bpf-armhf.txt 2021-04-15 09:24:40.961010643 +0900 @@ -2,9 +2,7 @@ CONFIG_BPF=y CONFIG_BPF_LSM=y CONFIG_BPF_SYSCALL=y -CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y # CONFIG_BPF_JIT_ALWAYS_ON is not set -CONFIG_BPF_JIT_DEFAULT_ON=y # CONFIG_BPF_PRELOAD is not set CONFIG_IPV6_SEG6_BPF=y CONFIG_NETFILTER_XT_MATCH_BPF=m @@ -15,6 +13,6 @@ CONFIG_BPF_STREAM_PARSER=y CONFIG_LWTUNNEL_BPF=y CONFIG_HAVE_EBPF_JIT=y +# CONFIG_NBPFAXI_DMA is not set CONFIG_BPF_EVENTS=y -# CONFIG_BPF_KPROBE_OVERRIDE is not set CONFIG_TEST_BPF=m Best regards, Ryutaroh
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Control: tags -1 + moreinfo Am 14.04.2021 um 02:33 schrieb Ryutaroh Matsumoto: Package: systemd Version: 247.3-5 Severity: minor Tags: sid bullseye Dear Maintainer, When I do "journalctl -b -g BPF" on Debian Bullseye armhf, I have the following message unseen on arm64 nor amd64. It seems coming from DefaultIPAccounting=yes in /etc/systemd/system.conf.d/ but systemctl status something shows IP accounting information. This is observed both on QEMU VM and a bare-metal hardware (RasPi 4B). Do you have instructions how I can reproduce this (on a amd64 box)?
Bug#986905: systemd armhf: Attaching egress BPF program to cgroup failed
Am 14.04.21 um 02:33 schrieb Ryutaroh Matsumoto: When I do "journalctl -b -g BPF" on Debian Bullseye armhf, Can you attach the output of grep BPF /boot/config-$(uname -r) Sounds a bit like https://github.com/systemd/systemd/issues/7054 which should long be fixed. Michael OpenPGP_signature Description: OpenPGP digital signature
