** Attachment removed: "CurrentDmesg.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998061/+files/CurrentDmesg.txt
** Attachment removed: "Dependencies.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998062/+files/Dependencies.txt ** Attachment removed: "Lspci.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998063/+files/Lspci.txt ** Attachment removed: "Lspci-vt.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998064/+files/Lspci-vt.txt ** Attachment removed: "Lsusb-v.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998065/+files/Lsusb-v.txt ** Attachment removed: "ProcCpuinfo.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998066/+files/ProcCpuinfo.txt ** Attachment removed: "ProcEnviron.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998068/+files/ProcEnviron.txt ** Attachment removed: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998067/+files/ProcCpuinfoMinimal.txt ** Attachment removed: "ProcInterrupts.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998069/+files/ProcInterrupts.txt ** Attachment removed: "acpidump.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998074/+files/acpidump.txt ** Attachment removed: "modified.conffile..etc.systemd.journald.conf.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998075/+files/modified.conffile..etc.systemd.journald.conf.txt ** Attachment removed: "ProcModules.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998070/+files/ProcModules.txt ** Attachment removed: "SystemdDelta.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998071/+files/SystemdDelta.txt ** Attachment removed: "SystemdFailedUnits.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998072/+files/SystemdFailedUnits.txt ** Attachment removed: "UdevDb.txt" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+attachment/5998073/+files/UdevDb.txt ** Description changed: # System hang caused by systemd-logind wall broadcast retry loop into privileged LXD containers (AppArmor mount-rootfs denial) ## Summary On a scheduled reboot (triggered by unattended-upgrades after a kernel update), the host completely froze ~26 minutes before the actual reboot time. The cause appears to be an unbounded retry loop: systemd-logind's periodic wall-broadcast mechanism (announcing "system will reboot at HH:MM") attempts to enter the mount namespace of privileged LXD containers to deliver the notification, via a bind-mount of `/` to `/run/systemd/mount-rootfs/`. This bind-mount is denied by the LXD "privileged container" AppArmor profile. Instead of failing once, the attempt appears to retry continuously and rapidly (observed ~1 attempt every 3 seconds, 205 occurrences in a 10-minute window, with the spawning PID incrementing continuously), eventually exhausting host resources and freezing the system entirely — with no further kernel logging of any kind (no OOM, no panic, no MCE) after the freeze point. ## Environment - Ubuntu 24.04.x (noble), server install - Kernel: upgraded 6.8.0-138-generic -> 6.8.0-139-generic via unattended-upgrades just prior to the incident - LXD (snap), two containers involved: both configured with `security.privileged: "true"` and GPU passthrough (`gpu0` device) - Reboot was scheduled by unattended-upgrades ("Reboot scheduled for Sun 2026-09-06 06:00:00 CEST") ## Timeline of the incident - 05:20:00 — systemd-logind logs: "The system will reboot at Sun 2026-09-06 06:00:00 CEST!" (the standard 40-minutes-before-reboot announcement) - 05:25:00 to 05:34:05 — repeated kernel audit denials, escalating in frequency, e.g.: - ``` - audit: type=1400 audit(...): apparmor="DENIED" operation="mount" class="mount" - info="failed perms check" error=-13 - profile="lxd-<container>_</var/snap/lxd/common/lxd>" - name="/run/systemd/mount-rootfs/" pid=<incrementing PID> comm="(d-logind)" - srcname="/" flags="rw, rbind" - ``` - 205 such denials counted in the 05:25–05:35 window alone, for both containers, with PIDs increasing continuously (~1000 new PIDs over ~9 minutes) — i.e., a tight retry loop, not periodic isolated attempts. + ``` + audit: type=1400 audit(...): apparmor="DENIED" operation="mount" class="mount" + info="failed perms check" error=-13 + profile="lxd-<container>_</var/snap/lxd/common/lxd>" + name="/run/systemd/mount-rootfs/" pid=<incrementing PID> comm="(d-logind)" + srcname="/" flags="rw, rbind" + ``` + 205 such denials counted in the 05:25–05:35 window alone, for both containers, with PIDs increasing continuously (~1000 new PIDs over ~9 minutes) — i.e., a tight retry loop, not periodic isolated attempts. - 05:34:05 — last kernel log entry of any kind. All logging stops abruptly (network drop logging via nftables, which had been continuous throughout the boot, also stops at the exact same second). - The scheduled reboot at 06:00:00 never actually occurred — the host was completely unresponsive (no SSH, no Samba, no web services) until it was manually power-cycled several hours later. `journalctl --list-boots` shows a ~6.5 hour gap with zero log activity between the last entry of the frozen boot and the first entry of the next (forced) boot. ## What has been ruled out Over the course of investigation the following were checked and excluded as the cause: - ZFS: not in use (`zpool status` → "no pools available"); storage is mdadm RAID1, not ZFS. - mdadm RAID check: no scheduled check was running at the time (next `mdcheck` timer only fired later that day, after the incident). - Disk/SATA/SCSI errors: none found in kernel log for the affected boot. - - NVIDIA/DKMS: module correctly built and installed for the new kernel (`dkms status` confirms `installed`); no dependency-broken packages (`apt-get install -f` reports nothing to fix). - MCE / kernel panic / oops / GPU Xid errors: none found anywhere in the kernel log for the affected boot. - The AppArmor "mount" denials themselves are otherwise a long-standing, apparently harmless, occurrence for these two containers (13,000+ occurrences on a normal, healthy boot) — the issue is not the denial itself but its behavior specifically in the context of the logind wall-broadcast mechanism around the scheduled reboot window. ## Workaround found Adding an explicit AppArmor allow rule for the two privileged containers stops the retry storm by letting the mount succeed instead of being denied and retried: ``` lxc config set <container> raw.apparmor "mount options=(rw, rbind) -> /run/systemd/mount-rootfs/," ``` ## Expected behaviour A denied namespace-entry attempt by systemd-logind (for the purpose of delivering a wall-style shutdown notification) should fail gracefully and not be retried in an unbounded/high-frequency loop that can exhaust host resources and freeze the entire machine. ## Actual behaviour The denied attempt appears to retry very rapidly and continuously (new PID for each attempt), for roughly 9+ minutes straight, ending in a complete, unrecoverable system freeze with no further kernel/system logging. - - ProblemType: Bug - DistroRelease: Ubuntu 24.04 - Package: systemd 255.4-1ubuntu8.17 - ProcVersionSignature: Ubuntu 6.8.0-139.139-generic 6.8.12 - Uname: Linux 6.8.0-139-generic x86_64 - NonfreeKernelModules: zfs - ApportVersion: 2.28.1-0ubuntu3.8 - Architecture: amd64 - CasperMD5CheckResult: pass - Date: Sun Sep 6 22:14:38 2026 - InstallationDate: Installed on 2024-07-28 (770 days ago) - InstallationMedia: Ubuntu-Server 24.04 LTS "Noble Numbat" - Release amd64 (20240423) - Lsusb: - Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub - Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub - Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub - Bus 003 Device 002: ID 051d:0002 American Power Conversion Uninterruptible Power Supply - Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub - Lsusb-t: - /: Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/10p, 480M - /: Bus 002.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M - /: Bus 003.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 480M - |__ Port 002: Dev 002, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M - /: Bus 004.Port 001: Dev 001, Class=root_hub, Driver=xhci_hcd/4p, 10000M - MachineType: ASUS System Product Name - ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.8.0-139-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro - SourcePackage: systemd - UpgradeStatus: No upgrade log present (probably fresh install) - dmi.bios.date: 09/10/2025 - dmi.bios.release: 5.17 - dmi.bios.vendor: American Megatrends Inc. - dmi.bios.version: 4001 - dmi.board.asset.tag: Default string - dmi.board.name: PRIME B550M-K - dmi.board.vendor: ASUSTeK COMPUTER INC. - dmi.board.version: Rev X.0x - dmi.chassis.asset.tag: Default string - dmi.chassis.type: 3 - dmi.chassis.vendor: Default string - dmi.chassis.version: Default string - dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr4001:bd09/10/2025:br5.17:svnASUS:pnSystemProductName:pvrSystemVersion:rvnASUSTeKCOMPUTERINC.:rnPRIMEB550M-K:rvrRevX.0x:cvnDefaultstring:ct3:cvrDefaultstring:skuSKU: - dmi.product.family: To be filled by O.E.M. - dmi.product.name: System Product Name - dmi.product.sku: SKU - dmi.product.version: System Version - dmi.sys.vendor: ASUS - mtime.conffile..etc.systemd.journald.conf: 2025-05-01T15:01:58.487662 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2166624 Title: System hang caused by systemd-logind wall broadcast retry loop into privileged LXD containers To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2166624/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
