Update after further investigation: the initrdless boot path described in the
original report is NOT the trigger — it only correlated on the first machine.
The actual cause is KHO (Kexec HandOver).
This kernel is built with:
CONFIG_KEXEC_HANDOVER=y
CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y
CONFIG_MEMBLOCK_KHO_SCRATCH=y
so KHO attempts to reserve its scratch area on every boot, without any kho=
parameter on the command line. On a 512MB t3a.nano the reservation is ~316MB
(~62% of RAM). When the reservation SUCCEEDS, the system boots with only ~75MB
usable and OOM-kills itself into an unusable state within ~2 minutes. When it
FAILS, KHO disables itself and the system is completely healthy. Whether it
succeeds depends on the early-boot memblock layout (KASLR kernel placement,
initrd size/position), so a 512MB instance surviving a reboot is a per-boot
lottery.
Evidence — three boots, two identically-specced t3a.nano instances (identical
BIOS-e820 maps, same Ubuntu 24.04.4, same 7.0.0-1009-aws kernel):
1) Failing boot (machine A, 06:55 UTC — the boot from the original report).
No "KHO:" message appears in the log, i.e. the scratch reservation
succeeded:
Memory: 74904K/503328K available (... 422584K reserved, 0K cma-
reserved)
Runtime state from the first OOM report shows the reserved memory exposed
as CMA-type pages that unmovable GFP_KERNEL allocations cannot use — while
the boot banner said "0K cma-reserved" and healthy boots show CmaTotal 0:
Node 0 DMA32 free:317020kB ... managed:428984kB ... free_cma:315604kB
active_anon:0 inactive_anon:15 ... all_unreclaimable? yes
2) Healthy boot of the SAME machine A (14:01 UTC, identical cmdline apart
from GRUB's panic=-1 fast-path flag):
KHO: Failed to reserve lowmem scratch buffer
KHO: Failed to reserve scratch area, disabling kexec handover
Memory: 349536K/503324K available (... 148204K reserved, 0K cma-reserved)
3) Machine B, same day, booted via the same GRUB initrdless fast path that
machine A's failing boot used (RAMDISK is exactly the 15,106,048-byte
microcode.cpio) — healthy, again only because the reservation failed:
KHO: Failed to reserve nid 0 scratch buffer
KHO: Failed to reserve scratch area, disabling kexec handover
Memory: 412636K/503328K available (... 84664K reserved, 0K cma-reserved)
Boot (3) disproves the initrdless theory from the original report: the
initrd choice merely shifts the memory layout enough to change whether KHO's
reservation succeeds.
Additional notes:
- Neither machine has kexec-tools or kdump-tools installed; nothing here uses
kexec, so the reservation buys nothing on these systems.
- One machine was dist-upgraded from an earlier release, the other freshly
installed from a 24.04 cloud image — identical behavior, so this is not
config drift.
- The previous kernel (6.17.0-1019-aws) had no issue on the same instances.
Impact assessment: because CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y ships in
noble-updates/noble-security via the linux-aws HWE track, and unattended-
upgrades reboots into this kernel automatically, small-memory instances
(512MB) can take themselves down overnight, intermittently and without any
userspace cause. The scratch sizing (~62% of RAM on a 512MB machine) seems
disproportionate for small systems; either the default-on packaging decision
or the upstream sizing/failure policy needs revisiting for low-memory
machines.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161986
Title:
7.0.0-1009-aws reserves 422MB of 503MB at boot on t3a.nano when
booting initrdless (GRUB_FORCE_PARTUUID), causing immediate system-
wide OOM
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws-7.0/+bug/2161986/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
