Public bug reported:
[Summary]
A PREEMPT_RT machine hit a general protection fault in timerqueue_add()
while a userspace thread was arming a futex timeout. The oopsing task
held the per-CPU hrtimer base lock (IRQs off), so the whole system hung
silently afterwards (no further logs, no ping) until a manual power cycle.
The only record is the oops saved in efi-pstore.
[System]
Ubuntu 24.04 LTS (noble), amd64
Kernel: 6.8.1-1015-realtime #16-Ubuntu SMP PREEMPT_RT (linux-image-realtime
6.8.1-1015.16)
CPU: Intel Core i5-13400F, 16 GB DDR4 (non-ECC)
Out-of-tree modules: nvidia 595.99.02 (open kernel module, DKMS)
Cmdline: quiet splash pci=realloc isolcpus=nohz,domain,managed_irq,1-3
nohz_full=1-3 rcu_nocbs=1-3 rcu_nocb_poll irqaffinity=0,4-9
nosoftlockup skew_tick=1
Workload: ROS 2 (Fast DDS, shared memory transport) robot control at
500 Hz, Docker containers, several GPU inference processes.
[Oops]
Uptime 789 s, CPU 5 (not an isolated CPU).
general protection fault, probably for non-canonical address
0xb8ffffbff4644e95: 0000 [#1] PREEMPT_RT SMP NOPTI
CPU: 5 PID: 4189 Comm: mara_grpc_gatew Tainted: G OE
6.8.1-1015-realtime #16-Ubuntu
RIP: 0010:timerqueue_add+0x2e/0xd0
RSP: 0018:ffffbff44dd9f910 EFLAGS: 00010086
RAX: b8ffffbff4644e7d RBX: ffffbff44dd9fb18 RCX: b8ffffbff4644e8d
RDX: b8ffffbff4644e7d RSI: 000000b7b41bd3f0 RDI: ffff9e0eafaa4e68
Call Trace:
enqueue_hrtimer+0x4d/0xc0
__hrtimer_start_range_ns+0x232/0x370
hrtimer_start_range_ns+0x79/0xf0
hrtimer_sleeper_start_expires+0x24/0x40
futex_wait_queue+0x50/0xa0
__futex_wait+0xae/0x120
futex_wait+0x74/0x120
do_futex+0x105/0x260
__x64_sys_futex+0x12a/0x200
do_syscall_64+0x7f/0x180
Full pstore dmesg is attached.
[Analysis]
The fault is in the rbtree descent loop of timerqueue_add(), at
"cmp 0x18(%rax),%rsi" with RAX = 0xb8ffffbff4644e7d. RAX == RDX and
R12 == 0, so at least one node had already been visited and RAX was
loaded from a child pointer of the previous node.
The bad value looks like a valid on-stack hrtimer node address
(0xffffbff4644e7dXX, same vmalloc stack area as the current task's
stack) read one byte off: the upper 7 bytes of the pointer followed by
the low byte of the adjacent field. That is what you get if the
previous node pointer had bit 0 set (node address + 1), i.e. a single
bit, the rb colour bit position, set in a child/root pointer of the
per-CPU hrtimer timerqueue.
This would be consistent with an on-stack hrtimer_sleeper still being
linked in the timerqueue after its stack frame was reused, but a
single-bit memory error cannot be excluded (non-ECC RAM, memtest not
yet run). There were no MCE, EDAC, Xid or other kernel warnings before
the oops in this boot, and no other kernel oops in the previous weeks.
[Possibly related]
LP: #2143849 (6.8.0-101-generic, NULL deref in rb_insert_color via
timerqueue_add/enqueue_hrtimer, CPU then stuck on the hrtimer base
lock). Expired without a root cause.
[Reproducibility]
Seen once. No known reproducer. kdump is now configured so a vmcore
will be available if it happens again.
[Question]
Is there a known fix in the hrtimer/futex sleeper path that is missing
from 6.8.1-1015.16? This is the latest linux-image-realtime available
in noble-updates/universe.
** Affects: linux-realtime (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2175025
Title:
general protection fault in timerqueue_add (hrtimer rbtree corruption)
on 6.8.1-1015-realtime, system hangs silently
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-realtime/+bug/2175025/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs