Public bug reported:

  [Summary]
  A PREEMPT_RT machine hit a general protection fault in timerqueue_add()
  while a userspace thread was arming a futex timeout. The oopsing task
  held the per-CPU hrtimer base lock (IRQs off), so the whole system hung
  silently afterwards (no further logs, no ping) until a manual power cycle.
  The only record is the oops saved in efi-pstore.

  [System]
  Ubuntu 24.04 LTS (noble), amd64
  Kernel: 6.8.1-1015-realtime #16-Ubuntu SMP PREEMPT_RT (linux-image-realtime 
6.8.1-1015.16)
  CPU: Intel Core i5-13400F, 16 GB DDR4 (non-ECC)
  Out-of-tree modules: nvidia 595.99.02 (open kernel module, DKMS)
  Cmdline: quiet splash pci=realloc isolcpus=nohz,domain,managed_irq,1-3
    nohz_full=1-3 rcu_nocbs=1-3 rcu_nocb_poll irqaffinity=0,4-9
    nosoftlockup skew_tick=1
  Workload: ROS 2 (Fast DDS, shared memory transport) robot control at
  500 Hz, Docker containers, several GPU inference processes.

  [Oops]
  Uptime 789 s, CPU 5 (not an isolated CPU).

  general protection fault, probably for non-canonical address 
0xb8ffffbff4644e95: 0000 [#1] PREEMPT_RT SMP NOPTI
  CPU: 5 PID: 4189 Comm: mara_grpc_gatew Tainted: G           OE      
6.8.1-1015-realtime #16-Ubuntu
  RIP: 0010:timerqueue_add+0x2e/0xd0
  RSP: 0018:ffffbff44dd9f910 EFLAGS: 00010086
  RAX: b8ffffbff4644e7d RBX: ffffbff44dd9fb18 RCX: b8ffffbff4644e8d
  RDX: b8ffffbff4644e7d RSI: 000000b7b41bd3f0 RDI: ffff9e0eafaa4e68
  Call Trace:
   enqueue_hrtimer+0x4d/0xc0
   __hrtimer_start_range_ns+0x232/0x370
   hrtimer_start_range_ns+0x79/0xf0
   hrtimer_sleeper_start_expires+0x24/0x40
   futex_wait_queue+0x50/0xa0
   __futex_wait+0xae/0x120
   futex_wait+0x74/0x120
   do_futex+0x105/0x260
   __x64_sys_futex+0x12a/0x200
   do_syscall_64+0x7f/0x180

  Full pstore dmesg is attached.

  [Analysis]
  The fault is in the rbtree descent loop of timerqueue_add(), at
  "cmp 0x18(%rax),%rsi" with RAX = 0xb8ffffbff4644e7d. RAX == RDX and
  R12 == 0, so at least one node had already been visited and RAX was
  loaded from a child pointer of the previous node.

  The bad value looks like a valid on-stack hrtimer node address
  (0xffffbff4644e7dXX, same vmalloc stack area as the current task's
  stack) read one byte off: the upper 7 bytes of the pointer followed by
  the low byte of the adjacent field. That is what you get if the
  previous node pointer had bit 0 set (node address + 1), i.e. a single
  bit, the rb colour bit position, set in a child/root pointer of the
  per-CPU hrtimer timerqueue.

  This would be consistent with an on-stack hrtimer_sleeper still being
  linked in the timerqueue after its stack frame was reused, but a
  single-bit memory error cannot be excluded (non-ECC RAM, memtest not
  yet run). There were no MCE, EDAC, Xid or other kernel warnings before
  the oops in this boot, and no other kernel oops in the previous weeks.

  [Possibly related]
  LP: #2143849 (6.8.0-101-generic, NULL deref in rb_insert_color via
  timerqueue_add/enqueue_hrtimer, CPU then stuck on the hrtimer base
  lock). Expired without a root cause.

  [Reproducibility]
  Seen once. No known reproducer. kdump is now configured so a vmcore
  will be available if it happens again.

  [Question]
  Is there a known fix in the hrtimer/futex sleeper path that is missing
  from 6.8.1-1015.16? This is the latest linux-image-realtime available
  in noble-updates/universe.

** Affects: linux-realtime (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2175025

Title:
  general protection fault in timerqueue_add (hrtimer rbtree corruption)
  on 6.8.1-1015-realtime, system hangs silently

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-realtime/+bug/2175025/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to