I have verified this in noble proposed and validated that it fixes the
bug.
** Description changed:
- Impact]
+ [Impact]
On systems that utilize both uprobes and perf_events style profiling, it
is possible to hit a panic in the uprobe_free_utask code. This occurs
during process exit. If the profiler fires while uprobe_free_utask is
in the process of cleaning up the utask, the NMI may read freed memory
because the cleanup code frees the utask before setting its pointer to
NULL. This submitter has encountered the problem on systems running
workloads without intentionally trying to trigger the problem.
The stacks look something like this:
- RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
- ...
- ? die_addr+0x36/0x90
- ? exc_general_protection+0x217/0x420
- ? asm_exc_general_protection+0x26/0x30
- ? is_uprobe_at_func_entry+0x28/0x80
- perf_callchain_user+0x20a/0x360
- get_perf_callchain+0x147/0x1d0
- bpf_get_stackid+0x60/0x90
- bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
- ? __smp_call_single_queue+0xad/0x120
- bpf_overflow_handler+0x75/0x110
- ...
- asm_sysvec_apic_timer_interrupt+0x1a/0x20
- RIP: 0010:__kmem_cache_free+0x1cb/0x350
- ...
- ? uprobe_free_utask+0x62/0x80
- ? acct_collect+0x4c/0x220
- uprobe_free_utask+0x62/0x80
- mm_release+0x12/0xb0
- do_exit+0x26b/0xaa0
- __x64_sys_exit+0x1b/0x20
- do_syscall_64+0x5a/0x80
+ RIP: 0010:is_uprobe_at_func_entry+0x28/0x80
+ ...
+ ? die_addr+0x36/0x90
+ ? exc_general_protection+0x217/0x420
+ ? asm_exc_general_protection+0x26/0x30
+ ? is_uprobe_at_func_entry+0x28/0x80
+ perf_callchain_user+0x20a/0x360
+ get_perf_callchain+0x147/0x1d0
+ bpf_get_stackid+0x60/0x90
+ bpf_prog_9aac297fb833e2f5_do_perf_event+0x434/0x53b
+ ? __smp_call_single_queue+0xad/0x120
+ bpf_overflow_handler+0x75/0x110
+ ...
+ asm_sysvec_apic_timer_interrupt+0x1a/0x20
+ RIP: 0010:__kmem_cache_free+0x1cb/0x350
+ ...
+ ? uprobe_free_utask+0x62/0x80
+ ? acct_collect+0x4c/0x220
+ uprobe_free_utask+0x62/0x80
+ mm_release+0x12/0xb0
+ do_exit+0x26b/0xaa0
+ __x64_sys_exit+0x1b/0x20
+ do_syscall_64+0x5a/0x80
The person who reported the issue upstream provided this reproducer.
(Run each command in a separate terminal):
- # while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c
ls; done
- # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
+ # while :; do bpftrace -e 'uprobe:/bin/ls:_start { printf("hit\n"); }' -c
ls; done
+ # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
However, since the binutils are stripped on some of the releases where I
tested this, I ran the following instead:
- # while :; do bpftrace -e 'uprobe:libc:malloc { printf("hit\n"); }' -c ls;
done
- # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
+ # while :; do bpftrace -e 'uprobe:libc:malloc { printf("hit\n"); }' -c ls;
done
+ # bpftrace -e 'profile:hz:100000 { @[ustack()] = count(); }'
[Backport]
The fix is upstream as commit b583ef82b671 ("uprobes: Fix race in
uprobe_free_utask")
However this patch was massaged by stable for its inclusion in 6.12,
6.6, and 6.1. Instead of re-doing stable's conflict resolution, take
the patch directly from 6.6.x instead, at commit eff00c5e29ab.
This patch is in stable as of 6.12.19, 6.6.83, and 6.1.131.
[Test]
I've run the provided reproducer and validated that I can reproduce the
problem without the patch applied and that I cannot reproduce it again
once I have applied the patch.
[Potential Regression]
The regression potential here seems quite low. The fix has been
upstream for a couple releases and no subsequent issues have been
reported. It makes no functional change beyond ensuring that the utask
pointer is set to NULL before the utask structure itself is freed. The
dereference and free occur on the same cpu.
** Tags removed: verification-needed-noble-linux
** Tags added: verification-done-noble-linux
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2104210
Title:
uprobe-related panics during profiling
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2104210/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
