Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Sun, Jan 28, 2024 at 04:51:53PM +0900, Masami Hiramatsu wrote:
> On Sat, 27 Jan 2024 19:44:06 +0100
> Jiri Olsa wrote:
>
> > On Sat, Jan 27, 2024 at 12:14:05AM +0900, Masami Hiramatsu wrote:
> > > On Thu, 25 Jan 2024 15:54:53 +0100
> > > Jiri Olsa wrote:
> > >
> > > > On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google)
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > Here is the 6th version of the series to re-implement the fprobe on
> > > > > function-graph tracer. The previous version is;
> > > > >
> > > > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/
> > > > >
> > > > > This version fixes use-after-unregister bug and arm64 stack unwinding
> > > > > bug [13/36], add an improvement for multiple interrupts during push
> > > > > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using
> > > > > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new
> > > > > fprobe can switch to SAVE_ARGS[32/36] safely.
> > > > > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which
> > > > > should be pushed separatedly as a stable bugfix.
> > > > >
> > > > > There are some TODOs:
> > > > > - Add s390x and loongarch support to fprobe (multiple fgraph).
> > > > > - Fix to get the symbol address from ftrace entry address on arm64.
> > > > >(This should be done in BPF trace event)
> > > > > - Cleanup code, rename some terms(offset/index) and
> > > > > FGRAPH_TYPE_BITMAP
> > > > >part should be merged to FGRAPH_TYPE_ARRAY patch.
> > > >
> > > > hi,
> > > > I'm getting kasan bugs below when running bpf selftests on top of this
> > > > patchset.. I think it's probably the reason I see failures in some bpf
> > > > kprobe_multi/fprobe tests
> > > >
> > > > so far I couldn't find the reason.. still checking ;-)
> > >
> > > Thanks for reporting! Have you built the kernel with debuginfo? In that
> > > case, can you decode the line from the address?
> > >
> > > $ eu-addr2line -fi -e vmlinux ftrace_push_return_trace.isra.0+0x346
> > >
> > > This helps me a lot.
> >
> > I had to recompile/regenerate the fault, it points in here:
> >
> > 8149b390 :
> > ...
> >
> > current->ret_stack[rindex - 1] = val;
> > 8149b6b1: 48 8d bd 78 28 00 00lea
> > 0x2878(%rbp),%rdi
> > 8149b6b8: e8 63 e4 28 00 call
> > 81729b20 <__asan_load8>
> > 8149b6bd: 48 8b 95 78 28 00 00mov
> > 0x2878(%rbp),%rdx
> > 8149b6c4: 41 8d 47 ff lea
> > -0x1(%r15),%eax
> > 8149b6c8: 48 98 cltq
> > 8149b6ca: 4c 8d 24 c2 lea
> > (%rdx,%rax,8),%r12
> > 8149b6ce: 4c 89 e7mov%r12,%rdi
> > 8149b6d1: e8 ea e4 28 00 call
> > 81729bc0 <__asan_store8>
> > --->8149b6d6: 49 89 1c 24 mov%rbx,(%r12)
> > current->curr_ret_stack = index = rindex;
> > 8149b6da: 48 8d bd 6c 28 00 00lea
> > 0x286c(%rbp),%rdi
> > 8149b6e1: e8 9a e3 28 00 call
> > 81729a80 <__asan_store4>
> > 8149b6e6: 44 89 bd 6c 28 00 00mov
> > %r15d,0x286c(%rbp)
> > 8149b6ed: e9 8d fd ff ff jmp
> > 8149b47f
> > if (WARN_ON_ONCE(idx <= 0))
> >
>
> Thanks! So this shows that this bug is failed to check the boundary of
> shadow stack while pushing the return trace.
>
> diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
> index 0f11f80bdd6c..8e1fcc3f4bda 100644
> --- a/kernel/trace/fgraph.c
> +++ b/kernel/trace/fgraph.c
> @@ -550,7 +550,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long
> func,
> smp_rmb();
>
> /* The return trace stack is full */
> - if (current->curr_ret_stack + FGRAPH_RET_INDEX >=
> SHADOW_STACK_MAX_INDEX) {
> + if (current->curr_ret_stack + FGRAPH_RET_INDEX + 1 >=
> SHADOW_STACK_MAX_INDEX) {
> atomic_inc(¤t->trace_overrun);
> return -EBUSY;
> }
>
> Sorry, I forgot to increment the space for reserved entry...
hum, I'm getting same error even with the change above, same backtrace/line
jirka
Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Sat, 27 Jan 2024 19:44:06 +0100
Jiri Olsa wrote:
> On Sat, Jan 27, 2024 at 12:14:05AM +0900, Masami Hiramatsu wrote:
> > On Thu, 25 Jan 2024 15:54:53 +0100
> > Jiri Olsa wrote:
> >
> > > On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google) wrote:
> > > > Hi,
> > > >
> > > > Here is the 6th version of the series to re-implement the fprobe on
> > > > function-graph tracer. The previous version is;
> > > >
> > > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/
> > > >
> > > > This version fixes use-after-unregister bug and arm64 stack unwinding
> > > > bug [13/36], add an improvement for multiple interrupts during push
> > > > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using
> > > > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new
> > > > fprobe can switch to SAVE_ARGS[32/36] safely.
> > > > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which
> > > > should be pushed separatedly as a stable bugfix.
> > > >
> > > > There are some TODOs:
> > > > - Add s390x and loongarch support to fprobe (multiple fgraph).
> > > > - Fix to get the symbol address from ftrace entry address on arm64.
> > > >(This should be done in BPF trace event)
> > > > - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP
> > > >part should be merged to FGRAPH_TYPE_ARRAY patch.
> > >
> > > hi,
> > > I'm getting kasan bugs below when running bpf selftests on top of this
> > > patchset.. I think it's probably the reason I see failures in some bpf
> > > kprobe_multi/fprobe tests
> > >
> > > so far I couldn't find the reason.. still checking ;-)
> >
> > Thanks for reporting! Have you built the kernel with debuginfo? In that
> > case, can you decode the line from the address?
> >
> > $ eu-addr2line -fi -e vmlinux ftrace_push_return_trace.isra.0+0x346
> >
> > This helps me a lot.
>
> I had to recompile/regenerate the fault, it points in here:
>
> 8149b390 :
> ...
>
> current->ret_stack[rindex - 1] = val;
> 8149b6b1: 48 8d bd 78 28 00 00lea
> 0x2878(%rbp),%rdi
> 8149b6b8: e8 63 e4 28 00 call
> 81729b20 <__asan_load8>
> 8149b6bd: 48 8b 95 78 28 00 00mov
> 0x2878(%rbp),%rdx
> 8149b6c4: 41 8d 47 ff lea-0x1(%r15),%eax
> 8149b6c8: 48 98 cltq
> 8149b6ca: 4c 8d 24 c2 lea
> (%rdx,%rax,8),%r12
> 8149b6ce: 4c 89 e7mov%r12,%rdi
> 8149b6d1: e8 ea e4 28 00 call
> 81729bc0 <__asan_store8>
> --->8149b6d6: 49 89 1c 24 mov%rbx,(%r12)
> current->curr_ret_stack = index = rindex;
> 8149b6da: 48 8d bd 6c 28 00 00lea
> 0x286c(%rbp),%rdi
> 8149b6e1: e8 9a e3 28 00 call
> 81729a80 <__asan_store4>
> 8149b6e6: 44 89 bd 6c 28 00 00mov
> %r15d,0x286c(%rbp)
> 8149b6ed: e9 8d fd ff ff jmp
> 8149b47f
> if (WARN_ON_ONCE(idx <= 0))
>
Thanks! So this shows that this bug is failed to check the boundary of
shadow stack while pushing the return trace.
diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
index 0f11f80bdd6c..8e1fcc3f4bda 100644
--- a/kernel/trace/fgraph.c
+++ b/kernel/trace/fgraph.c
@@ -550,7 +550,7 @@ ftrace_push_return_trace(unsigned long ret, unsigned long
func,
smp_rmb();
/* The return trace stack is full */
- if (current->curr_ret_stack + FGRAPH_RET_INDEX >=
SHADOW_STACK_MAX_INDEX) {
+ if (current->curr_ret_stack + FGRAPH_RET_INDEX + 1 >=
SHADOW_STACK_MAX_INDEX) {
atomic_inc(¤t->trace_overrun);
return -EBUSY;
}
Sorry, I forgot to increment the space for reserved entry...
Thanks,
>
> the dump is attached below (same address as in previous email)
>
> jirka
>
>
> ---
> [ 360.152200][C3] BUG: KASAN: slab-out-of-bounds in
> ftrace_push_return_trace.isra.0+0x346/0x370
> [ 360.153195][C3] Write of size 8 at addr 8881a0e10ff8 by task
> kworker/3:4/728
> [ 360.154101][C3]
> [ 360.154414][C3] CPU: 3 PID: 728 Comm: kworker/3:4 Tainted: G
> OE 6.7.0+ #316 c9b0d53b3491b547d06b6b50629b74711600ddc9
> [ 360.155679][C3] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> BIOS 1.16.2-1.fc38 04/01/2014
> [ 360.156611][C3] Workqueue: events free_obj_work
> [ 360.157175][C3] Call Trace:
> [ 360.157561][C3]
> [ 360.157904][C3] dump_stack_lvl+0xf6/0x180
> [ 360.158404][C3] print_report+0xc4/0x610
> [ 360.158853][C3] ? lock_release+0xba/0x760
> [ 360.159375][C3] ? __phys_addr+0x5/0x80
> [
Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Sat, Jan 27, 2024 at 12:14:05AM +0900, Masami Hiramatsu wrote: > On Thu, 25 Jan 2024 15:54:53 +0100 > Jiri Olsa wrote: > > > On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google) wrote: > > > Hi, > > > > > > Here is the 6th version of the series to re-implement the fprobe on > > > function-graph tracer. The previous version is; > > > > > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/ > > > > > > This version fixes use-after-unregister bug and arm64 stack unwinding > > > bug [13/36], add an improvement for multiple interrupts during push > > > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using > > > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new > > > fprobe can switch to SAVE_ARGS[32/36] safely. > > > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which > > > should be pushed separatedly as a stable bugfix. > > > > > > There are some TODOs: > > > - Add s390x and loongarch support to fprobe (multiple fgraph). > > > - Fix to get the symbol address from ftrace entry address on arm64. > > >(This should be done in BPF trace event) > > > - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP > > >part should be merged to FGRAPH_TYPE_ARRAY patch. > > > > hi, > > I'm getting kasan bugs below when running bpf selftests on top of this > > patchset.. I think it's probably the reason I see failures in some bpf > > kprobe_multi/fprobe tests > > > > so far I couldn't find the reason.. still checking ;-) > > Thanks for reporting! Have you built the kernel with debuginfo? In that > case, can you decode the line from the address? > > $ eu-addr2line -fi -e vmlinux ftrace_push_return_trace.isra.0+0x346 > > This helps me a lot. I had to recompile/regenerate the fault, it points in here: 8149b390 : ... current->ret_stack[rindex - 1] = val; 8149b6b1: 48 8d bd 78 28 00 00lea0x2878(%rbp),%rdi 8149b6b8: e8 63 e4 28 00 call 81729b20 <__asan_load8> 8149b6bd: 48 8b 95 78 28 00 00mov0x2878(%rbp),%rdx 8149b6c4: 41 8d 47 ff lea-0x1(%r15),%eax 8149b6c8: 48 98 cltq 8149b6ca: 4c 8d 24 c2 lea (%rdx,%rax,8),%r12 8149b6ce: 4c 89 e7mov%r12,%rdi 8149b6d1: e8 ea e4 28 00 call 81729bc0 <__asan_store8> --->8149b6d6: 49 89 1c 24 mov%rbx,(%r12) current->curr_ret_stack = index = rindex; 8149b6da: 48 8d bd 6c 28 00 00lea0x286c(%rbp),%rdi 8149b6e1: e8 9a e3 28 00 call 81729a80 <__asan_store4> 8149b6e6: 44 89 bd 6c 28 00 00mov %r15d,0x286c(%rbp) 8149b6ed: e9 8d fd ff ff jmp8149b47f if (WARN_ON_ONCE(idx <= 0)) the dump is attached below (same address as in previous email) jirka --- [ 360.152200][C3] BUG: KASAN: slab-out-of-bounds in ftrace_push_return_trace.isra.0+0x346/0x370 [ 360.153195][C3] Write of size 8 at addr 8881a0e10ff8 by task kworker/3:4/728 [ 360.154101][C3] [ 360.154414][C3] CPU: 3 PID: 728 Comm: kworker/3:4 Tainted: G OE 6.7.0+ #316 c9b0d53b3491b547d06b6b50629b74711600ddc9 [ 360.155679][C3] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014 [ 360.156611][C3] Workqueue: events free_obj_work [ 360.157175][C3] Call Trace: [ 360.157561][C3] [ 360.157904][C3] dump_stack_lvl+0xf6/0x180 [ 360.158404][C3] print_report+0xc4/0x610 [ 360.158853][C3] ? lock_release+0xba/0x760 [ 360.159375][C3] ? __phys_addr+0x5/0x80 [ 360.159872][C3] ? __phys_addr+0x33/0x80 [ 360.161309][C3] kasan_report+0xbe/0xf0 [ 360.161940][C3] ? ftrace_push_return_trace.isra.0+0x346/0x370 [ 360.162817][C3] ? ftrace_push_return_trace.isra.0+0x346/0x370 [ 360.163518][C3] ? __pfx_kernel_text_address+0x10/0x10 [ 360.164152][C3] ? __kernel_text_address+0xe/0x40 [ 360.164715][C3] ftrace_push_return_trace.isra.0+0x346/0x370 [ 360.165324][C3] ? __pfx_kernel_text_address+0x10/0x10 [ 360.165940][C3] function_graph_enter_ops+0xbb/0x2d0 [ 360.166555][C3] ? __kernel_text_address+0xe/0x40 [ 360.167134][C3] ? __pfx_function_graph_enter_ops+0x10/0x10 [ 360.167801][C3] ? __pfx_function_graph_enter_ops+0x10/0x10 [ 360.168454][C3] ? __pfx___kernel_text_address+0x10/0x10 [ 360.169086][C3] ? __pfx_unwind_get_return_address+0x10/0x10 [ 360.169781][C3] ftrace_graph_func+0x142/0x270 [ 360.170341][C3] ? __pfx_kernel_text_address+0x10/0x10 [ 360.170960][C3] ? o
Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Sat, 27 Jan 2024 00:14:05 +0900 Masami Hiramatsu (Google) wrote: > On Thu, 25 Jan 2024 15:54:53 +0100 > Jiri Olsa wrote: > > > On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google) wrote: > > > Hi, > > > > > > Here is the 6th version of the series to re-implement the fprobe on > > > function-graph tracer. The previous version is; > > > > > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/ > > > > > > This version fixes use-after-unregister bug and arm64 stack unwinding > > > bug [13/36], add an improvement for multiple interrupts during push > > > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using > > > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new > > > fprobe can switch to SAVE_ARGS[32/36] safely. > > > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which > > > should be pushed separatedly as a stable bugfix. > > > > > > There are some TODOs: > > > - Add s390x and loongarch support to fprobe (multiple fgraph). > > > - Fix to get the symbol address from ftrace entry address on arm64. > > >(This should be done in BPF trace event) > > > - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP > > >part should be merged to FGRAPH_TYPE_ARRAY patch. > > > > hi, > > I'm getting kasan bugs below when running bpf selftests on top of this > > patchset.. I think it's probably the reason I see failures in some bpf > > kprobe_multi/fprobe tests > > > > so far I couldn't find the reason.. still checking ;-) > > Thanks for reporting! Have you built the kernel with debuginfo? In that > case, can you decode the line from the address? > > $ eu-addr2line -fi -e vmlinux ftrace_push_return_trace.isra.0+0x346 > > This helps me a lot. I also got the same KASAN error from intel's test bot: https://lore.kernel.org/all/202401172217.36e37075-oliver.s...@intel.com/ And another one (it should be different one) https://lore.kernel.org/all/202401172200.c8731564-oliver.s...@intel.com/ This is a selftest failure on i386. I might break something on 32bit. Let me check. Thank you, > > Thank you, > > > > > jirka > > > > > > --- > > [ 507.585913][ T697] BUG: KASAN: slab-out-of-bounds in > > ftrace_push_return_trace.isra.0+0x346/0x370 > > [ 507.586747][ T697] Write of size 8 at addr 888148193ff8 by task > > test_progs/697 > > [ 507.587460][ T697] > > [ 507.587713][ T697] CPU: 2 PID: 697 Comm: test_progs Tainted: G > > OE 6.7.0+ #309 d8e2cbcdc10865c6eb2d28ed0cbf958842aa75a8 > > [ 507.588821][ T697] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), > > BIOS 1.16.2-1.fc38 04/01/2014 > > [ 507.589681][ T697] Call Trace: > > [ 507.590044][ T697] > > [ 507.590357][ T697] dump_stack_lvl+0xf6/0x180 > > [ 507.590807][ T697] print_report+0xc4/0x610 > > [ 507.591259][ T697] ? fixup_red_left+0x5/0x20 > > [ 507.591781][ T697] kasan_report+0xbe/0xf0 > > [ 507.592241][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 > > [ 507.592928][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 > > [ 507.593535][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > > [ 507.594076][ T697] ? ftrace_replace_code+0x17a/0x230 > > [ 507.594586][ T697] ftrace_push_return_trace.isra.0+0x346/0x370 > > [ 507.595192][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > > [ 507.595747][ T697] function_graph_enter_ops+0xbb/0x2d0 > > [ 507.596271][ T697] ? ftrace_replace_code+0x17a/0x230 > > [ 507.596784][ T697] ? __pfx_function_graph_enter_ops+0x10/0x10 > > [ 507.597353][ T697] ? preempt_count_sub+0x14/0xc0 > > [ 507.598576][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > > [ 507.599145][ T697] ? __pfx_fuse_sync_fs+0x10/0x10 > > [ 507.599718][ T697] ftrace_graph_func+0x142/0x270 > > [ 507.600293][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > > [ 507.600892][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > > [ 507.601484][ T697] 0xa0560097 > > [ 507.602067][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > > [ 507.602715][ T697] ? text_poke_loc_init+0x5/0x2e0 > > [ 507.603288][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > > [ 507.603923][ T697] text_poke_loc_init+0x5/0x2e0 > > [ 507.604468][ T697] ftrace_replace_code+0x17a/0x230 > > [ 507.605071][ T697] ftrace_modify_all_code+0x131/0x1a0 > > [ 507.605663][ T697] ftrace_startup+0x10b/0x210 > > [ 507.606200][ T697] register_ftrace_graph+0x313/0x8a0 > > [ 507.606805][ T697] ? register_ftrace_graph+0x3fe/0x8a0 > > [ 507.607427][ T697] register_fprobe_ips.part.0+0x25a/0x3f0 > > [ 507.608090][ T697] bpf_kprobe_multi_link_attach+0x49e/0x850 > > [ 507.608781][ T697] ? __pfx_bpf_kprobe_multi_link_attach+0x10/0x10 > > [ 507.609500][ T697] ? __debug_check_no_obj_freed+0x1d8/0x3a0 > > [ 507.610194][ T697] ? __fget_light+0x96/0xe0 > > [ 507.610741][ T697] __sys_bpf+0x307a/0x3180 > > [ 507.611286][ T697] ? __pfx___sys_bpf+0x10/0x10 > > [ 507.6
Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Thu, 25 Jan 2024 15:54:53 +0100 Jiri Olsa wrote: > On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google) wrote: > > Hi, > > > > Here is the 6th version of the series to re-implement the fprobe on > > function-graph tracer. The previous version is; > > > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/ > > > > This version fixes use-after-unregister bug and arm64 stack unwinding > > bug [13/36], add an improvement for multiple interrupts during push > > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using > > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new > > fprobe can switch to SAVE_ARGS[32/36] safely. > > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which > > should be pushed separatedly as a stable bugfix. > > > > There are some TODOs: > > - Add s390x and loongarch support to fprobe (multiple fgraph). > > - Fix to get the symbol address from ftrace entry address on arm64. > >(This should be done in BPF trace event) > > - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP > >part should be merged to FGRAPH_TYPE_ARRAY patch. > > hi, > I'm getting kasan bugs below when running bpf selftests on top of this > patchset.. I think it's probably the reason I see failures in some bpf > kprobe_multi/fprobe tests > > so far I couldn't find the reason.. still checking ;-) Thanks for reporting! Have you built the kernel with debuginfo? In that case, can you decode the line from the address? $ eu-addr2line -fi -e vmlinux ftrace_push_return_trace.isra.0+0x346 This helps me a lot. Thank you, > > jirka > > > --- > [ 507.585913][ T697] BUG: KASAN: slab-out-of-bounds in > ftrace_push_return_trace.isra.0+0x346/0x370 > [ 507.586747][ T697] Write of size 8 at addr 888148193ff8 by task > test_progs/697 > [ 507.587460][ T697] > [ 507.587713][ T697] CPU: 2 PID: 697 Comm: test_progs Tainted: G > OE 6.7.0+ #309 d8e2cbcdc10865c6eb2d28ed0cbf958842aa75a8 > [ 507.588821][ T697] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), > BIOS 1.16.2-1.fc38 04/01/2014 > [ 507.589681][ T697] Call Trace: > [ 507.590044][ T697] > [ 507.590357][ T697] dump_stack_lvl+0xf6/0x180 > [ 507.590807][ T697] print_report+0xc4/0x610 > [ 507.591259][ T697] ? fixup_red_left+0x5/0x20 > [ 507.591781][ T697] kasan_report+0xbe/0xf0 > [ 507.592241][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 > [ 507.592928][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 > [ 507.593535][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > [ 507.594076][ T697] ? ftrace_replace_code+0x17a/0x230 > [ 507.594586][ T697] ftrace_push_return_trace.isra.0+0x346/0x370 > [ 507.595192][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > [ 507.595747][ T697] function_graph_enter_ops+0xbb/0x2d0 > [ 507.596271][ T697] ? ftrace_replace_code+0x17a/0x230 > [ 507.596784][ T697] ? __pfx_function_graph_enter_ops+0x10/0x10 > [ 507.597353][ T697] ? preempt_count_sub+0x14/0xc0 > [ 507.598576][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > [ 507.599145][ T697] ? __pfx_fuse_sync_fs+0x10/0x10 > [ 507.599718][ T697] ftrace_graph_func+0x142/0x270 > [ 507.600293][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 > [ 507.600892][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > [ 507.601484][ T697] 0xa0560097 > [ 507.602067][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > [ 507.602715][ T697] ? text_poke_loc_init+0x5/0x2e0 > [ 507.603288][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 > [ 507.603923][ T697] text_poke_loc_init+0x5/0x2e0 > [ 507.604468][ T697] ftrace_replace_code+0x17a/0x230 > [ 507.605071][ T697] ftrace_modify_all_code+0x131/0x1a0 > [ 507.605663][ T697] ftrace_startup+0x10b/0x210 > [ 507.606200][ T697] register_ftrace_graph+0x313/0x8a0 > [ 507.606805][ T697] ? register_ftrace_graph+0x3fe/0x8a0 > [ 507.607427][ T697] register_fprobe_ips.part.0+0x25a/0x3f0 > [ 507.608090][ T697] bpf_kprobe_multi_link_attach+0x49e/0x850 > [ 507.608781][ T697] ? __pfx_bpf_kprobe_multi_link_attach+0x10/0x10 > [ 507.609500][ T697] ? __debug_check_no_obj_freed+0x1d8/0x3a0 > [ 507.610194][ T697] ? __fget_light+0x96/0xe0 > [ 507.610741][ T697] __sys_bpf+0x307a/0x3180 > [ 507.611286][ T697] ? __pfx___sys_bpf+0x10/0x10 > [ 507.611838][ T697] ? __kasan_slab_free+0x12d/0x1c0 > [ 507.612434][ T697] ? audit_log_exit+0x8e0/0x1960 > [ 507.613003][ T697] ? kmem_cache_free+0x19d/0x460 > [ 507.613644][ T697] ? rcu_is_watching+0x34/0x60 > [ 507.614202][ T697] ? lockdep_hardirqs_on_prepare+0xe/0x250 > [ 507.614865][ T697] ? > seqcount_lockdep_reader_access.constprop.0+0x105/0x120 > [ 507.615662][ T697] ? > seqcount_lockdep_reader_access.constprop.0+0xb2/0x120 > [ 507.616431][ T697] __x64_sys_bpf+0x44/0x60 > [ 507.616940][ T697] do_syscall_64+0x87/0x1b0 > [ 507.617495][ T697] entry_SYSCALL_64_after_hwframe+
Re: [PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
On Fri, Jan 12, 2024 at 07:10:50PM +0900, Masami Hiramatsu (Google) wrote: > Hi, > > Here is the 6th version of the series to re-implement the fprobe on > function-graph tracer. The previous version is; > > https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/ > > This version fixes use-after-unregister bug and arm64 stack unwinding > bug [13/36], add an improvement for multiple interrupts during push > operation[20/36], keep SAVE_REGS until BPF and fprobe_event using > ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new > fprobe can switch to SAVE_ARGS[32/36] safely. > This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which > should be pushed separatedly as a stable bugfix. > > There are some TODOs: > - Add s390x and loongarch support to fprobe (multiple fgraph). > - Fix to get the symbol address from ftrace entry address on arm64. >(This should be done in BPF trace event) > - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP >part should be merged to FGRAPH_TYPE_ARRAY patch. hi, I'm getting kasan bugs below when running bpf selftests on top of this patchset.. I think it's probably the reason I see failures in some bpf kprobe_multi/fprobe tests so far I couldn't find the reason.. still checking ;-) jirka --- [ 507.585913][ T697] BUG: KASAN: slab-out-of-bounds in ftrace_push_return_trace.isra.0+0x346/0x370 [ 507.586747][ T697] Write of size 8 at addr 888148193ff8 by task test_progs/697 [ 507.587460][ T697] [ 507.587713][ T697] CPU: 2 PID: 697 Comm: test_progs Tainted: G OE 6.7.0+ #309 d8e2cbcdc10865c6eb2d28ed0cbf958842aa75a8 [ 507.588821][ T697] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc38 04/01/2014 [ 507.589681][ T697] Call Trace: [ 507.590044][ T697] [ 507.590357][ T697] dump_stack_lvl+0xf6/0x180 [ 507.590807][ T697] print_report+0xc4/0x610 [ 507.591259][ T697] ? fixup_red_left+0x5/0x20 [ 507.591781][ T697] kasan_report+0xbe/0xf0 [ 507.592241][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 [ 507.592928][ T697] ? ftrace_push_return_trace.isra.0+0x346/0x370 [ 507.593535][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 [ 507.594076][ T697] ? ftrace_replace_code+0x17a/0x230 [ 507.594586][ T697] ftrace_push_return_trace.isra.0+0x346/0x370 [ 507.595192][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 [ 507.595747][ T697] function_graph_enter_ops+0xbb/0x2d0 [ 507.596271][ T697] ? ftrace_replace_code+0x17a/0x230 [ 507.596784][ T697] ? __pfx_function_graph_enter_ops+0x10/0x10 [ 507.597353][ T697] ? preempt_count_sub+0x14/0xc0 [ 507.598576][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 [ 507.599145][ T697] ? __pfx_fuse_sync_fs+0x10/0x10 [ 507.599718][ T697] ftrace_graph_func+0x142/0x270 [ 507.600293][ T697] ? __pfx_text_poke_loc_init+0x10/0x10 [ 507.600892][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 [ 507.601484][ T697] 0xa0560097 [ 507.602067][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 [ 507.602715][ T697] ? text_poke_loc_init+0x5/0x2e0 [ 507.603288][ T697] ? __pfx_fuse_conn_put.part.0+0x10/0x10 [ 507.603923][ T697] text_poke_loc_init+0x5/0x2e0 [ 507.604468][ T697] ftrace_replace_code+0x17a/0x230 [ 507.605071][ T697] ftrace_modify_all_code+0x131/0x1a0 [ 507.605663][ T697] ftrace_startup+0x10b/0x210 [ 507.606200][ T697] register_ftrace_graph+0x313/0x8a0 [ 507.606805][ T697] ? register_ftrace_graph+0x3fe/0x8a0 [ 507.607427][ T697] register_fprobe_ips.part.0+0x25a/0x3f0 [ 507.608090][ T697] bpf_kprobe_multi_link_attach+0x49e/0x850 [ 507.608781][ T697] ? __pfx_bpf_kprobe_multi_link_attach+0x10/0x10 [ 507.609500][ T697] ? __debug_check_no_obj_freed+0x1d8/0x3a0 [ 507.610194][ T697] ? __fget_light+0x96/0xe0 [ 507.610741][ T697] __sys_bpf+0x307a/0x3180 [ 507.611286][ T697] ? __pfx___sys_bpf+0x10/0x10 [ 507.611838][ T697] ? __kasan_slab_free+0x12d/0x1c0 [ 507.612434][ T697] ? audit_log_exit+0x8e0/0x1960 [ 507.613003][ T697] ? kmem_cache_free+0x19d/0x460 [ 507.613644][ T697] ? rcu_is_watching+0x34/0x60 [ 507.614202][ T697] ? lockdep_hardirqs_on_prepare+0xe/0x250 [ 507.614865][ T697] ? seqcount_lockdep_reader_access.constprop.0+0x105/0x120 [ 507.615662][ T697] ? seqcount_lockdep_reader_access.constprop.0+0xb2/0x120 [ 507.616431][ T697] __x64_sys_bpf+0x44/0x60 [ 507.616940][ T697] do_syscall_64+0x87/0x1b0 [ 507.617495][ T697] entry_SYSCALL_64_after_hwframe+0x6e/0x76 [ 507.618179][ T697] RIP: 0033:0x7ff2edca6b4d [ 507.618745][ T697] Code: c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8b 92 0c 00 f7 d8 64 89 01 48 [ 507.620863][ T697] RSP: 002b:7ffe2e58a8f8 EFLAGS: 0206 ORIG_RAX: 0141 [ 507.621749][ T697] RAX: ffda RBX: 7ffe2e58b018 RCX: 7ff2edca6b4d
[PATCH v6 00/36] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
Hi, Here is the 6th version of the series to re-implement the fprobe on function-graph tracer. The previous version is; https://lore.kernel.org/all/170290509018.220107.1347127510564358608.stgit@devnote2/ This version fixes use-after-unregister bug and arm64 stack unwinding bug [13/36], add an improvement for multiple interrupts during push operation[20/36], keep SAVE_REGS until BPF and fprobe_event using ftrace_regs[26/36], also reorder the patches[30/36][31/36] so that new fprobe can switch to SAVE_ARGS[32/36] safely. This series also temporarily adds a DIRECT_CALLS bugfix[1/36], which should be pushed separatedly as a stable bugfix. There are some TODOs: - Add s390x and loongarch support to fprobe (multiple fgraph). - Fix to get the symbol address from ftrace entry address on arm64. (This should be done in BPF trace event) - Cleanup code, rename some terms(offset/index) and FGRAPH_TYPE_BITMAP part should be merged to FGRAPH_TYPE_ARRAY patch. Overview This series does major 2 changes, enable multiple function-graphs on the ftrace (e.g. allow function-graph on sub instances) and rewrite the fprobe on this function-graph. The former changes had been sent from Steven Rostedt 4 years ago (*), which allows users to set different setting function-graph tracer (and other tracers based on function-graph) in each trace-instances at the same time. (*) https://lore.kernel.org/all/20190525031633.811342...@goodmis.org/ The purpose of latter change are; 1) Remove dependency of the rethook from fprobe so that we can reduce the return hook code and shadow stack. 2) Make 'ftrace_regs' the common trace interface for the function boundary. 1) Currently we have 2(or 3) different function return hook codes, the function-graph tracer and rethook (and legacy kretprobe). But since this is redundant and needs double maintenance cost, I would like to unify those. From the user's viewpoint, function- graph tracer is very useful to grasp the execution path. For this purpose, it is hard to use the rethook in the function-graph tracer, but the opposite is possible. (Strictly speaking, kretprobe can not use it because it requires 'pt_regs' for historical reasons.) 2) Now the fprobe provides the 'pt_regs' for its handler, but that is wrong for the function entry and exit. Moreover, depending on the architecture, there is no way to accurately reproduce 'pt_regs' outside of interrupt or exception handlers. This means fprobe should not use 'pt_regs' because it does not use such exceptions. (Conversely, kprobe should use 'pt_regs' because it is an abstract interface of the software breakpoint exception.) This series changes fprobe to use function-graph tracer for tracing function entry and exit, instead of mixture of ftrace and rethook. Unlike the rethook which is a per-task list of system-wide allocated nodes, the function graph's ret_stack is a per-task shadow stack. Thus it does not need to set 'nr_maxactive' (which is the number of pre-allocated nodes). Also the handlers will get the 'ftrace_regs' instead of 'pt_regs'. Since eBPF mulit_kprobe/multi_kretprobe events still use 'pt_regs' as their register interface, this changes it to convert 'ftrace_regs' to 'pt_regs'. Of course this conversion makes an incomplete 'pt_regs', so users must access only registers for function parameters or return value. Design -- Instead of using ftrace's function entry hook directly, the new fprobe is built on top of the function-graph's entry and return callbacks with 'ftrace_regs'. Since the fprobe requires access to 'ftrace_regs', the architecture must support CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS, which enables to call function-graph entry callback with 'ftrace_regs', and also CONFIG_HAVE_FUNCTION_GRAPH_FREGS, which passes the ftrace_regs to return_to_handler. All fprobes share a single function-graph ops (means shares a common ftrace filter) similar to the kprobe-on-ftrace. This needs another layer to find corresponding fprobe in the common function-graph callbacks, but has much better scalability, since the number of registered function-graph ops is limited. In the entry callback, the fprobe runs its entry_handler and saves the address of 'fprobe' on the function-graph's shadow stack as data. The return callback decodes the data to get the 'fprobe' address, and runs the exit_handler. The fprobe introduces two hash-tables, one is for entry callback which searches fprobes related to the given function address passed by entry callback. The other is for a return callback which checks if the given 'fprobe' data structure pointer is still valid. Note that it is possible to unregister fprobe before the return callback runs. Thus the address validation must be done before using it in the return callback. This series can be applied against the v6.7 kernel. This series can also be found below branch. https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/fprobe-on-fgraph
