Re: Trying to use 'perf probe' to debug perf itself
Hi Arnaldo, (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > Hi Masami, > > I'm adding thread refcounting in perf, and in this process I am > investigating a problem, so thought about using perf itself to match > thread__get with thread__put: > > [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put > Available variables at thread__put > @ > struct thread* thread > [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get > Available variables at thread__get > @ > struct thread* thread > [root@ssdandy ~]# > > cool, so I thought it would be just a matter of asking to put the probes > and get the value of 'thread', then match things, but: > > > [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > probe-definition(0): thread__put thread > symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null) > parsing arg: thread into thread > 1 arguments > Open Debuginfo file: /root/bin/perf > Try to find probe point from debuginfo. > Probe point found: thread__put+0 > Searching 'thread' variable in context. > Converting variable thread into trace event. > thread type is (null). > Found 1 probe_trace_events. > Opening /sys/kernel/debug/tracing/uprobe_events write=1 > Added new event: > Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 > thread=-32(%sp):u64 > Failed to write event: Invalid argument > Error: Failed to add events. Reason: Invalid argument (Code: -22) > [root@ssdandy ~]# > > Not possible :-\ Hmm, strange. Could you tell me the version of your kernel? It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. (it was actually introduced by 5baaa59e, and git-describe told it was v3.13-rc4-22-g5baaa59) And also, could you try to write the result command to uprobe_event as below? # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \ /sys/kernel/debug/tracing/uprobe_events > > please let me know if you need some file, here is the readelf -wi for > those two routines: This should not be the problem of dwarf-analysis. It seems kernel-side (uprobe) problem. Thank you, > > # readelf -wi ~/bin/perf > > <1>: Abbrev Number: 35 (DW_TAG_subprogram) >DW_AT_external: 1 >DW_AT_name: (indirect string, offset: 0x4c5c3): > thread__get >DW_AT_decl_file : 6 >DW_AT_decl_line : 84 >DW_AT_prototyped : 1 >DW_AT_type: <0xe15cc> >DW_AT_low_pc : 0x4d038c >DW_AT_high_pc : 0x46 >DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) >DW_AT_GNU_all_tail_call_sites: 1 >DW_AT_sibling : <0xe2621> > <2>: Abbrev Number: 32 (DW_TAG_formal_parameter) >DW_AT_name: (indirect string, offset: 0x43dc7): thread > >DW_AT_decl_file : 6 >DW_AT_decl_line : 84 >DW_AT_type: <0xe15cc> >DW_AT_location: 2 byte block: 91 58 (DW_OP_fbreg: -40) > <2>: Abbrev Number: 0 > <1>: Abbrev Number: 39 (DW_TAG_subprogram) >DW_AT_external: 1 >DW_AT_name: (indirect string, offset: 0x4c699): > thread__put >DW_AT_decl_file : 6 >DW_AT_decl_line : 90 >DW_AT_prototyped : 1 >DW_AT_low_pc : 0x4d03d2 >DW_AT_high_pc : 0x6c >DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) >DW_AT_GNU_all_tail_call_sites: 1 >DW_AT_sibling : <0xe264d> > <2>: Abbrev Number: 32 (DW_TAG_formal_parameter) >DW_AT_name: (indirect string, offset: 0x43dc7): thread > >DW_AT_decl_file : 6 >DW_AT_decl_line : 90 >DW_AT_type: <0xe15cc> > > > Will see what I can do with just: > > [root@ssdandy ~]# perf probe ~/bin/perf thread__get > Added new event: > probe_perf:thread__get (on thread__get in /root/bin/perf) > > You can now use it in all perf tools, such as: > > perf record -e probe_perf:thread__get -aR sleep 1 > > [root@ssdandy ~]# perf probe ~/bin/perf thread__put > Added new event: > probe_perf:thread__put (on thread__put in /root/bin/perf) > > You can now use it in all perf tools, such as: > > perf record -e probe_perf:thread__put -aR sleep 1 > > [root@ssdandy ~]# perf list probe*:* > > List of pre-defined events (to be used in -e): > > probe_perf:thread__put [Tracepoint event] > probe_perf:thread__get [Tracepoint event] > [root@ssdandy ~]# perf probe -l > probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf) > probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf) > [root@ssdandy ~]# > > - Arnaldo > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body
Re: Trying to use 'perf probe' to debug perf itself
Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: > (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > > Available variables at thread__get > > @ > > struct thread* thread > > [root@ssdandy ~]# > > cool, so I thought it would be just a matter of asking to put the probes > > and get the value of 'thread', then match things, but: > > [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > > Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 > > thread=-32(%sp):u64 > > Failed to write event: Invalid argument > > Error: Failed to add events. Reason: Invalid argument (Code: -22) > > [root@ssdandy ~]# > > Not possible :-\ > Hmm, strange. Could you tell me the version of your kernel? [root@ssdandy ~]# uname -r 3.10.0-210.el7.x86_64 > It seems that the kernel newer than 3.14 supports uprobes with > memory dereference (e.g. -32(%sp) )feature. Right, that must be the case, will test, but then, would it be possible for the kernel, in such cases, return something line EOPNOTSUP? I will try to figure out a better error message on the tooling side, something like: . Realize we're asking for memory dereference in uprobes . If it fails with EINVAL, check the kernel version and say something like: Please upgrade your kernel to at least x.y.z to have access to feature FOO_BAR. - Arnaldo > (it was actually introduced by 5baaa59e, and git-describe told > it was v3.13-rc4-22-g5baaa59) > > And also, could you try to write the result command to uprobe_event as > below? > > # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" > >> \ > /sys/kernel/debug/tracing/uprobe_events Well, I'll try that if it fails after I upgrade to 3.14. > > please let me know if you need some file, here is the readelf -wi for > > those two routines: > > This should not be the problem of dwarf-analysis. It seems kernel-side > (uprobe) problem. Thanks a lot! - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
(2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: > Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: >> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: >>> Available variables at thread__get >>> @ >>> struct thread* thread >>> [root@ssdandy ~]# > >>> cool, so I thought it would be just a matter of asking to put the probes >>> and get the value of 'thread', then match things, but: > >>> [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > >>> Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 >>> thread=-32(%sp):u64 >>> Failed to write event: Invalid argument >>> Error: Failed to add events. Reason: Invalid argument (Code: -22) >>> [root@ssdandy ~]# > >>> Not possible :-\ > >> Hmm, strange. Could you tell me the version of your kernel? > > [root@ssdandy ~]# uname -r > 3.10.0-210.el7.x86_64 > >> It seems that the kernel newer than 3.14 supports uprobes with >> memory dereference (e.g. -32(%sp) )feature. > > Right, that must be the case, will test, but then, would it be possible > for the kernel, in such cases, return something line EOPNOTSUP? Yeah, but for now, it is already supported in kernel. Of cause we can try to test feature with adding temporary event from perftools too. > I will try to figure out a better error message on the tooling side, > something like: > > . Realize we're asking for memory dereference in uprobes > . If it fails with EINVAL, check the kernel version and say something > like: > > Please upgrade your kernel to at least x.y.z to have access to feature > FOO_BAR. OK, it may be worth for users (I'm not sure RHEL can update their kernel to include that enhancement) Thank you, > > - Arnaldo > >> (it was actually introduced by 5baaa59e, and git-describe told >> it was v3.13-rc4-22-g5baaa59) >> >> And also, could you try to write the result command to uprobe_event as >> below? >> >> # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> >> \ >> /sys/kernel/debug/tracing/uprobe_events > > Well, I'll try that if it fails after I upgrade to 3.14. > >>> please let me know if you need some file, here is the readelf -wi for >>> those two routines: >> >> This should not be the problem of dwarf-analysis. It seems kernel-side >> (uprobe) problem. > > Thanks a lot! > > - Arnaldo > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
Em Thu, Feb 26, 2015 at 01:57:42PM +0900, Masami Hiramatsu escreveu: > (2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: > > Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: > >> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > >> Hmm, strange. Could you tell me the version of your kernel? > > [root@ssdandy ~]# uname -r > > 3.10.0-210.el7.x86_64 > >> It seems that the kernel newer than 3.14 supports uprobes with > >> memory dereference (e.g. -32(%sp) )feature. > > Right, that must be the case, will test, but then, would it be possible > > for the kernel, in such cases, return something line EOPNOTSUP? > Yeah, but for now, it is already supported in kernel. > Of cause we can try to test feature with adding temporary event > from perftools too. Yeah, capability querying is done in several places in tools/perf/ already, like when checking if we have perf_event_attr enabled features, such as sample_id_all, etc. Sometimes we can fallback to alternative mechanisms, sometimes we just need to provide a less cryptic message. > > I will try to figure out a better error message on the tooling side, > > something like: > > . Realize we're asking for memory dereference in uprobes > > . If it fails with EINVAL, check the kernel version and say something > > like: > > > > Please upgrade your kernel to at least x.y.z to have access to feature > > FOO_BAR. > OK, it may be worth for users (I'm not sure RHEL can update their kernel > to include that enhancement) That may be possible, will check with Jiri if we can include that in a future backport. Talking about these tools in conferences will make more people try the things mentioned, and as we talk about the latest and greatest, we need to try harder to handle error paths in a better way, providing less cryptic messages. Rebuilding a newer kernel now to retry all this, thanks! - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
