Re: Trying to use 'perf probe' to debug perf itself
Em Thu, Feb 26, 2015 at 01:57:42PM +0900, Masami Hiramatsu escreveu: > (2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: > > Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: > >> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > >> Hmm, strange. Could you tell me the version of your kernel? > > [root@ssdandy ~]# uname -r > > 3.10.0-210.el7.x86_64 > >> It seems that the kernel newer than 3.14 supports uprobes with > >> memory dereference (e.g. -32(%sp) )feature. > > Right, that must be the case, will test, but then, would it be possible > > for the kernel, in such cases, return something line EOPNOTSUP? > Yeah, but for now, it is already supported in kernel. > Of cause we can try to test feature with adding temporary event > from perftools too. Yeah, capability querying is done in several places in tools/perf/ already, like when checking if we have perf_event_attr enabled features, such as sample_id_all, etc. Sometimes we can fallback to alternative mechanisms, sometimes we just need to provide a less cryptic message. > > I will try to figure out a better error message on the tooling side, > > something like: > > . Realize we're asking for memory dereference in uprobes > > . If it fails with EINVAL, check the kernel version and say something > > like: > > > > Please upgrade your kernel to at least x.y.z to have access to feature > > FOO_BAR. > OK, it may be worth for users (I'm not sure RHEL can update their kernel > to include that enhancement) That may be possible, will check with Jiri if we can include that in a future backport. Talking about these tools in conferences will make more people try the things mentioned, and as we talk about the latest and greatest, we need to try harder to handle error paths in a better way, providing less cryptic messages. Rebuilding a newer kernel now to retry all this, thanks! - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
Em Thu, Feb 26, 2015 at 01:57:42PM +0900, Masami Hiramatsu escreveu: (2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: Hmm, strange. Could you tell me the version of your kernel? [root@ssdandy ~]# uname -r 3.10.0-210.el7.x86_64 It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. Right, that must be the case, will test, but then, would it be possible for the kernel, in such cases, return something line EOPNOTSUP? Yeah, but for now, it is already supported in kernel. Of cause we can try to test feature with adding temporary event from perftools too. Yeah, capability querying is done in several places in tools/perf/ already, like when checking if we have perf_event_attr enabled features, such as sample_id_all, etc. Sometimes we can fallback to alternative mechanisms, sometimes we just need to provide a less cryptic message. I will try to figure out a better error message on the tooling side, something like: . Realize we're asking for memory dereference in uprobes . If it fails with EINVAL, check the kernel version and say something like: Please upgrade your kernel to at least x.y.z to have access to feature FOO_BAR. OK, it may be worth for users (I'm not sure RHEL can update their kernel to include that enhancement) That may be possible, will check with Jiri if we can include that in a future backport. Talking about these tools in conferences will make more people try the things mentioned, and as we talk about the latest and greatest, we need to try harder to handle error paths in a better way, providing less cryptic messages. Rebuilding a newer kernel now to retry all this, thanks! - Arnaldo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
(2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: > Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: >> (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: >>> Available variables at thread__get >>> @ >>> struct thread* thread >>> [root@ssdandy ~]# > >>> cool, so I thought it would be just a matter of asking to put the probes >>> and get the value of 'thread', then match things, but: > >>> [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > >>> Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 >>> thread=-32(%sp):u64 >>> Failed to write event: Invalid argument >>> Error: Failed to add events. Reason: Invalid argument (Code: -22) >>> [root@ssdandy ~]# > >>> Not possible :-\ > >> Hmm, strange. Could you tell me the version of your kernel? > > [root@ssdandy ~]# uname -r > 3.10.0-210.el7.x86_64 > >> It seems that the kernel newer than 3.14 supports uprobes with >> memory dereference (e.g. -32(%sp) )feature. > > Right, that must be the case, will test, but then, would it be possible > for the kernel, in such cases, return something line EOPNOTSUP? Yeah, but for now, it is already supported in kernel. Of cause we can try to test feature with adding temporary event from perftools too. > I will try to figure out a better error message on the tooling side, > something like: > > . Realize we're asking for memory dereference in uprobes > . If it fails with EINVAL, check the kernel version and say something > like: > > Please upgrade your kernel to at least x.y.z to have access to feature > FOO_BAR. OK, it may be worth for users (I'm not sure RHEL can update their kernel to include that enhancement) Thank you, > > - Arnaldo > >> (it was actually introduced by 5baaa59e, and git-describe told >> it was v3.13-rc4-22-g5baaa59) >> >> And also, could you try to write the result command to uprobe_event as >> below? >> >> # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> >> \ >> /sys/kernel/debug/tracing/uprobe_events > > Well, I'll try that if it fails after I upgrade to 3.14. > >>> please let me know if you need some file, here is the readelf -wi for >>> those two routines: >> >> This should not be the problem of dwarf-analysis. It seems kernel-side >> (uprobe) problem. > > Thanks a lot! > > - Arnaldo > -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: > (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > > Available variables at thread__get > > @ > > struct thread* thread > > [root@ssdandy ~]# > > cool, so I thought it would be just a matter of asking to put the probes > > and get the value of 'thread', then match things, but: > > [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > > Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 > > thread=-32(%sp):u64 > > Failed to write event: Invalid argument > > Error: Failed to add events. Reason: Invalid argument (Code: -22) > > [root@ssdandy ~]# > > Not possible :-\ > Hmm, strange. Could you tell me the version of your kernel? [root@ssdandy ~]# uname -r 3.10.0-210.el7.x86_64 > It seems that the kernel newer than 3.14 supports uprobes with > memory dereference (e.g. -32(%sp) )feature. Right, that must be the case, will test, but then, would it be possible for the kernel, in such cases, return something line EOPNOTSUP? I will try to figure out a better error message on the tooling side, something like: . Realize we're asking for memory dereference in uprobes . If it fails with EINVAL, check the kernel version and say something like: Please upgrade your kernel to at least x.y.z to have access to feature FOO_BAR. - Arnaldo > (it was actually introduced by 5baaa59e, and git-describe told > it was v3.13-rc4-22-g5baaa59) > > And also, could you try to write the result command to uprobe_event as > below? > > # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" > >> \ > /sys/kernel/debug/tracing/uprobe_events Well, I'll try that if it fails after I upgrade to 3.14. > > please let me know if you need some file, here is the readelf -wi for > > those two routines: > > This should not be the problem of dwarf-analysis. It seems kernel-side > (uprobe) problem. Thanks a lot! - Arnaldo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: Available variables at thread__get @thread__get+0 struct thread* thread [root@ssdandy ~]# cool, so I thought it would be just a matter of asking to put the probes and get the value of 'thread', then match things, but: [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread SNIP Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 Failed to write event: Invalid argument Error: Failed to add events. Reason: Invalid argument (Code: -22) [root@ssdandy ~]# Not possible :-\ Hmm, strange. Could you tell me the version of your kernel? [root@ssdandy ~]# uname -r 3.10.0-210.el7.x86_64 It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. Right, that must be the case, will test, but then, would it be possible for the kernel, in such cases, return something line EOPNOTSUP? I will try to figure out a better error message on the tooling side, something like: . Realize we're asking for memory dereference in uprobes . If it fails with EINVAL, check the kernel version and say something like: Please upgrade your kernel to at least x.y.z to have access to feature FOO_BAR. - Arnaldo (it was actually introduced by 5baaa59e, and git-describe told it was v3.13-rc4-22-g5baaa59) And also, could you try to write the result command to uprobe_event as below? # echo p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 \ /sys/kernel/debug/tracing/uprobe_events Well, I'll try that if it fails after I upgrade to 3.14. please let me know if you need some file, here is the readelf -wi for those two routines: This should not be the problem of dwarf-analysis. It seems kernel-side (uprobe) problem. Thanks a lot! - Arnaldo -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
(2015/02/25 22:25), Arnaldo Carvalho de Melo wrote: Em Wed, Feb 25, 2015 at 11:53:16AM +0900, Masami Hiramatsu escreveu: (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: Available variables at thread__get @thread__get+0 struct thread* thread [root@ssdandy ~]# cool, so I thought it would be just a matter of asking to put the probes and get the value of 'thread', then match things, but: [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread SNIP Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 Failed to write event: Invalid argument Error: Failed to add events. Reason: Invalid argument (Code: -22) [root@ssdandy ~]# Not possible :-\ Hmm, strange. Could you tell me the version of your kernel? [root@ssdandy ~]# uname -r 3.10.0-210.el7.x86_64 It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. Right, that must be the case, will test, but then, would it be possible for the kernel, in such cases, return something line EOPNOTSUP? Yeah, but for now, it is already supported in kernel. Of cause we can try to test feature with adding temporary event from perftools too. I will try to figure out a better error message on the tooling side, something like: . Realize we're asking for memory dereference in uprobes . If it fails with EINVAL, check the kernel version and say something like: Please upgrade your kernel to at least x.y.z to have access to feature FOO_BAR. OK, it may be worth for users (I'm not sure RHEL can update their kernel to include that enhancement) Thank you, - Arnaldo (it was actually introduced by 5baaa59e, and git-describe told it was v3.13-rc4-22-g5baaa59) And also, could you try to write the result command to uprobe_event as below? # echo p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 \ /sys/kernel/debug/tracing/uprobe_events Well, I'll try that if it fails after I upgrade to 3.14. please let me know if you need some file, here is the readelf -wi for those two routines: This should not be the problem of dwarf-analysis. It seems kernel-side (uprobe) problem. Thanks a lot! - Arnaldo -- Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: masami.hiramatsu...@hitachi.com -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Trying to use 'perf probe' to debug perf itself
Hi Arnaldo, (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: > Hi Masami, > > I'm adding thread refcounting in perf, and in this process I am > investigating a problem, so thought about using perf itself to match > thread__get with thread__put: > > [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put > Available variables at thread__put > @ > struct thread* thread > [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get > Available variables at thread__get > @ > struct thread* thread > [root@ssdandy ~]# > > cool, so I thought it would be just a matter of asking to put the probes > and get the value of 'thread', then match things, but: > > > [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread > probe-definition(0): thread__put thread > symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null) > parsing arg: thread into thread > 1 arguments > Open Debuginfo file: /root/bin/perf > Try to find probe point from debuginfo. > Probe point found: thread__put+0 > Searching 'thread' variable in context. > Converting variable thread into trace event. > thread type is (null). > Found 1 probe_trace_events. > Opening /sys/kernel/debug/tracing/uprobe_events write=1 > Added new event: > Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 > thread=-32(%sp):u64 > Failed to write event: Invalid argument > Error: Failed to add events. Reason: Invalid argument (Code: -22) > [root@ssdandy ~]# > > Not possible :-\ Hmm, strange. Could you tell me the version of your kernel? It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. (it was actually introduced by 5baaa59e, and git-describe told it was v3.13-rc4-22-g5baaa59) And also, could you try to write the result command to uprobe_event as below? # echo "p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64" >> \ /sys/kernel/debug/tracing/uprobe_events > > please let me know if you need some file, here is the readelf -wi for > those two routines: This should not be the problem of dwarf-analysis. It seems kernel-side (uprobe) problem. Thank you, > > # readelf -wi ~/bin/perf > > <1>: Abbrev Number: 35 (DW_TAG_subprogram) >DW_AT_external: 1 >DW_AT_name: (indirect string, offset: 0x4c5c3): > thread__get >DW_AT_decl_file : 6 >DW_AT_decl_line : 84 >DW_AT_prototyped : 1 >DW_AT_type: <0xe15cc> >DW_AT_low_pc : 0x4d038c >DW_AT_high_pc : 0x46 >DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) >DW_AT_GNU_all_tail_call_sites: 1 >DW_AT_sibling : <0xe2621> > <2>: Abbrev Number: 32 (DW_TAG_formal_parameter) >DW_AT_name: (indirect string, offset: 0x43dc7): thread > >DW_AT_decl_file : 6 >DW_AT_decl_line : 84 >DW_AT_type: <0xe15cc> >DW_AT_location: 2 byte block: 91 58 (DW_OP_fbreg: -40) > <2>: Abbrev Number: 0 > <1>: Abbrev Number: 39 (DW_TAG_subprogram) >DW_AT_external: 1 >DW_AT_name: (indirect string, offset: 0x4c699): > thread__put >DW_AT_decl_file : 6 >DW_AT_decl_line : 90 >DW_AT_prototyped : 1 >DW_AT_low_pc : 0x4d03d2 >DW_AT_high_pc : 0x6c >DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) >DW_AT_GNU_all_tail_call_sites: 1 >DW_AT_sibling : <0xe264d> > <2>: Abbrev Number: 32 (DW_TAG_formal_parameter) >DW_AT_name: (indirect string, offset: 0x43dc7): thread > >DW_AT_decl_file : 6 >DW_AT_decl_line : 90 >DW_AT_type: <0xe15cc> > > > Will see what I can do with just: > > [root@ssdandy ~]# perf probe ~/bin/perf thread__get > Added new event: > probe_perf:thread__get (on thread__get in /root/bin/perf) > > You can now use it in all perf tools, such as: > > perf record -e probe_perf:thread__get -aR sleep 1 > > [root@ssdandy ~]# perf probe ~/bin/perf thread__put > Added new event: > probe_perf:thread__put (on thread__put in /root/bin/perf) > > You can now use it in all perf tools, such as: > > perf record -e probe_perf:thread__put -aR sleep 1 > > [root@ssdandy ~]# perf list probe*:* > > List of pre-defined events (to be used in -e): > > probe_perf:thread__put [Tracepoint event] > probe_perf:thread__get [Tracepoint event] > [root@ssdandy ~]# perf probe -l > probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf) > probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf) > [root@ssdandy ~]# > > - Arnaldo > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the
Re: Trying to use 'perf probe' to debug perf itself
Hi Arnaldo, (2015/02/25 3:49), Arnaldo Carvalho de Melo wrote: Hi Masami, I'm adding thread refcounting in perf, and in this process I am investigating a problem, so thought about using perf itself to match thread__get with thread__put: [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__put Available variables at thread__put @thread__put+0 struct thread* thread [root@ssdandy ~]# perf probe -x ~/bin/perf -V thread__get Available variables at thread__get @thread__get+0 struct thread* thread [root@ssdandy ~]# cool, so I thought it would be just a matter of asking to put the probes and get the value of 'thread', then match things, but: [root@ssdandy ~]# perf probe -v ~/bin/perf thread__put thread probe-definition(0): thread__put thread symbol:thread__put file:(null) line:0 offset:0 return:0 lazy:(null) parsing arg: thread into thread 1 arguments Open Debuginfo file: /root/bin/perf Try to find probe point from debuginfo. Probe point found: thread__put+0 Searching 'thread' variable in context. Converting variable thread into trace event. thread type is (null). Found 1 probe_trace_events. Opening /sys/kernel/debug/tracing/uprobe_events write=1 Added new event: Writing event: p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 Failed to write event: Invalid argument Error: Failed to add events. Reason: Invalid argument (Code: -22) [root@ssdandy ~]# Not possible :-\ Hmm, strange. Could you tell me the version of your kernel? It seems that the kernel newer than 3.14 supports uprobes with memory dereference (e.g. -32(%sp) )feature. (it was actually introduced by 5baaa59e, and git-describe told it was v3.13-rc4-22-g5baaa59) And also, could you try to write the result command to uprobe_event as below? # echo p:probe_perf/thread__put /root/bin/perf:0xd03d2 thread=-32(%sp):u64 \ /sys/kernel/debug/tracing/uprobe_events please let me know if you need some file, here is the readelf -wi for those two routines: This should not be the problem of dwarf-analysis. It seems kernel-side (uprobe) problem. Thank you, # readelf -wi ~/bin/perf SNIP 1e25f1: Abbrev Number: 35 (DW_TAG_subprogram) e25f2 DW_AT_external: 1 e25f2 DW_AT_name: (indirect string, offset: 0x4c5c3): thread__get e25f6 DW_AT_decl_file : 6 e25f7 DW_AT_decl_line : 84 e25f8 DW_AT_prototyped : 1 e25f8 DW_AT_type: 0xe15cc e25fc DW_AT_low_pc : 0x4d038c e2604 DW_AT_high_pc : 0x46 e260c DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) e260e DW_AT_GNU_all_tail_call_sites: 1 e260e DW_AT_sibling : 0xe2621 2e2612: Abbrev Number: 32 (DW_TAG_formal_parameter) e2613 DW_AT_name: (indirect string, offset: 0x43dc7): thread e2617 DW_AT_decl_file : 6 e2618 DW_AT_decl_line : 84 e2619 DW_AT_type: 0xe15cc e261d DW_AT_location: 2 byte block: 91 58 (DW_OP_fbreg: -40) 2e2620: Abbrev Number: 0 1e2621: Abbrev Number: 39 (DW_TAG_subprogram) e2622 DW_AT_external: 1 e2622 DW_AT_name: (indirect string, offset: 0x4c699): thread__put e2626 DW_AT_decl_file : 6 e2627 DW_AT_decl_line : 90 e2628 DW_AT_prototyped : 1 e2628 DW_AT_low_pc : 0x4d03d2 e2630 DW_AT_high_pc : 0x6c e2638 DW_AT_frame_base : 1 byte block: 9c (DW_OP_call_frame_cfa) e263a DW_AT_GNU_all_tail_call_sites: 1 e263a DW_AT_sibling : 0xe264d 2e263e: Abbrev Number: 32 (DW_TAG_formal_parameter) e263f DW_AT_name: (indirect string, offset: 0x43dc7): thread e2643 DW_AT_decl_file : 6 e2644 DW_AT_decl_line : 90 e2645 DW_AT_type: 0xe15cc SNIP Will see what I can do with just: [root@ssdandy ~]# perf probe ~/bin/perf thread__get Added new event: probe_perf:thread__get (on thread__get in /root/bin/perf) You can now use it in all perf tools, such as: perf record -e probe_perf:thread__get -aR sleep 1 [root@ssdandy ~]# perf probe ~/bin/perf thread__put Added new event: probe_perf:thread__put (on thread__put in /root/bin/perf) You can now use it in all perf tools, such as: perf record -e probe_perf:thread__put -aR sleep 1 [root@ssdandy ~]# perf list probe*:* List of pre-defined events (to be used in -e): probe_perf:thread__put [Tracepoint event] probe_perf:thread__get [Tracepoint event] [root@ssdandy ~]# perf probe -l probe_perf:thread__get (on thread__get@util/thread.c in /root/bin/perf) probe_perf:thread__put (on thread__put@util/thread.c in /root/bin/perf) [root@ssdandy ~]# - Arnaldo -- To unsubscribe from
