Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, Jan 14, 2010 at 01:29:09PM +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 13:23 +0100, Frederic Weisbecker wrote: I see, so what you suggest is to have the probe set up as generic first. Then the process that activates it becomes a consumer, right? Right, so either we have it always on, for things like ftrace, in which case the creation traverses rmap and installs the probes all existing mmap()s, and a mmap() hook will install it on all new ones. Or they're strictly consumer driver, like perf, in which case the act of enabling the event will install the probe (if its not there yet). Looks like a good plan.
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, 2010-01-14 at 12:23 +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Just to clarify, this means you can do things like: p:uprobe_event dso:symbol[+offs] Irrespective of whether there are any current user of that file.
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, Jan 14, 2010 at 12:23:11PM +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Then on creation it uses rmap to find all current maps of that file and install the probe if there is a consumer for that map. Then for each new mmap() of that file, we also need to check if there's a consumer ready and install the probe. That looks racy. Say you first create a probe on /bin/ls: perf probe p addr_in_ls /bin/ls then something else launches /bin/ls behind you, probe is set on it then you launch: perf record -e probe: /bin/ls Then it goes recording the previous instance.
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, 2010-01-14 at 12:35 +0100, Frederic Weisbecker wrote: On Thu, Jan 14, 2010 at 12:23:11PM +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Then on creation it uses rmap to find all current maps of that file and install the probe if there is a consumer for that map. Then for each new mmap() of that file, we also need to check if there's a consumer ready and install the probe. That looks racy. Say you first create a probe on /bin/ls: perf probe p addr_in_ls /bin/ls then something else launches /bin/ls behind you, probe is set on it then you launch: perf record -e probe: /bin/ls Then it goes recording the previous instance. Uhm, why? Only the perf /bin/ls instance has a consumer and will thus have a probe installed. (Or if you want to use ftrace you need to always have all instances probed anyway)
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, 2010-01-14 at 12:29 +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 12:23 +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Just to clarify, this means you can do things like: p:uprobe_event dso:symbol[+offs] Irrespective of whether there are any current user of that file. Yes, that is a good idea, you can then also refine that with a filter on a target pid. That is what systemtap also does, you define files (whether they are executables or shared libraries, etc) plus symbols/offsets/etc as targets and monitor when they get mapped in (either system wide, per executable or pid based). Cheers, Mark
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, 2010-01-14 at 13:16 +0100, Mark Wielaard wrote: On Thu, 2010-01-14 at 12:29 +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 12:23 +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Just to clarify, this means you can do things like: p:uprobe_event dso:symbol[+offs] Irrespective of whether there are any current user of that file. Yes, that is a good idea, you can then also refine that with a filter on a target pid. That is what systemtap also does, you define files (whether they are executables or shared libraries, etc) plus symbols/offsets/etc as targets and monitor when they get mapped in (either system wide, per executable or pid based). Well, the pid part is more the concern of the consumer of the trace-event. The event itself is task invariant and only cares about the particular code in question getting executed.
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Thu, Jan 14, 2010 at 12:43:01PM +0100, Peter Zijlstra wrote: On Thu, 2010-01-14 at 12:35 +0100, Frederic Weisbecker wrote: On Thu, Jan 14, 2010 at 12:23:11PM +0100, Peter Zijlstra wrote: On Mon, 2010-01-11 at 17:56 +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Right, like others have said, trace events is a much saner interface. So the easiest way I can see that working is to register uprobes against a file (not a pid). Then on creation it uses rmap to find all current maps of that file and install the probe if there is a consumer for that map. Then for each new mmap() of that file, we also need to check if there's a consumer ready and install the probe. That looks racy. Say you first create a probe on /bin/ls: perf probe p addr_in_ls /bin/ls then something else launches /bin/ls behind you, probe is set on it then you launch: perf record -e probe: /bin/ls Then it goes recording the previous instance. Uhm, why? Only the perf /bin/ls instance has a consumer and will thus have a probe installed. (Or if you want to use ftrace you need to always have all instances probed anyway) I see, so what you suggest is to have the probe set up as generic first. Then the process that activates it becomes a consumer, right? Would work, yeah.
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
Steven Rostedt wrote: On Tue, 2010-01-12 at 05:54 +0100, Frederic Weisbecker wrote: Now what if I want to launch ls and want to profile a function inside. What can I do with a trace event. I can't create the probe event based on a pid as I don't know it in advance. I could give it the ls cmdline and it manages to activate on the next ls launched. This is racy as another ls can be launched concurrently. You make a wrapper script: #!/bin/sh add probe to ls with pid $$ exec $* I do this all the time to limit the function tracer to a specific app. #!/bin/sh echo $$ /debug/tracing/set_ftrace_pid echo function /debug/tracing/current_tracer exec $* I recommend you to add below line at the end of the script, from my experience. :) echo nop /debug/tracing/current_tracer Thank you, -- Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: mhira...@redhat.com
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
Masami Hiramatsu wrote: Steven Rostedt wrote: On Tue, 2010-01-12 at 05:54 +0100, Frederic Weisbecker wrote: Now what if I want to launch ls and want to profile a function inside. What can I do with a trace event. I can't create the probe event based on a pid as I don't know it in advance. I could give it the ls cmdline and it manages to activate on the next ls launched. This is racy as another ls can be launched concurrently. You make a wrapper script: #!/bin/sh add probe to ls with pid $$ exec $* I do this all the time to limit the function tracer to a specific app. #!/bin/sh echo $$ /debug/tracing/set_ftrace_pid echo function /debug/tracing/current_tracer exec $* I recommend you to add below line at the end of the script, from my experience. :) echo nop /debug/tracing/current_tracer Oops, my bad, it doesn't work after exec... But, it is very important to disable function tracer after tracing target process. So, perhaps, below script may work. #!/bin/sh (echo $BASHPID /debug/tracing/set_ftrace_pid echo function /debug/tracing/current_tracer exec $*) echo nop /debug/tracing/current_tracer Thanks, -- Masami Hiramatsu Software Engineer Hitachi Computer Products (America), Inc. Software Solutions Division e-mail: mhira...@redhat.com
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
Frederic Weisbecker fweis...@gmail.com writes: [...] This is much more tricky in the case of uprobes as I see two ways to work with it: - probing on an already running process - probing on a process we are about to run [...] As you might expect, in systemtap we've had to figure out this area some time ago. We use another utrace consumer called task finder that registers interest in present / future processes, and gives us kernel-space callbacks when these come and go. You could merge it or something like it. - FChE
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
Hi - As you might expect, in systemtap we've had to figure out this area some time ago. We use another utrace consumer called task finder [...] So, could you tell us how the task-finder works and is implemented? The code may be found at runtime/task_finder* in the systemtap sources. There is a simple interest-registration structure/API that identifies processes / shared libraries of interest, and a set of callbacks to be invoked when said processes/shared libraries are mapped or unmapped. It is implemented in terms of utrace callbacks for process/thread lifetime monitoring, and utrace syscall callbacks for tracking shared library segments being mapped and unmapped. http://sourceware.org/git/?p=systemtap.git;a=tree;f=runtime I think we'd better clarify what functions are required for uprobes and pmu, and I think we may be able to re-implement improved pmu on utrace. I don't see any collision between pmu / perf / utrace, so nothing is really required for them or simple usage of uprobes. If you wish to track dynamic process/shared-library lifetimes, then you need extra code somewhere to respond to those changes. Layering this dynamic capability seems like the natural way to go, and is easily done with utrace and/or tracepoints. - FChE
Re: [RFC] [PATCH 7/7] Ftrace plugin for Uprobes
On Mon, Jan 11, 2010 at 05:56:08PM +0530, Srikar Dronamraju wrote: This patch implements ftrace plugin for uprobes. Description: Ftrace plugin provides an interface to dump data at a given address, top of the stack and function arguments when a user program calls a specific function. So, as told before, ftrace plugins tend to be relegated to obsolescence and I first suggested to plug this into kprobe events so that we have a unified interface to control/create u|k|kret probe events. But after digging more into first appearances, uprobe creation can follow the kprobes creation flow. kprobe can be created whenever we want. This is about probing kernel text and it is already there so that we can set the probe, default deactivated, in advance. This is much more tricky in the case of uprobes as I see two ways to work with it: - probing on an already running process - probing on a process we are about to run Now say we create to create a uprobe trace event for an already running process. No problem in the workflow, we just need to set the address and the pid. Fine. Now what if I want to launch ls and want to profile a function inside. What can I do with a trace event. I can't create the probe event based on a pid as I don't know it in advance. I could give it the ls cmdline and it manages to activate on the next ls launched. This is racy as another ls can be launched concurrently. So I can only say there that an ftrace plugin or an ftrace trace event would be only a half-useful interface to exploit utrace possibilities because it only lets us trace already running apps. Moreover I bet the most chosen workflow to profile/trace uprobes is by launching an app and profile it from the beginning, not by profiling an already running one, which makes an ftrace interface even less than half useful there. ftrace is cool to trace the kernel, but this kind of tricky userspace tracing workflow is not adapted to it. What do you think?
