Re: [RFC PATCH 00/11] mcount tracing utility
Steven Rostedt wrote: > The following patch series brings to vanilla Linux a bit of the RT kernel > trace facility. This incorporates the "-pg" profiling option of gcc > that will call the "mcount" function for all functions called in > the kernel. > > This patch series implements the code for x86 (32 and 64 bit), but > other archs can easily be implemented as well. Steven, This is really exciting! As a former maintainer of the (out-of-tree) Kernel Function Trace system, I really welcome this. I'm just getting out from under a backlog of work due to the holiday break, but I'm very interested. I will take a detailed look at this this week. I have been working with -finstrument-functions for a few years now, so I know of a few gotchas with that (e.g. It's currently broken on ARM EABI with GCC 4.x) This bug is one of the issues that has prevented me from attempting to mainline the KFT work this last year. Please keep me CC'ed on developments in this area, and let me know if there are any specific things I can do to help. I'd be very interested in helping out with non-x86 arch support. Regards, -- Tim = Tim Bird Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America = -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Steven Rostedt wrote: The following patch series brings to vanilla Linux a bit of the RT kernel trace facility. This incorporates the -pg profiling option of gcc that will call the mcount function for all functions called in the kernel. This patch series implements the code for x86 (32 and 64 bit), but other archs can easily be implemented as well. Steven, This is really exciting! As a former maintainer of the (out-of-tree) Kernel Function Trace system, I really welcome this. I'm just getting out from under a backlog of work due to the holiday break, but I'm very interested. I will take a detailed look at this this week. I have been working with -finstrument-functions for a few years now, so I know of a few gotchas with that (e.g. It's currently broken on ARM EABI with GCC 4.x) This bug is one of the issues that has prevented me from attempting to mainline the KFT work this last year. Please keep me CC'ed on developments in this area, and let me know if there are any specific things I can do to help. I'd be very interested in helping out with non-x86 arch support. Regards, -- Tim = Tim Bird Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America = -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Steven Rostedt <[EMAIL PROTECTED]> writes: > The following patch series brings to vanilla Linux a bit of the RT kernel > trace facility. This incorporates the "-pg" profiling option of gcc > that will call the "mcount" function for all functions called in > the kernel. > [...] > [Future:] SystemTap: > -- > One thing that Arnaldo and I discussed last year was using systemtap to > add hooks into the kernel to start and stop tracing. Sure. The dual of this makes sense too: letting systemtap scripts hook up to the mcount callback itself, for purposes beyond just tracing the function calls. > kprobes is too heavy to do on all funtion calls, but it would be > perfect to add to non hot paths to start the tracer and stop the > tracer. (Note that kprobes are not the only event sources systemtap can use: markers, timers, procfs control files, and some others. Any combination of these can be used in a script to express start/stop decisions.) - FChE -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Steven Rostedt <[EMAIL PROTECTED]> writes: > The following patch series brings to vanilla Linux a bit of the RT kernel > trace facility. This incorporates the "-pg" profiling option of gcc > that will call the "mcount" function for all functions called in > the kernel. My personal feeling regarding this code was that it would be much simpler/cleaner to implement a driver for the "jump tracer"s implemented in various CPUs. Basically the CPU will write all jumps into a buffer by itself. That allows you to do many traces (although not latency traces) too. -Andi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
On Thu, 3 Jan 2008, Mathieu Desnoyers wrote: > Hi Steven, > > Great work! Thanks! > > (added Tim Bird, author of KFT/KFI to the CC list) I'm currently investigating using -finstrument-functions instead of -pg, but if the overhead is too much, I may try to incorporate both. > > One interesting aspect of LTTng is that is would be very lightweight. > You seem to use interrupt disabling with your simple tracer and do a > _lot_ of cacheline bouncing (trace_idx[NR_CPUS] is a very good exemple). Please note that this tracer is more of a "simple example". There's lots of improvements that can be made. It was meant more of to show what mcount can bring than to push the tracer itself. I want to stress that the tracer in this patch set is a *much* simplified version of the latency_tracer in the RT patch. I want to start out simple, complexity can come later ;-) > > LTTng would write the information to a per-cpu memory buffer in binary > format. I see that it would be especially useful in flight recorder > mode, where we overwrite the buffers without writing them to disk : when > a problematic condition is reached, (a kernel oops would be a good one), > then we just stop tracing and dump the last buffers to disk. In this > case, we would have the last function calls that led to an OOPS. This sounds great. My hope is that we can get the mcount (or cyg_profile) functionality in the kernel that many different users can deploy. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Hi Steven, Great work! (added Tim Bird, author of KFT/KFI to the CC list) * Steven Rostedt ([EMAIL PROTECTED]) wrote: > ... > > Future: > --- > The way the mcount hook is done here, other utilities can easily add their > own functions. Just care needs to be made not to call anything that is not > marked with notrace, or you will crash the box with recursion. But > even the simple tracer adds a "disabled" feature so in case it happens > to call something that is not marked with notrace, it is a safety net > not to kill the box. > > I was originally going to use the relay system to record the data, but > that had a chance of calling functions not marked with notrace. But, if > for example LTTng wanted to use this, it could disable tracing on a CPU > when doing the calls, and this will protect from recusion. > Yes, I'd love to add this information source to LTTng. It simply boils down to adding a "notrace" flag to LTTng tracing functions. Since I don't use relay code _at all_ in the tracing path, there is no problem with this (I only disable preemption and do "local" atomic operations on per-cpu variables). Then I would have to do the glue code that registers the LTTng handler to your mcount mechanism. One interesting aspect of LTTng is that is would be very lightweight. You seem to use interrupt disabling with your simple tracer and do a _lot_ of cacheline bouncing (trace_idx[NR_CPUS] is a very good exemple). LTTng would write the information to a per-cpu memory buffer in binary format. I see that it would be especially useful in flight recorder mode, where we overwrite the buffers without writing them to disk : when a problematic condition is reached, (a kernel oops would be a good one), then we just stop tracing and dump the last buffers to disk. In this case, we would have the last function calls that led to an OOPS. Mathieu -- Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Hi Steven, Great work! (added Tim Bird, author of KFT/KFI to the CC list) * Steven Rostedt ([EMAIL PROTECTED]) wrote: ... Future: --- The way the mcount hook is done here, other utilities can easily add their own functions. Just care needs to be made not to call anything that is not marked with notrace, or you will crash the box with recursion. But even the simple tracer adds a disabled feature so in case it happens to call something that is not marked with notrace, it is a safety net not to kill the box. I was originally going to use the relay system to record the data, but that had a chance of calling functions not marked with notrace. But, if for example LTTng wanted to use this, it could disable tracing on a CPU when doing the calls, and this will protect from recusion. Yes, I'd love to add this information source to LTTng. It simply boils down to adding a notrace flag to LTTng tracing functions. Since I don't use relay code _at all_ in the tracing path, there is no problem with this (I only disable preemption and do local atomic operations on per-cpu variables). Then I would have to do the glue code that registers the LTTng handler to your mcount mechanism. One interesting aspect of LTTng is that is would be very lightweight. You seem to use interrupt disabling with your simple tracer and do a _lot_ of cacheline bouncing (trace_idx[NR_CPUS] is a very good exemple). LTTng would write the information to a per-cpu memory buffer in binary format. I see that it would be especially useful in flight recorder mode, where we overwrite the buffers without writing them to disk : when a problematic condition is reached, (a kernel oops would be a good one), then we just stop tracing and dump the last buffers to disk. In this case, we would have the last function calls that led to an OOPS. Mathieu -- Mathieu Desnoyers Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
On Thu, 3 Jan 2008, Mathieu Desnoyers wrote: Hi Steven, Great work! Thanks! (added Tim Bird, author of KFT/KFI to the CC list) I'm currently investigating using -finstrument-functions instead of -pg, but if the overhead is too much, I may try to incorporate both. One interesting aspect of LTTng is that is would be very lightweight. You seem to use interrupt disabling with your simple tracer and do a _lot_ of cacheline bouncing (trace_idx[NR_CPUS] is a very good exemple). Please note that this tracer is more of a simple example. There's lots of improvements that can be made. It was meant more of to show what mcount can bring than to push the tracer itself. I want to stress that the tracer in this patch set is a *much* simplified version of the latency_tracer in the RT patch. I want to start out simple, complexity can come later ;-) LTTng would write the information to a per-cpu memory buffer in binary format. I see that it would be especially useful in flight recorder mode, where we overwrite the buffers without writing them to disk : when a problematic condition is reached, (a kernel oops would be a good one), then we just stop tracing and dump the last buffers to disk. In this case, we would have the last function calls that led to an OOPS. This sounds great. My hope is that we can get the mcount (or cyg_profile) functionality in the kernel that many different users can deploy. -- Steve -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [RFC PATCH 00/11] mcount tracing utility
Steven Rostedt [EMAIL PROTECTED] writes: The following patch series brings to vanilla Linux a bit of the RT kernel trace facility. This incorporates the -pg profiling option of gcc that will call the mcount function for all functions called in the kernel. My personal feeling regarding this code was that it would be much simpler/cleaner to implement a driver for the jump tracers implemented in various CPUs. Basically the CPU will write all jumps into a buffer by itself. That allows you to do many traces (although not latency traces) too. -Andi -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
