Re: [PATCH v9 00/22] tracing: 'hist' triggers
Hi Brendan, On Wed, 2015-07-22 at 11:29 -0700, Brendan Gregg wrote: > G'Day Tom, > > On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi > wrote: > > > > This is v9 of the 'hist triggers' patchset. > > > [...] > > I've browsed the functionality (sorry, catching up), and it looks like > this will solve a number of common problems. But it seems > tantalizingly close to solving a few more. These may already be on > your future todo list. > > A) CPU stack profiling > > Kernel stacktrace as a key will be hugely useful; is there a way to > enable this for a sampling profile? (eg, what perf record -F 99 does). > I take CPU profiles daily, and would prefer to aggregate stacks > in-kernel. Also, would like user stacktrace as a key (even if it's > just the hex). > This wasn't on my todo list but I can see how it would be useful. On the list now. ;-) > B) Key buckets > > Eg, imagine: > > echo 'hist:keys=common_pid.execname,count.log2:val=count' > > /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger > > to get a log2 bucketized histogram of syscall read request size. Same > for any value where using the value as a key gets too verbose, and you > just want a rough look at the distribution. (Would make it more > readable if it could also be sorted by the log2 value.) > This I actually had an early implementation of, which I plan on reviving... > C) Latency as a bucket key > > With kprobes, we could then have a log2 histogram of any function call > latency, collected efficiently. (There's already the function timers > in ftrace, which I'm using from function_graph with filters sets to > only match the target function.) > My original thought for doing this kind of thing was to generalize the concept of a 'difference'. I used it in v1 as a way to calculate the difference between requested and allocated sizes for memory allocations, which was kind of pointless, though convenient. The real value would of course be in applying it to inter-event values rather than intra-event. In this case it would be an inter-event difference between timestamps. And in my previous patchset, I had a 'function_hist' tracer, similar to and in fact based on the same code as function_graph, that would simply aggregate hitcounts for every function call in the kernel, which indeed was pretty efficient. So the pieces are or have been there to do something like this, just a matter of putting them together. > ... Those are the other common use cases, that the hist functionality > seemed suited for. Beyond that gets more custom, and we can use eBPF. > Exactly. Tom > Brendan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v9 00/22] tracing: 'hist' triggers
G'Day Tom, On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi wrote: > > This is v9 of the 'hist triggers' patchset. > [...] I've browsed the functionality (sorry, catching up), and it looks like this will solve a number of common problems. But it seems tantalizingly close to solving a few more. These may already be on your future todo list. A) CPU stack profiling Kernel stacktrace as a key will be hugely useful; is there a way to enable this for a sampling profile? (eg, what perf record -F 99 does). I take CPU profiles daily, and would prefer to aggregate stacks in-kernel. Also, would like user stacktrace as a key (even if it's just the hex). B) Key buckets Eg, imagine: echo 'hist:keys=common_pid.execname,count.log2:val=count' > /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger to get a log2 bucketized histogram of syscall read request size. Same for any value where using the value as a key gets too verbose, and you just want a rough look at the distribution. (Would make it more readable if it could also be sorted by the log2 value.) C) Latency as a bucket key With kprobes, we could then have a log2 histogram of any function call latency, collected efficiently. (There's already the function timers in ftrace, which I'm using from function_graph with filters sets to only match the target function.) ... Those are the other common use cases, that the hist functionality seemed suited for. Beyond that gets more custom, and we can use eBPF. Brendan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v9 00/22] tracing: 'hist' triggers
G'Day Tom, On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi tom.zanu...@linux.intel.com wrote: This is v9 of the 'hist triggers' patchset. [...] I've browsed the functionality (sorry, catching up), and it looks like this will solve a number of common problems. But it seems tantalizingly close to solving a few more. These may already be on your future todo list. A) CPU stack profiling Kernel stacktrace as a key will be hugely useful; is there a way to enable this for a sampling profile? (eg, what perf record -F 99 does). I take CPU profiles daily, and would prefer to aggregate stacks in-kernel. Also, would like user stacktrace as a key (even if it's just the hex). B) Key buckets Eg, imagine: echo 'hist:keys=common_pid.execname,count.log2:val=count' /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger to get a log2 bucketized histogram of syscall read request size. Same for any value where using the value as a key gets too verbose, and you just want a rough look at the distribution. (Would make it more readable if it could also be sorted by the log2 value.) C) Latency as a bucket key With kprobes, we could then have a log2 histogram of any function call latency, collected efficiently. (There's already the function timers in ftrace, which I'm using from function_graph with filters sets to only match the target function.) ... Those are the other common use cases, that the hist functionality seemed suited for. Beyond that gets more custom, and we can use eBPF. Brendan -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v9 00/22] tracing: 'hist' triggers
Hi Brendan, On Wed, 2015-07-22 at 11:29 -0700, Brendan Gregg wrote: G'Day Tom, On Thu, Jul 16, 2015 at 10:22 AM, Tom Zanussi tom.zanu...@linux.intel.com wrote: This is v9 of the 'hist triggers' patchset. [...] I've browsed the functionality (sorry, catching up), and it looks like this will solve a number of common problems. But it seems tantalizingly close to solving a few more. These may already be on your future todo list. A) CPU stack profiling Kernel stacktrace as a key will be hugely useful; is there a way to enable this for a sampling profile? (eg, what perf record -F 99 does). I take CPU profiles daily, and would prefer to aggregate stacks in-kernel. Also, would like user stacktrace as a key (even if it's just the hex). This wasn't on my todo list but I can see how it would be useful. On the list now. ;-) B) Key buckets Eg, imagine: echo 'hist:keys=common_pid.execname,count.log2:val=count' /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger to get a log2 bucketized histogram of syscall read request size. Same for any value where using the value as a key gets too verbose, and you just want a rough look at the distribution. (Would make it more readable if it could also be sorted by the log2 value.) This I actually had an early implementation of, which I plan on reviving... C) Latency as a bucket key With kprobes, we could then have a log2 histogram of any function call latency, collected efficiently. (There's already the function timers in ftrace, which I'm using from function_graph with filters sets to only match the target function.) My original thought for doing this kind of thing was to generalize the concept of a 'difference'. I used it in v1 as a way to calculate the difference between requested and allocated sizes for memory allocations, which was kind of pointless, though convenient. The real value would of course be in applying it to inter-event values rather than intra-event. In this case it would be an inter-event difference between timestamps. And in my previous patchset, I had a 'function_hist' tracer, similar to and in fact based on the same code as function_graph, that would simply aggregate hitcounts for every function call in the kernel, which indeed was pretty efficient. So the pieces are or have been there to do something like this, just a matter of putting them together. ... Those are the other common use cases, that the hist functionality seemed suited for. Beyond that gets more custom, and we can use eBPF. Exactly. Tom Brendan -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH v9 00/22] tracing: 'hist' triggers
On Tue, 2015-07-21 at 18:40 +0900, Masami Hiramatsu wrote:
> Hi Tom,
>
> Thank you for updating your patches, I'm testing it.
>
> And when I'm testing hist trigger, I've found that the .hex modifiers on value
> doesn't work, but no semantic error.
>
Yeah, the hex modifier on values got dropped when refactoring the
patchset, will fix. Thanks for testing and pointing it out.
Tom
> [root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.hex' >
> events/sched/sched_process_fork/trigger
> [root@localhost tracing]#
>
> [root@localhost tracing]# cat events/sched/sched_process_fork/hist
> # trigger info:
> hist:keys=parent_pid:vals=hitcount,common_pid.hex:sort=hitcount:size=2048
> [active]
>
> { parent_pid: 26582 } hitcount: 1 common_pid: 26582
> { parent_pid: 11968 } hitcount: 1 common_pid: 11968
> { parent_pid: 11956 } hitcount: 2 common_pid: 23912
>
> Totals:
> Hits: 4
> Entries: 3
> Dropped: 0
>
> while other modifiers return -EINVAL.
>
> [root@localhost tracing]# echo
> 'hist:keys=parent_pid:vals=common_pid.execname' >
> events/sched/sched_process_fork/trigger
> -bash: echo: write error: Invalid argument
>
> Thank you,
>
> On 2015/07/17 2:22, Tom Zanussi wrote:
> > This is v9 of the 'hist triggers' patchset.
> >
> > Changes from v8:
> >
> > Same as v8, but with the RFC patch [ftrace: Add function_hist tracer]
> > removed, and rebased to latest trace/for-next.
> >
> > Changes from v7:
> >
> > This version refactors the commits as suggested by Masami. There are
> > now more commits, but the result should be much more reviewable. The
> > ending code is the same as before, modulo a couple minor bug fixes I
> > discovered while refactoring and testing.
> >
> > I've also reviewed and fixed a number of shortcomings and errors in
> > the comments, and have added a new discussion of the tracing_map data
> > structures after Steve mentioned he found them confusing and/or
> > insufficiently documented.
> >
> > Also, I kept Namhyung's string patch [tracing: Support string type key
> > properly] as submitted, but added a follow-on patch that refactors it
> > and fixes a problem I found with it that enabled static string keys to
> > contain random chars and therefore incorrect map insertions.
> >
> > Changes from v6:
> >
> > This version adds a new 'sym-offset' modifier as requested by Masami.
> > I implemented it as a modifier rather than using the trace option as
> > suggested, in part because I wanted to keep it all self-contained and
> > it seemed more consistent to just add it alongside the 'sym' modifier.
> > Also, hist triggers arent't really a tracer and therefore don't
> > directly tie into the option update/callback mechanism so making use
> > of it isn't as simple as a normal tracer.
> >
> > I also changed the sort key specification to be stricter and signal an
> > error if the specified sort key wasn't found (rather than defaulting
> > to hitcount in those cases), also suggested by Masami. Thanks,
> > Masami, for your input!
> >
> > Also updated the Documentation and tracing/README to reflect the
> > changes.
> >
> > Changes from v5:
> >
> > This version adds support for compound keys, along with the related
> > ability to sort using primary and secondary keys. This was mentioned
> > in previous versions as the last important piece that remained
> > unimplemented, and is now implemented. (I didn't have time to get to
> > the couple of enhancements suggested by Masami, but I expect to be
> > able to add those later on top of these.)
> >
> > Because we now support compound keys and it's not immediately clear in
> > the output exactly which fields correspond to keys, the key(s),
> > compound or not, are now enclosed by curly braces.
> >
> > The Documentation and README have been updated to reflect the changes,
> > and several new examples have been added to illustrate how to use
> > compound keys.
> >
> > Also, the code was updated to work with the new ftrace_event_file,
> > etc, renaming in tracing/for-next.
> >
> > Changes from v4:
> >
> > This version addresses some problems and suggestions made by Daniel
> > Wagner - a lot of the code was reworked to get rid of the distinction
> > between keys and values, and as a result, both keys and values can be
> > used as sort keys. As suggested, it also allows 'val=' to be absent
> > in a trigger command - if no 'val' is specified, hitcount is assumed
> > and automatically used as the only val.
> >
> > The map code was also separated out into a separate file,
> > tracing_map.c, allowing it to be reused. It also adds a second tracer
> > called function_hist that actually does reuse the code, as an RFC
> > patch.
> >
> > Patch 01/10 [tracing: Update cond flag when enabling or disabling..]
> > is a fix for a problem noticed by Daniel and that fixes a problem in
> > existing trigger code and should be applied regardless of whether the
> > rest
Re: [PATCH v9 00/22] tracing: 'hist' triggers
Hi Tom,
Thank you for updating your patches, I'm testing it.
And when I'm testing hist trigger, I've found that the .hex modifiers on value
doesn't work, but no semantic error.
[root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.hex' >
events/sched/sched_process_fork/trigger
[root@localhost tracing]#
[root@localhost tracing]# cat events/sched/sched_process_fork/hist
# trigger info:
hist:keys=parent_pid:vals=hitcount,common_pid.hex:sort=hitcount:size=2048
[active]
{ parent_pid: 26582 } hitcount: 1 common_pid: 26582
{ parent_pid: 11968 } hitcount: 1 common_pid: 11968
{ parent_pid: 11956 } hitcount: 2 common_pid: 23912
Totals:
Hits: 4
Entries: 3
Dropped: 0
while other modifiers return -EINVAL.
[root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.execname'
> events/sched/sched_process_fork/trigger
-bash: echo: write error: Invalid argument
Thank you,
On 2015/07/17 2:22, Tom Zanussi wrote:
> This is v9 of the 'hist triggers' patchset.
>
> Changes from v8:
>
> Same as v8, but with the RFC patch [ftrace: Add function_hist tracer]
> removed, and rebased to latest trace/for-next.
>
> Changes from v7:
>
> This version refactors the commits as suggested by Masami. There are
> now more commits, but the result should be much more reviewable. The
> ending code is the same as before, modulo a couple minor bug fixes I
> discovered while refactoring and testing.
>
> I've also reviewed and fixed a number of shortcomings and errors in
> the comments, and have added a new discussion of the tracing_map data
> structures after Steve mentioned he found them confusing and/or
> insufficiently documented.
>
> Also, I kept Namhyung's string patch [tracing: Support string type key
> properly] as submitted, but added a follow-on patch that refactors it
> and fixes a problem I found with it that enabled static string keys to
> contain random chars and therefore incorrect map insertions.
>
> Changes from v6:
>
> This version adds a new 'sym-offset' modifier as requested by Masami.
> I implemented it as a modifier rather than using the trace option as
> suggested, in part because I wanted to keep it all self-contained and
> it seemed more consistent to just add it alongside the 'sym' modifier.
> Also, hist triggers arent't really a tracer and therefore don't
> directly tie into the option update/callback mechanism so making use
> of it isn't as simple as a normal tracer.
>
> I also changed the sort key specification to be stricter and signal an
> error if the specified sort key wasn't found (rather than defaulting
> to hitcount in those cases), also suggested by Masami. Thanks,
> Masami, for your input!
>
> Also updated the Documentation and tracing/README to reflect the
> changes.
>
> Changes from v5:
>
> This version adds support for compound keys, along with the related
> ability to sort using primary and secondary keys. This was mentioned
> in previous versions as the last important piece that remained
> unimplemented, and is now implemented. (I didn't have time to get to
> the couple of enhancements suggested by Masami, but I expect to be
> able to add those later on top of these.)
>
> Because we now support compound keys and it's not immediately clear in
> the output exactly which fields correspond to keys, the key(s),
> compound or not, are now enclosed by curly braces.
>
> The Documentation and README have been updated to reflect the changes,
> and several new examples have been added to illustrate how to use
> compound keys.
>
> Also, the code was updated to work with the new ftrace_event_file,
> etc, renaming in tracing/for-next.
>
> Changes from v4:
>
> This version addresses some problems and suggestions made by Daniel
> Wagner - a lot of the code was reworked to get rid of the distinction
> between keys and values, and as a result, both keys and values can be
> used as sort keys. As suggested, it also allows 'val=' to be absent
> in a trigger command - if no 'val' is specified, hitcount is assumed
> and automatically used as the only val.
>
> The map code was also separated out into a separate file,
> tracing_map.c, allowing it to be reused. It also adds a second tracer
> called function_hist that actually does reuse the code, as an RFC
> patch.
>
> Patch 01/10 [tracing: Update cond flag when enabling or disabling..]
> is a fix for a problem noticed by Daniel and that fixes a problem in
> existing trigger code and should be applied regardless of whether the
> rest of the patchset is merged.
>
> As mentioned, patch 10/10 is an RFC patch implementing a new tracer
> based on the function tracer code. It's a fun little tool and is
> useful for a specific problem I'm working on (and is also a nice test
> of the tracing_map code), but is an RFC because first, I'm not sure it
> would really be of general interest and secondly, it's POC-level
> quality and I'd need to spend more time
Re: [PATCH v9 00/22] tracing: 'hist' triggers
Hi Tom,
Thank you for updating your patches, I'm testing it.
And when I'm testing hist trigger, I've found that the .hex modifiers on value
doesn't work, but no semantic error.
[root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.hex'
events/sched/sched_process_fork/trigger
[root@localhost tracing]#
[root@localhost tracing]# cat events/sched/sched_process_fork/hist
# trigger info:
hist:keys=parent_pid:vals=hitcount,common_pid.hex:sort=hitcount:size=2048
[active]
{ parent_pid: 26582 } hitcount: 1 common_pid: 26582
{ parent_pid: 11968 } hitcount: 1 common_pid: 11968
{ parent_pid: 11956 } hitcount: 2 common_pid: 23912
Totals:
Hits: 4
Entries: 3
Dropped: 0
while other modifiers return -EINVAL.
[root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.execname'
events/sched/sched_process_fork/trigger
-bash: echo: write error: Invalid argument
Thank you,
On 2015/07/17 2:22, Tom Zanussi wrote:
This is v9 of the 'hist triggers' patchset.
Changes from v8:
Same as v8, but with the RFC patch [ftrace: Add function_hist tracer]
removed, and rebased to latest trace/for-next.
Changes from v7:
This version refactors the commits as suggested by Masami. There are
now more commits, but the result should be much more reviewable. The
ending code is the same as before, modulo a couple minor bug fixes I
discovered while refactoring and testing.
I've also reviewed and fixed a number of shortcomings and errors in
the comments, and have added a new discussion of the tracing_map data
structures after Steve mentioned he found them confusing and/or
insufficiently documented.
Also, I kept Namhyung's string patch [tracing: Support string type key
properly] as submitted, but added a follow-on patch that refactors it
and fixes a problem I found with it that enabled static string keys to
contain random chars and therefore incorrect map insertions.
Changes from v6:
This version adds a new 'sym-offset' modifier as requested by Masami.
I implemented it as a modifier rather than using the trace option as
suggested, in part because I wanted to keep it all self-contained and
it seemed more consistent to just add it alongside the 'sym' modifier.
Also, hist triggers arent't really a tracer and therefore don't
directly tie into the option update/callback mechanism so making use
of it isn't as simple as a normal tracer.
I also changed the sort key specification to be stricter and signal an
error if the specified sort key wasn't found (rather than defaulting
to hitcount in those cases), also suggested by Masami. Thanks,
Masami, for your input!
Also updated the Documentation and tracing/README to reflect the
changes.
Changes from v5:
This version adds support for compound keys, along with the related
ability to sort using primary and secondary keys. This was mentioned
in previous versions as the last important piece that remained
unimplemented, and is now implemented. (I didn't have time to get to
the couple of enhancements suggested by Masami, but I expect to be
able to add those later on top of these.)
Because we now support compound keys and it's not immediately clear in
the output exactly which fields correspond to keys, the key(s),
compound or not, are now enclosed by curly braces.
The Documentation and README have been updated to reflect the changes,
and several new examples have been added to illustrate how to use
compound keys.
Also, the code was updated to work with the new ftrace_event_file,
etc, renaming in tracing/for-next.
Changes from v4:
This version addresses some problems and suggestions made by Daniel
Wagner - a lot of the code was reworked to get rid of the distinction
between keys and values, and as a result, both keys and values can be
used as sort keys. As suggested, it also allows 'val=' to be absent
in a trigger command - if no 'val' is specified, hitcount is assumed
and automatically used as the only val.
The map code was also separated out into a separate file,
tracing_map.c, allowing it to be reused. It also adds a second tracer
called function_hist that actually does reuse the code, as an RFC
patch.
Patch 01/10 [tracing: Update cond flag when enabling or disabling..]
is a fix for a problem noticed by Daniel and that fixes a problem in
existing trigger code and should be applied regardless of whether the
rest of the patchset is merged.
As mentioned, patch 10/10 is an RFC patch implementing a new tracer
based on the function tracer code. It's a fun little tool and is
useful for a specific problem I'm working on (and is also a nice test
of the tracing_map code), but is an RFC because first, I'm not sure it
would really be of general interest and secondly, it's POC-level
quality and I'd need to spend more time fixing it up to make it
upstreamable, but I don't want to waste my time if not.
There
Re: [PATCH v9 00/22] tracing: 'hist' triggers
On Tue, 2015-07-21 at 18:40 +0900, Masami Hiramatsu wrote:
Hi Tom,
Thank you for updating your patches, I'm testing it.
And when I'm testing hist trigger, I've found that the .hex modifiers on value
doesn't work, but no semantic error.
Yeah, the hex modifier on values got dropped when refactoring the
patchset, will fix. Thanks for testing and pointing it out.
Tom
[root@localhost tracing]# echo 'hist:keys=parent_pid:vals=common_pid.hex'
events/sched/sched_process_fork/trigger
[root@localhost tracing]#
[root@localhost tracing]# cat events/sched/sched_process_fork/hist
# trigger info:
hist:keys=parent_pid:vals=hitcount,common_pid.hex:sort=hitcount:size=2048
[active]
{ parent_pid: 26582 } hitcount: 1 common_pid: 26582
{ parent_pid: 11968 } hitcount: 1 common_pid: 11968
{ parent_pid: 11956 } hitcount: 2 common_pid: 23912
Totals:
Hits: 4
Entries: 3
Dropped: 0
while other modifiers return -EINVAL.
[root@localhost tracing]# echo
'hist:keys=parent_pid:vals=common_pid.execname'
events/sched/sched_process_fork/trigger
-bash: echo: write error: Invalid argument
Thank you,
On 2015/07/17 2:22, Tom Zanussi wrote:
This is v9 of the 'hist triggers' patchset.
Changes from v8:
Same as v8, but with the RFC patch [ftrace: Add function_hist tracer]
removed, and rebased to latest trace/for-next.
Changes from v7:
This version refactors the commits as suggested by Masami. There are
now more commits, but the result should be much more reviewable. The
ending code is the same as before, modulo a couple minor bug fixes I
discovered while refactoring and testing.
I've also reviewed and fixed a number of shortcomings and errors in
the comments, and have added a new discussion of the tracing_map data
structures after Steve mentioned he found them confusing and/or
insufficiently documented.
Also, I kept Namhyung's string patch [tracing: Support string type key
properly] as submitted, but added a follow-on patch that refactors it
and fixes a problem I found with it that enabled static string keys to
contain random chars and therefore incorrect map insertions.
Changes from v6:
This version adds a new 'sym-offset' modifier as requested by Masami.
I implemented it as a modifier rather than using the trace option as
suggested, in part because I wanted to keep it all self-contained and
it seemed more consistent to just add it alongside the 'sym' modifier.
Also, hist triggers arent't really a tracer and therefore don't
directly tie into the option update/callback mechanism so making use
of it isn't as simple as a normal tracer.
I also changed the sort key specification to be stricter and signal an
error if the specified sort key wasn't found (rather than defaulting
to hitcount in those cases), also suggested by Masami. Thanks,
Masami, for your input!
Also updated the Documentation and tracing/README to reflect the
changes.
Changes from v5:
This version adds support for compound keys, along with the related
ability to sort using primary and secondary keys. This was mentioned
in previous versions as the last important piece that remained
unimplemented, and is now implemented. (I didn't have time to get to
the couple of enhancements suggested by Masami, but I expect to be
able to add those later on top of these.)
Because we now support compound keys and it's not immediately clear in
the output exactly which fields correspond to keys, the key(s),
compound or not, are now enclosed by curly braces.
The Documentation and README have been updated to reflect the changes,
and several new examples have been added to illustrate how to use
compound keys.
Also, the code was updated to work with the new ftrace_event_file,
etc, renaming in tracing/for-next.
Changes from v4:
This version addresses some problems and suggestions made by Daniel
Wagner - a lot of the code was reworked to get rid of the distinction
between keys and values, and as a result, both keys and values can be
used as sort keys. As suggested, it also allows 'val=' to be absent
in a trigger command - if no 'val' is specified, hitcount is assumed
and automatically used as the only val.
The map code was also separated out into a separate file,
tracing_map.c, allowing it to be reused. It also adds a second tracer
called function_hist that actually does reuse the code, as an RFC
patch.
Patch 01/10 [tracing: Update cond flag when enabling or disabling..]
is a fix for a problem noticed by Daniel and that fixes a problem in
existing trigger code and should be applied regardless of whether the
rest of the patchset is merged.
As mentioned, patch 10/10 is an RFC patch implementing a new tracer
based on the function tracer code. It's a fun little tool and is
useful for a specific
