Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-12-01 Thread Jiri Olsa
On Mon, Dec 01, 2014 at 12:28:08PM -0500, Jérémie Galarneau wrote:
> On Fri, Nov 28, 2014 at 6:26 AM, Jiri Olsa  wrote:
> > On Wed, Nov 26, 2014 at 06:37:21PM +0100, Sebastian Andrzej Siewior wrote:
> >> * Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:
> >
> > SNIP
> >
> >>
> >> >This was based on the most recent file format I was aware of, we will
> >> >update it accordingly if required.
> >> >
> >> >Testing welcome!
> >>
> >> I pushed the perf changes I mentioned to
> >>
> >>   git://git.breakpoint.cc/bigeasy/linux.git ctf_convert_7
> >>
> >
> > from perf data convert point of view (and libbabeltrace).. ;-)
> >
> > I tried to convert big perf.data (2GB) and my laptop got stuck
> > for few minutes.. the reason is that perf allocated too much memory
> >
> > I needed to add occasional bt_ctf_stream_flush into the
> > processing (patch attached)
> >
> > What I do now is checking if we processed given amount of events
> > for the stream and once the value is crossed I flush it.
> >
> > My question is if there's a way to find out the allocated memory
> > for the stream?  It'd be nicer to setup maximum allocation size
> > rather than the number of events.
> >
> 
> Not currently, although I agree that this would be a nice feature.
> Perhaps by setting a maximal packet size, CTF Writer could handle the
> automatic flushing.
> 
> Thoughts?

hum, it looks like 2 separated things to me.. the packet size is
result format related size.. I'm more interested in the size of
the 'struct ctf_stream' within the writer during the processing

it seems it wont be trivial to track this AFAICS from sources, but
I guess I can live so far with tracking the the number of events,
until we find something else

thanks,
jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-12-01 Thread Jérémie Galarneau
On Fri, Nov 28, 2014 at 6:26 AM, Jiri Olsa  wrote:
> On Wed, Nov 26, 2014 at 06:37:21PM +0100, Sebastian Andrzej Siewior wrote:
>> * Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:
>
> SNIP
>
>>
>> >This was based on the most recent file format I was aware of, we will
>> >update it accordingly if required.
>> >
>> >Testing welcome!
>>
>> I pushed the perf changes I mentioned to
>>
>>   git://git.breakpoint.cc/bigeasy/linux.git ctf_convert_7
>>
>
> from perf data convert point of view (and libbabeltrace).. ;-)
>
> I tried to convert big perf.data (2GB) and my laptop got stuck
> for few minutes.. the reason is that perf allocated too much memory
>
> I needed to add occasional bt_ctf_stream_flush into the
> processing (patch attached)
>
> What I do now is checking if we processed given amount of events
> for the stream and once the value is crossed I flush it.
>
> My question is if there's a way to find out the allocated memory
> for the stream?  It'd be nicer to setup maximum allocation size
> rather than the number of events.
>

Not currently, although I agree that this would be a nice feature.
Perhaps by setting a maximal packet size, CTF Writer could handle the
automatic flushing.

Thoughts?

Jérémie

> thanks,
> jirka
>
>
>
> diff --git a/tools/perf/util/data-bt.c b/tools/perf/util/data-bt.c
> index c5720e13d8f1..981e8ff2c32a 100644
> --- a/tools/perf/util/data-bt.c
> +++ b/tools/perf/util/data-bt.c
> @@ -39,10 +39,15 @@ struct evsel_priv {
>
>  #define MAX_CPUS   4096
>
> +struct ctf_stream {
> +   struct bt_ctf_stream *stream;
> +   u64 count;
> +};
> +
>  struct ctf_writer {
> /* writer primitives */
> struct bt_ctf_writer*writer;
> -   struct bt_ctf_stream*stream[MAX_CPUS];
> +   struct ctf_stream   *stream[MAX_CPUS];
> struct bt_ctf_stream_class  *stream_class;
> struct bt_ctf_clock *clock;
> u32 last_cpu;
> @@ -377,6 +382,73 @@ static int add_generic_values(struct ctf_writer *cw,
> return 0;
>  }
>
> +#define STREAM_FLUSH_COUNT 1000
> +
> +static bool is_flush_needed(struct ctf_stream *cstream)
> +{
> +   return cstream && (cstream->count >= STREAM_FLUSH_COUNT);
> +}
> +
> +static int flush_stream(struct ctf_writer *cw, int cpu)
> +{
> +   struct ctf_stream *cstream = cw->stream[cpu];
> +   int err = 0;
> +
> +   if (cstream) {
> +   err = bt_ctf_stream_flush(cstream->stream);
> +   if (err)
> +   pr_err("CTF stream %d flush failed\n", cpu);
> +
> +   cstream->count = 0;
> +   }
> +
> +   return err;
> +}
> +
> +static int create_stream(struct ctf_writer *cw, int cpu)
> +{
> +   struct ctf_stream *cstream;
> +   struct bt_ctf_field *pkt_ctx;
> +   struct bt_ctf_field *cpu_field;
> +   struct bt_ctf_stream *stream;
> +   u32 i = cpu;
> +   int ret;
> +
> +   cstream = zalloc(sizeof(*cstream));
> +   if (!cstream) {
> +   pr_err("Failed to allocate ctf stream\n");
> +   return -1;
> +   }
> +
> +   stream = bt_ctf_writer_create_stream(cw->writer, cw->stream_class);
> +   if (!stream)
> +   return -1;
> +
> +   pkt_ctx = bt_ctf_stream_get_packet_context(stream);
> +   if (!pkt_ctx) {
> +   pr_err("Failed to obtain packet context\n");
> +   return -1;
> +   }
> +
> +   cpu_field = bt_ctf_field_structure_get_field(pkt_ctx, "cpu_id");
> +   bt_ctf_field_put(pkt_ctx);
> +   if (!cpu_field) {
> +   pr_err("Failed to obtain cpu field\n");
> +   return -1;
> +   }
> +
> +   ret = bt_ctf_field_unsigned_integer_set_value(cpu_field, i);
> +   if (ret) {
> +   pr_err("Failed to update CPU number\n");
> +   return -1;
> +   }
> +   bt_ctf_field_put(cpu_field);
> +
> +   cstream->stream = stream;
> +   cw->stream[i] = cstream;
> +   return 0;
> +}
> +
>  static int process_sample_event(struct perf_tool *tool,
> union perf_event *_event __maybe_unused,
> struct perf_sample *sample,
> @@ -431,40 +503,14 @@ static int process_sample_event(struct perf_tool *tool,
> cpu = 0;
> }
>
> -   if (!cw->stream[cpu]) {
> -   struct bt_ctf_field *pkt_ctx;
> -   struct bt_ctf_field *cpu_field;
> -   struct bt_ctf_stream *stream;
> -   u32 i = sample->cpu;
> -
> -   stream = bt_ctf_writer_create_stream(cw->writer, 
> cw->stream_class);
> -   if (!stream)
> -   return -1;
> -
> -   pkt_ctx = bt_ctf_stream_get_packet_context(stream);
> -   if (!pkt_ctx) {
> -   pr_err("Failed to obtain packet context\n");
> -   return -1;
> -   }
> -
> -

RE: [tracecompass-dev] Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-29 Thread Jerome CORRENOZ
Hi Alexandre,

Catching with this thread of email: could you please share some links to 
documentation/screenshot about perf support with trace compass ?

Regards,
  Jerome

-Original Message-
From: tracecompass-dev-boun...@eclipse.org 
[mailto:tracecompass-dev-boun...@eclipse.org] On Behalf Of Alexandre Montplaisir
Sent: Thursday, November 27, 2014 5:28 AM
To: Sebastian Andrzej Siewior
Cc: Jeremie Galarneau; linux-kernel@vger.kernel.org; Arnaldo Carvalho de Melo; 
Tom Zanussi; Mathieu Desnoyers; David Ahern; Dominique Toupin; Trace Compass 
Developer Discussions; Jiri Olsa; andr...@linutronix.de
Subject: Re: [tracecompass-dev] Support for Perf CTF traces now in master (was 
Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

On 2014-11-26 12:37 PM, Sebastian Andrzej Siewior wrote:
> * Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:
>
>> Just a quick note, this branch is now merged to master. So anyone who 
>> pulls the code from the master branch at 
>> git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass.g
>> it should be able to load perf-CTF traces in the viewer. The trace 
>> type is now called "Common Trace Format -> Linux Kernel Trace" and 
>> should support both LTTng kernel and perf traces in CTF format 
>> (although auto-detection should work in most cases).
> Thank you for all the work.
> Let me try to reply to the emails at once here:
> - I added to the environment metadata the following (comparing to the
>last version):
> domain => kernel
> tracer_name => perf
>
>There is no tracer_major + minor. Instead I added
>version => perf's version
>On my system I have:
>   release = "3.16.0-4-amd64";
>   version = "3.18.rc3.g91405a";
>
>Because I run Debian's v3.16 and recorded the trace with perf from the
>kernel 3.18.rc3.
>There is no version of perf doing the convert of perf.data => ctf.
>Any objections, rename of the version member?
>
> - Mathieu decided that it makes no sense to add the kernel version to
>each event we trace. Instead each event should have its own version
>with a major/minor member. Once the event is changed the "ABI"
>version should be adjusted. I second this since it makes sense.
>Therefore there are no changes that were made to the converter.
>
> - Alexandre (you) noticed that there are syscall names in the events
>recorded via "sys_enter and sys_exit". This is true, but there is a
>hint. There is an event for instance:
>
> [03:37:07.579969498] (+?.?) raw_syscalls:sys_enter: { cpu_id = 
> 2 }, { perf_ip = 0x81020EBC, perf_tid = 30004, perf_pid = 
> 30004, perf_id = 382, perf_period = 1, common_type = 76, common_flags 
> = 0, common_preempt_count = 0, common_pid = 30004, id = 16, args = [ 
> [0] = 0xE, [1] = 0x2400, [2] = 0x0, [3] = 0x0, [4] = 0xA20F00, [5] = 
> 0xA1FDA0 ] }

Oh ok, so this "id" field is really the system call ID then. Good to know, 
thanks!

>
>By the end you notice id=16 and args. args are the Arguments passed
>to syscall and id is the syscall number. Together with machine =
>x86_64 you know which architecture you need to lookup the number 16.
>The numbers are from unistd.h (and may be different between architectures,
>even between i386 & x86_64). strace has for instance the following [0] 
> table.
>
> { 3,TD, sys_read,   "read"  },  /* 0 */
> …
> { 3,TD, sys_ioctl,  "ioctl" },  /* 16 */
> …
>
> So 16 is ioctl. strace has those tables for a bunch of architectures 
> so it might be helpful to suck them in. I know no other way to ease 
> things here.

Indeed. Well this information could be part of the trace metadata too, but I 
guess that wouldn't be very practical.
We'll just need to add a way for each supported tracer to advertize how it gets 
its system call names.

>
> [0] https://github.com/bnoordhuis/strace/blob/master/linux/x86_64/syscallent.h
>
> The same thing is true for softirq_entry events for instance. This event
> will give you you only vec=9 and you need to lookup that 9 => RCU. That
> one is easy however:
>
>   const char * const softirq_to_name[NR_SOFTIRQS] = {
>   "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "BLOCK_IOPOLL",
>   "TASKLET", "SCHED", "HRTIMER", "RCU"
>   };
>
> this has been taken from kernel/softirq.c.

Oh, that's right, we never got around to getting/showing the actual 
names of the soft IRQs. Thanks for reminding us. ;)

>> This was based on the most recent file format I was

Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-28 Thread Jiri Olsa
On Wed, Nov 26, 2014 at 06:37:21PM +0100, Sebastian Andrzej Siewior wrote:
> * Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:

SNIP

> 
> >This was based on the most recent file format I was aware of, we will
> >update it accordingly if required.
> >
> >Testing welcome!
> 
> I pushed the perf changes I mentioned to 
> 
>   git://git.breakpoint.cc/bigeasy/linux.git ctf_convert_7
> 

from perf data convert point of view (and libbabeltrace).. ;-)

I tried to convert big perf.data (2GB) and my laptop got stuck
for few minutes.. the reason is that perf allocated too much memory

I needed to add occasional bt_ctf_stream_flush into the
processing (patch attached)

What I do now is checking if we processed given amount of events
for the stream and once the value is crossed I flush it.

My question is if there's a way to find out the allocated memory
for the stream?  It'd be nicer to setup maximum allocation size
rather than the number of events.

thanks,
jirka



diff --git a/tools/perf/util/data-bt.c b/tools/perf/util/data-bt.c
index c5720e13d8f1..981e8ff2c32a 100644
--- a/tools/perf/util/data-bt.c
+++ b/tools/perf/util/data-bt.c
@@ -39,10 +39,15 @@ struct evsel_priv {
 
 #define MAX_CPUS   4096
 
+struct ctf_stream {
+   struct bt_ctf_stream *stream;
+   u64 count;
+};
+
 struct ctf_writer {
/* writer primitives */
struct bt_ctf_writer*writer;
-   struct bt_ctf_stream*stream[MAX_CPUS];
+   struct ctf_stream   *stream[MAX_CPUS];
struct bt_ctf_stream_class  *stream_class;
struct bt_ctf_clock *clock;
u32 last_cpu;
@@ -377,6 +382,73 @@ static int add_generic_values(struct ctf_writer *cw,
return 0;
 }
 
+#define STREAM_FLUSH_COUNT 1000
+
+static bool is_flush_needed(struct ctf_stream *cstream)
+{
+   return cstream && (cstream->count >= STREAM_FLUSH_COUNT);
+}
+
+static int flush_stream(struct ctf_writer *cw, int cpu)
+{
+   struct ctf_stream *cstream = cw->stream[cpu];
+   int err = 0;
+
+   if (cstream) {
+   err = bt_ctf_stream_flush(cstream->stream);
+   if (err)
+   pr_err("CTF stream %d flush failed\n", cpu);
+
+   cstream->count = 0;
+   }
+
+   return err;
+}
+
+static int create_stream(struct ctf_writer *cw, int cpu)
+{
+   struct ctf_stream *cstream;
+   struct bt_ctf_field *pkt_ctx;
+   struct bt_ctf_field *cpu_field;
+   struct bt_ctf_stream *stream;
+   u32 i = cpu;
+   int ret;
+
+   cstream = zalloc(sizeof(*cstream));
+   if (!cstream) {
+   pr_err("Failed to allocate ctf stream\n");
+   return -1;
+   }
+
+   stream = bt_ctf_writer_create_stream(cw->writer, cw->stream_class);
+   if (!stream)
+   return -1;
+
+   pkt_ctx = bt_ctf_stream_get_packet_context(stream);
+   if (!pkt_ctx) {
+   pr_err("Failed to obtain packet context\n");
+   return -1;
+   }
+
+   cpu_field = bt_ctf_field_structure_get_field(pkt_ctx, "cpu_id");
+   bt_ctf_field_put(pkt_ctx);
+   if (!cpu_field) {
+   pr_err("Failed to obtain cpu field\n");
+   return -1;
+   }
+
+   ret = bt_ctf_field_unsigned_integer_set_value(cpu_field, i);
+   if (ret) {
+   pr_err("Failed to update CPU number\n");
+   return -1;
+   }
+   bt_ctf_field_put(cpu_field);
+
+   cstream->stream = stream;
+   cw->stream[i] = cstream;
+   return 0;
+}
+
 static int process_sample_event(struct perf_tool *tool,
union perf_event *_event __maybe_unused,
struct perf_sample *sample,
@@ -431,40 +503,14 @@ static int process_sample_event(struct perf_tool *tool,
cpu = 0;
}
 
-   if (!cw->stream[cpu]) {
-   struct bt_ctf_field *pkt_ctx;
-   struct bt_ctf_field *cpu_field;
-   struct bt_ctf_stream *stream;
-   u32 i = sample->cpu;
-
-   stream = bt_ctf_writer_create_stream(cw->writer, 
cw->stream_class);
-   if (!stream)
-   return -1;
-
-   pkt_ctx = bt_ctf_stream_get_packet_context(stream);
-   if (!pkt_ctx) {
-   pr_err("Failed to obtain packet context\n");
-   return -1;
-   }
-
-   cpu_field = bt_ctf_field_structure_get_field(pkt_ctx, "cpu_id");
-   bt_ctf_field_put(pkt_ctx);
-   if (!cpu_field) {
-   pr_err("Failed to obtain cpu field\n");
-   return -1;
-   }
+   if (!cw->stream[cpu] && create_stream(cw, cpu))
+   return -1;
 
-   ret = bt_ctf_field_unsigned_integer_set_value(cpu_field, i);
-   if (ret) {
-   pr_err("Failed to upda

Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-28 Thread Jiri Olsa
On Thu, Nov 27, 2014 at 01:31:38PM -0500, Alexandre Montplaisir wrote:
> 
> On 11/27/2014 10:43 AM, Jiri Olsa wrote:
> >On Wed, Nov 12, 2014 at 05:14:45PM -0500, Alexandre Montplaisir wrote:
> >>
> >>Testing welcome!
> >hi,
> >any other way besides compiling eclipse to test this? For pure mortals
> >with Fedora eclipse rpm.. ;-)
> 
> If you already have an Eclipse installation, you can use the update site:
> http://download.eclipse.org/tracecompass/master/nightly/
> This would install the plugins into that Eclipse install.
> 
> We will also put nightly builds of the stand-alone version on the download
> page Very Soon(TM). We just have a few issues left to figure out.

cool, I'll check

thanks,
jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-27 Thread Alexandre Montplaisir


On 11/27/2014 10:43 AM, Jiri Olsa wrote:

On Wed, Nov 12, 2014 at 05:14:45PM -0500, Alexandre Montplaisir wrote:


Testing welcome!

hi,
any other way besides compiling eclipse to test this? For pure mortals
with Fedora eclipse rpm.. ;-)


If you already have an Eclipse installation, you can use the update site:
http://download.eclipse.org/tracecompass/master/nightly/
This would install the plugins into that Eclipse install.

We will also put nightly builds of the stand-alone version on the 
download page Very Soon(TM). We just have a few issues left to figure out.


Cheers,
Alexandre


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-27 Thread Sebastian Andrzej Siewior
On 11/27/2014 04:43 PM, Jiri Olsa wrote:

> hi,
> any other way besides compiling eclipse to test this? For pure mortals
> with Fedora eclipse rpm.. ;-)
> 
> or any instructions for the compilation.. I actually haven't checked yet

| mvn clean install -Pbuild-rcp -Dmaven.test.skip=true

does the trick. It took a while to complete especially since it sucked
some jars at 30KiB/sec

That archive at

https://breakpoint.cc/perf-ctf/trace-compass-0.1.0-20141126-1744-linux.gtk.x86_64.tar.xz

contains the standalone application "tracecompass" without eclipse. It
contains also the patch I quoted in the thread.

> thanks,
> jirka

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-27 Thread Jiri Olsa
On Wed, Nov 12, 2014 at 05:14:45PM -0500, Alexandre Montplaisir wrote:
> 
> On 11/09/2014 08:31 PM, Alexandre Montplaisir wrote:
> >On 2014-11-05 10:25 PM, Alexandre Montplaisir wrote:
> >>
> >>>
> But if you could for example tell me the perf equivalents of all the
> strings in that file, I could hack together such wrapper. With that,
> in theory, perf traces should behave exactly the same as LTTng traces
> in the viewer!
> >>>Oooh, that would be awesome. So I installed maven but didn't get much
> >>>further. Let me gather this for you.
> >>
> >>Awesome, thanks!
> >>
> >>I am travelling this week, so I'm a bit busy, but I will try to
> >>prototype a "wrapper" for the kernel analysis, and adding support for
> >>the perf events, whenever I have a chance. I'll keep you posted.
> >
> >Ok, some good news!
> >
> >I managed to get the CTF traces from perf working in Trace Compass! See
> >attached screenshots. This is showing the "ctf-out2" trace from your
> >previous email. The other trace seems to have less events enabled, so it
> >would only show some WAIT_FOR_CPU states in the view.
> >
> >If anybody wishes to try it, you can grab the whole branch ending at
> >https://git.eclipse.org/r/#/c/36200/ . Or run:
> >$ git fetch
> >git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass
> >refs/changes/00/36200/3 && git checkout FETCH_HEAD
> 
> Just a quick note, this branch is now merged to master. So anyone who pulls
> the code from the master branch at
> git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass.git
> should be able to load perf-CTF traces in the viewer. The trace type is now
> called "Common Trace Format -> Linux Kernel Trace" and should support both
> LTTng kernel and perf traces in CTF format (although auto-detection should
> work in most cases).
> 
> This was based on the most recent file format I was aware of, we will update
> it accordingly if required.
> 
> Testing welcome!

hi,
any other way besides compiling eclipse to test this? For pure mortals
with Fedora eclipse rpm.. ;-)

or any instructions for the compilation.. I actually haven't checked yet

thanks,
jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/


Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-26 Thread Alexandre Montplaisir

On 2014-11-26 12:37 PM, Sebastian Andrzej Siewior wrote:

* Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:


Just a quick note, this branch is now merged to master. So anyone who
pulls the code from the master branch at
git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass.git
should be able to load perf-CTF traces in the viewer. The trace type
is now called "Common Trace Format -> Linux Kernel Trace" and should
support both LTTng kernel and perf traces in CTF format (although
auto-detection should work in most cases).

Thank you for all the work.
Let me try to reply to the emails at once here:
- I added to the environment metadata the following (comparing to the
   last version):
domain => kernel
tracer_name => perf

   There is no tracer_major + minor. Instead I added
   version => perf's version
   On my system I have:
  release = "3.16.0-4-amd64";
  version = "3.18.rc3.g91405a";

   Because I run Debian's v3.16 and recorded the trace with perf from the
   kernel 3.18.rc3.
   There is no version of perf doing the convert of perf.data => ctf.
   Any objections, rename of the version member?

- Mathieu decided that it makes no sense to add the kernel version to
   each event we trace. Instead each event should have its own version
   with a major/minor member. Once the event is changed the "ABI"
   version should be adjusted. I second this since it makes sense.
   Therefore there are no changes that were made to the converter.

- Alexandre (you) noticed that there are syscall names in the events
   recorded via "sys_enter and sys_exit". This is true, but there is a
   hint. There is an event for instance:

[03:37:07.579969498] (+?.?) raw_syscalls:sys_enter: { cpu_id = 2 }, { 
perf_ip = 0x81020EBC, perf_tid = 30004, perf_pid = 30004, perf_id = 
382, perf_period = 1, common_type = 76, common_flags = 0, common_preempt_count 
= 0, common_pid = 30004, id = 16, args = [ [0] = 0xE, [1] = 0x2400, [2] = 0x0, 
[3] = 0x0, [4] = 0xA20F00, [5] = 0xA1FDA0 ] }


Oh ok, so this "id" field is really the system call ID then. Good to 
know, thanks!




   By the end you notice id=16 and args. args are the Arguments passed
   to syscall and id is the syscall number. Together with machine =
   x86_64 you know which architecture you need to lookup the number 16.
   The numbers are from unistd.h (and may be different between architectures,
   even between i386 & x86_64). strace has for instance the following [0] table.

{ 3,TD, sys_read,   "read"  },  /* 0 */
…
{ 3,TD, sys_ioctl,  "ioctl" },  /* 16 */
…

So 16 is ioctl. strace has those tables for a bunch of architectures
so it might be helpful to suck them in. I know no other way to ease
things here.


Indeed. Well this information could be part of the trace metadata too, 
but I guess that wouldn't be very practical.
We'll just need to add a way for each supported tracer to advertize how 
it gets its system call names.




[0] https://github.com/bnoordhuis/strace/blob/master/linux/x86_64/syscallent.h

The same thing is true for softirq_entry events for instance. This event
will give you you only vec=9 and you need to lookup that 9 => RCU. That
one is easy however:

  const char * const softirq_to_name[NR_SOFTIRQS] = {
  "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "BLOCK_IOPOLL",
  "TASKLET", "SCHED", "HRTIMER", "RCU"
  };

this has been taken from kernel/softirq.c.


Oh, that's right, we never got around to getting/showing the actual 
names of the soft IRQs. Thanks for reminding us. ;)



This was based on the most recent file format I was aware of, we will
update it accordingly if required.

Testing welcome!

I pushed the perf changes I mentioned to

   git://git.breakpoint.cc/bigeasy/linux.git ctf_convert_7

It is now based on Arnaldo's perf/core. If everything goes well from
the compass side and nobody complains here in any way, the next step
would be to present the patches on the mailing list and compass as a
user.

I took you tree and added the patch below. I uploaded the following
files to https://breakpoint.cc/perf-ctf/:
- ctf-out6.tar.xz from perf6.data.xz
   shows nothing


Hmm, indeed it throws exceptions in the console when trying to validate 
the trace. It seems to read a packet size in perf_stream_0 as a negative 
value. Babeltrace handles it fine though, so we're probably reading it 
wrong. We'll investigate.


Cheers,
Alexandre



- ctf-out7.tar.xz from perf7.data.xz
   shows something

The only obvious difference is the size of the CTF data. The size of out6
is almost 300MiB and it contains 3,259,929 events. The out7 is has only
15MiB and contains 152,900 events.


Cheers,
Alexandre

✂

 From 7ffa619d918f2010046b391ae29063ffc5329468 Mon Sep 17 00:00:00 2001
From: Sebastian Andrzej Siewior 
Date: Wed, 26 Nov 2014 18:04:53 +0100
Subject: [PATCH] CTF: use tracer_name for perf-CTF traces

domain will be set to kernel for both, perf and lttng traces. The
tracer_name will

Re: Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-26 Thread Sebastian Andrzej Siewior
* Alexandre Montplaisir | 2014-11-12 17:14:45 [-0500]:

>Just a quick note, this branch is now merged to master. So anyone who
>pulls the code from the master branch at
>git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass.git
>should be able to load perf-CTF traces in the viewer. The trace type
>is now called "Common Trace Format -> Linux Kernel Trace" and should
>support both LTTng kernel and perf traces in CTF format (although
>auto-detection should work in most cases).

Thank you for all the work.
Let me try to reply to the emails at once here:
- I added to the environment metadata the following (comparing to the 
  last version):
   domain => kernel
   tracer_name => perf

  There is no tracer_major + minor. Instead I added
  version => perf's version
  On my system I have:
 release = "3.16.0-4-amd64";
 version = "3.18.rc3.g91405a";

  Because I run Debian's v3.16 and recorded the trace with perf from the
  kernel 3.18.rc3.
  There is no version of perf doing the convert of perf.data => ctf.
  Any objections, rename of the version member?

- Mathieu decided that it makes no sense to add the kernel version to 
  each event we trace. Instead each event should have its own version 
  with a major/minor member. Once the event is changed the "ABI" 
  version should be adjusted. I second this since it makes sense.
  Therefore there are no changes that were made to the converter.

- Alexandre (you) noticed that there are syscall names in the events
  recorded via "sys_enter and sys_exit". This is true, but there is a
  hint. There is an event for instance:

[03:37:07.579969498] (+?.?) raw_syscalls:sys_enter: { cpu_id = 2 }, { 
perf_ip = 0x81020EBC, perf_tid = 30004, perf_pid = 30004, perf_id = 
382, perf_period = 1, common_type = 76, common_flags = 0, common_preempt_count 
= 0, common_pid = 30004, id = 16, args = [ [0] = 0xE, [1] = 0x2400, [2] = 0x0, 
[3] = 0x0, [4] = 0xA20F00, [5] = 0xA1FDA0 ] }

  By the end you notice id=16 and args. args are the Arguments passed 
  to syscall and id is the syscall number. Together with machine = 
  x86_64 you know which architecture you need to lookup the number 16.
  The numbers are from unistd.h (and may be different between architectures,
  even between i386 & x86_64). strace has for instance the following [0] table.

{ 3,TD, sys_read,   "read"  },  /* 0 */
…
{ 3,TD, sys_ioctl,  "ioctl" },  /* 16 */
…

So 16 is ioctl. strace has those tables for a bunch of architectures 
so it might be helpful to suck them in. I know no other way to ease
things here.

[0] https://github.com/bnoordhuis/strace/blob/master/linux/x86_64/syscallent.h

The same thing is true for softirq_entry events for instance. This event
will give you you only vec=9 and you need to lookup that 9 => RCU. That
one is easy however:

 const char * const softirq_to_name[NR_SOFTIRQS] = {
 "HI", "TIMER", "NET_TX", "NET_RX", "BLOCK", "BLOCK_IOPOLL",
 "TASKLET", "SCHED", "HRTIMER", "RCU"
 };

this has been taken from kernel/softirq.c. 

>This was based on the most recent file format I was aware of, we will
>update it accordingly if required.
>
>Testing welcome!

I pushed the perf changes I mentioned to 

  git://git.breakpoint.cc/bigeasy/linux.git ctf_convert_7

It is now based on Arnaldo's perf/core. If everything goes well from 
the compass side and nobody complains here in any way, the next step 
would be to present the patches on the mailing list and compass as a
user.

I took you tree and added the patch below. I uploaded the following
files to https://breakpoint.cc/perf-ctf/:
- ctf-out6.tar.xz from perf6.data.xz 
  shows nothing

- ctf-out7.tar.xz from perf7.data.xz
  shows something

The only obvious difference is the size of the CTF data. The size of out6
is almost 300MiB and it contains 3,259,929 events. The out7 is has only
15MiB and contains 152,900 events.

>Cheers,
>Alexandre

✂

>From 7ffa619d918f2010046b391ae29063ffc5329468 Mon Sep 17 00:00:00 2001
From: Sebastian Andrzej Siewior 
Date: Wed, 26 Nov 2014 18:04:53 +0100
Subject: [PATCH] CTF: use tracer_name for perf-CTF traces

domain will be set to kernel for both, perf and lttng traces. The
tracer_name will be set to perf if the trace is generated by perf and
otherwise lttng-modules if created by thet lttng tool.

Signed-off-by: Sebastian Andrzej Siewior 
---
 .../tracecompass/lttng2/kernel/core/trace/LttngKernelTrace.java  | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git 
a/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/lttng2/kernel/core/trace/LttngKernelTrace.java
 
b/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/lttng2/kernel/core/trace/LttngKernelTrace.java
index a58269f..03a09b9 100644
--- 
a/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/tracecompass/lttng2/kernel/core/trace/LttngKernelTrace.java
+++ 
b/org.eclipse.tracecompass.lttng2.kernel.core/src/org/eclipse/traceco

Support for Perf CTF traces now in master (was Re: FW: [RFC 0/5] perf tools: Add perf data CTF conversion)

2014-11-12 Thread Alexandre Montplaisir


On 11/09/2014 08:31 PM, Alexandre Montplaisir wrote:

On 2014-11-05 10:25 PM, Alexandre Montplaisir wrote:





But if you could for example tell me the perf equivalents of all the
strings in that file, I could hack together such wrapper. With that,
in theory, perf traces should behave exactly the same as LTTng traces
in the viewer!

Oooh, that would be awesome. So I installed maven but didn't get much
further. Let me gather this for you.


Awesome, thanks!

I am travelling this week, so I'm a bit busy, but I will try to 
prototype a "wrapper" for the kernel analysis, and adding support for 
the perf events, whenever I have a chance. I'll keep you posted.


Ok, some good news!

I managed to get the CTF traces from perf working in Trace Compass! 
See attached screenshots. This is showing the "ctf-out2" trace from 
your previous email. The other trace seems to have less events 
enabled, so it would only show some WAIT_FOR_CPU states in the view.


If anybody wishes to try it, you can grab the whole branch ending at 
https://git.eclipse.org/r/#/c/36200/ . Or run:
$ git fetch 
git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass 
refs/changes/00/36200/3 && git checkout FETCH_HEAD


Just a quick note, this branch is now merged to master. So anyone who 
pulls the code from the master branch at

git://git.eclipse.org/gitroot/tracecompass/org.eclipse.tracecompass.git
should be able to load perf-CTF traces in the viewer. The trace type is 
now called "Common Trace Format -> Linux Kernel Trace" and should 
support both LTTng kernel and perf traces in CTF format (although 
auto-detection should work in most cases).


This was based on the most recent file format I was aware of, we will 
update it accordingly if required.


Testing welcome!

Cheers,
Alexandre



It reuses much of the code from the LTTng analysis, which is why it 
was relatively quick to do. For now, it looks for the domain in the 
CTF environment to be "kernel-perf". But this will be easy to update, 
if needed, once the final format is decided.


Maybe I missed it, but I couldn't find the system call names in the 
trace. Using the sys_enter and sys_exit events, the viewer is able to 
determine the kernel-mode states (in blue), but we cannot show the 
exact system call names like we do with LTTng.
There is also something weird with the arrows in the Control Flow View 
(disabled in the screenshot), I don't know if it's due to the 
particularity of the trace or to a bug in the view. We'll investigate.


Feedback is very welcome.

Cheers,
Alexandre


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/