Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
>> I suppose that if you execute the stap script I pasted it will show the 
>> proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Sorry for the delay.  SystemTap static probes appear to work correctly on
> Fedora 24.

> I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
> tried the following:

> $ rpm -qi systemtap kernel-devel | grep Source
> Source RPM  : systemtap-3.0-3.fc24.src.rpm
> Source RPM  : kernel-4.7.2-201.fc24.src.rpm

> (By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap 
> as you.)

> $ cat test.stp
> probe begin {
>   printf("hello\n");
> }

> probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
> {
>   printf("%x %p\n", $arg1, $arg2)
> }

> $ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
> hello
> ae00 0x0
> ae03 0xa
> ae03 0x9
> ae03 0x42
> ae01 0x0

> These are valid argument values.  What happens on your Debian box?

Like in my example, I get all zeroes. So I'll open a bug on debian and assume my
example stap script shows the proper values.

Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Mon, Sep 05, 2016 at 04:37:01PM +0200, Lluís Vilanova wrote:
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Sorry for the delay.  SystemTap static probes appear to work correctly on
Fedora 24.

I built qemu.git/master from source with "--enable-trace-backends=dtrace" and
tried the following:

$ rpm -qi systemtap kernel-devel | grep Source
Source RPM  : systemtap-3.0-3.fc24.src.rpm
Source RPM  : kernel-4.7.2-201.fc24.src.rpm

(By the way, I hit the same mutex_lock() vs inode_lock() issue in systemtap as 
you.)

$ cat test.stp
probe begin {
printf("hello\n");
}

probe process("path/to/qemu-system-x86_64").mark("kvm_ioctl")
{
printf("%x %p\n", $arg1, $arg2)
}

$ sudo stap test.stp -c 'path/to/qemu-system-x86_64 -enable-kvm -m 1024'
hello
ae00 0x0
ae03 0xa
ae03 0x9
ae03 0x42
ae01 0x0

These are valid argument values.  What happens on your Debian box?

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Masami Hiramatsu writes:

> On Mon, 05 Sep 2016 16:37:01 +0200
> Lluís Vilanova  wrote:

>> Stefan Hajnoczi writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface 
>> >> >> from
>> >> >> within the probe code (probes execute in kernel mode, including "guru 
>> >> >> mode"
>> >> >> code).
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to 
>> >> change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include 
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
>> >> localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> >> 
>> >> probe begin {
>> >> printf("hello\n")
>> >> }
>> >> probe 
>> >> process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> >> {
>> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> >> event($arg1, $arg2, $arg3)
>> >> exit()
>> >> }
>> >> 
>> >> The only caveat is that you must pass the "-g" argument to stap.
>> >> 
>> >> Also, for some reason the printf in the probe always prints zeros, no 
>> >> matter
>> >> what the actual event receives (I've debugged QEMU down to the call to the
>> >> auto-generated stap functions). Could this be an error in systemtap?
>> 
>> > It's strange that arguments do not have valid values.  Debugging the
>> > stap functions is the next step if you want to figure out what happened.
>> > I've never had this issue before so maybe something with Debian
>> > SystemTap userspace probes is broken.
>> 
>> I already debugged it, to the point where QEMU executes the trap injected by
>> systemtap, and the register values that were supposed to hold the arguments 
>> are
>> correct.
>> 
>> I suppose that if you execute the stap script I pasted it will show the 
>> proper
>> values. Then it's definitely a problem with Debian's userspace probes.

> Would you have tried to update your kernel to mainline and tested it ?

I've compiled the tarball for 4.8-rc5 (.config from "make localmodconfig") and a
printf of the probe arguments still shows zeroes. Also, I've had to add a small
patch to [1] to properly lock/unlock inodes in this kernel version (using
inode_lock/unlock instead of mutex_lock/unlock on inode->i_mutex).

[1] /usr/share/systemtap/runtime/transport/transport.c


> If it occurs, you also should try to use a raw uprobe via 
> ftrace(uprobe_events)
> and perftools.
> If you have the latest perf (maybe you'll need checkout the latest tip tree),
> you can use SDT as below (currently it doesn't support args, so you'll need
> debuginfo.)

> # perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
> # perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 
> 'guest_mem_before_exec $vars'

> And you'll see new event is registered which can be traced by ftrace or perf.

It does show something (I'm interested in stap probe "guest_hypertrace", raised
on function "trace_guest_hypertrace"), but is incorrect:

50.00%  (55e36d5ee32b) __cpu=0x55e370446fd0 arg1=0xcafe
50.00%  (55e36d5ee32b) __cpu=0x7f2ee0a10b20 arg1=0x198

My test app calls "trace_guest_hypertrace" twice, always with the same "__cpu"
and "arg1" argument values. Just in case, this is runnign a QEMU compiled with
"-O0".

Running "record" multiple times shows different "random" values on the
arguments, and keeps changing which of the two trace elements shows incorrect
values.


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
>> Daniel P Berrange writes:
>> 
>> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Stefan Hajnoczi writes:
>> >> 
>> >> > When SystemTap is used the QEMU monitor interface does nothing.
>> >> 
>> >> That's not what I've experienced. I was able to use a stap script to 
>> >> change the
>> >> tracing state of events:
>> >> 
>> >> #!/usr/bin/env stap
>> >> 
>> >> %{
>> >> #include 
>> >> %}
>> >> 
>> >> function event:long(cpu:long, addr:long, info:long)
>> >> %{
>> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
>> >> localhost 1234", NULL};
>> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> >> STAP_RETURN(0);
>> >> %}
>> 
>> > I don't know what you're trying to achieve here.  The trace-event state,
>> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
>> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
>> > so the QEMU event state has no effect on them. The probe points in the
>> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
>> > will automatically enable an event if you write a dtrace script that uses
>> > the event.
>> 
>> Sorry, I did not properly explain the use case. This is an example of using
>> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
>> backend to trace events to disk, and the "dtrace" backend (systemtap) to 
>> control
>> the tracing state of such events.

> Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
> the data to be recorded, and avoid the log backend entirely.

Backends with a buffered output stream like "simple" should be faster than
"dtrace", which AFAIK injects a software trap. Also, you can take advantage of
existing QEMU infrastructure to control the tracing of events, instead of
implementing your own on each script.

Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Mon, 05 Sep 2016 16:37:01 +0200
Lluís Vilanova  wrote:

> Stefan Hajnoczi writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> >> Also, I'm still not sure how to interact with QEMU's monitor interface 
> >> >> from
> >> >> within the probe code (probes execute in kernel mode, including "guru 
> >> >> mode"
> >> >> code).
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to 
> >> change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include 
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
> >> localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> >> 
> >> probe begin {
> >> printf("hello\n")
> >> }
> >> probe 
> >> process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
> >> {
> >> printf("%x %d %d\n", $arg1, $arg2, $arg3)
> >> event($arg1, $arg2, $arg3)
> >> exit()
> >> }
> >> 
> >> The only caveat is that you must pass the "-g" argument to stap.
> >> 
> >> Also, for some reason the printf in the probe always prints zeros, no 
> >> matter
> >> what the actual event receives (I've debugged QEMU down to the call to the
> >> auto-generated stap functions). Could this be an error in systemtap?
> 
> > It's strange that arguments do not have valid values.  Debugging the
> > stap functions is the next step if you want to figure out what happened.
> > I've never had this issue before so maybe something with Debian
> > SystemTap userspace probes is broken.
> 
> I already debugged it, to the point where QEMU executes the trap injected by
> systemtap, and the register values that were supposed to hold the arguments 
> are
> correct.
> 
> I suppose that if you execute the stap script I pasted it will show the proper
> values. Then it's definitely a problem with Debian's userspace probes.

Would you have tried to update your kernel to mainline and tested it ?
If it occurs, you also should try to use a raw uprobe via ftrace(uprobe_events)
and perftools.
If you have the latest perf (maybe you'll need checkout the latest tip tree),
you can use SDT as below (currently it doesn't support args, so you'll need
debuginfo.)

# perf buildid-cache --add ./install/vanilla/bin/qemu-system-i386
# perf probe -x ./install/vanilla/bin/qemu-system-i386 -a 
'guest_mem_before_exec $vars'

And you'll see new event is registered which can be traced by ftrace or perf.

Thanks,

-- 
Masami Hiramatsu 

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Sep 05, 2016 at 08:29:54PM +0200, Lluís Vilanova wrote:
> Daniel P Berrange writes:
> 
> > On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Stefan Hajnoczi writes:
> >> 
> >> > When SystemTap is used the QEMU monitor interface does nothing.
> >> 
> >> That's not what I've experienced. I was able to use a stap script to 
> >> change the
> >> tracing state of events:
> >> 
> >> #!/usr/bin/env stap
> >> 
> >> %{
> >> #include 
> >> %}
> >> 
> >> function event:long(cpu:long, addr:long, info:long)
> >> %{
> >> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
> >> localhost 1234", NULL};
> >> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
> >> STAP_RETURN(0);
> >> %}
> 
> > I don't know what you're trying to achieve here.  The trace-event state,
> > as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> > backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> > so the QEMU event state has no effect on them. The probe points in the
> > binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> > will automatically enable an event if you write a dtrace script that uses
> > the event.
> 
> Sorry, I did not properly explain the use case. This is an example of using
> QEMU's tracing infrastructure to control itself. Here I'm using the "log"
> backend to trace events to disk, and the "dtrace" backend (systemtap) to 
> control
> the tracing state of such events.

Ah, I see. I guess I'd personally just have systemtap/dtrace directly emit
the data to be recorded, and avoid the log backend entirely.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Daniel P Berrange writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change 
>> the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include 
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
>> localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}

> I don't know what you're trying to achieve here.  The trace-event state,
> as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
> backend. dtrace and ltt-ust are both fully dynamic trace event backends,
> so the QEMU event state has no effect on them. The probe points in the
> binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
> will automatically enable an event if you write a dtrace script that uses
> the event.

Sorry, I did not properly explain the use case. This is an example of using
QEMU's tracing infrastructure to control itself. Here I'm using the "log"
backend to trace events to disk, and the "dtrace" backend (systemtap) to control
the tracing state of such events.

The guest code is something like:

  // some guest code executed *without* tracing enabled

  invoke_hypertrace_event(); // stap will enable some events here

  // some guest code executed *with* tracing enabled

  invoke_hypertrace_event(); // stap will disable some events here

  // some guest code executed *without* tracing enabled

So we can use it to mark regions of interest for tracing.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Daniel P. Berrange]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change 
> the
> tracing state of events:
> 
>#!/usr/bin/env stap
> 
>%{
>#include 
>%}
> 
>function event:long(cpu:long, addr:long, info:long)
>%{
>char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
> localhost 1234", NULL};
>call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>STAP_RETURN(0);
>%}

I don't know what you're trying to achieve here.  The trace-event state,
as changed/viewed via QEMU monitor, is irrelevant to the dtrace (systemtap)
backend. dtrace and ltt-ust are both fully dynamic trace event backends,
so the QEMU event state has no effect on them. The probe points in the
binary are dynamically enabled / disabled by the dtrace runtime. ie dtrace
will automatically enable an event if you write a dtrace script that uses
the event.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
>> >> Also, I'm still not sure how to interact with QEMU's monitor interface 
>> >> from
>> >> within the probe code (probes execute in kernel mode, including "guru 
>> >> mode"
>> >> code).
>> 
>> > When SystemTap is used the QEMU monitor interface does nothing.
>> 
>> That's not what I've experienced. I was able to use a stap script to change 
>> the
>> tracing state of events:
>> 
>> #!/usr/bin/env stap
>> 
>> %{
>> #include 
>> %}
>> 
>> function event:long(cpu:long, addr:long, info:long)
>> %{
>> char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
>> localhost 1234", NULL};
>> call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>> STAP_RETURN(0);
>> %}
>> 
>> probe begin {
>> printf("hello\n")
>> }
>> probe 
>> process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>> {
>> printf("%x %d %d\n", $arg1, $arg2, $arg3)
>> event($arg1, $arg2, $arg3)
>> exit()
>> }
>> 
>> The only caveat is that you must pass the "-g" argument to stap.
>> 
>> Also, for some reason the printf in the probe always prints zeros, no matter
>> what the actual event receives (I've debugged QEMU down to the call to the
>> auto-generated stap functions). Could this be an error in systemtap?

> It's strange that arguments do not have valid values.  Debugging the
> stap functions is the next step if you want to figure out what happened.
> I've never had this issue before so maybe something with Debian
> SystemTap userspace probes is broken.

I already debugged it, to the point where QEMU executes the trap injected by
systemtap, and the register values that were supposed to hold the arguments are
correct.

I suppose that if you execute the stap script I pasted it will show the proper
values. Then it's definitely a problem with Debian's userspace probes.


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Mon, Aug 29, 2016 at 08:46:02PM +0200, Lluís Vilanova wrote:
> >> Also, I'm still not sure how to interact with QEMU's monitor interface from
> >> within the probe code (probes execute in kernel mode, including "guru mode"
> >> code).
> 
> > When SystemTap is used the QEMU monitor interface does nothing.
> 
> That's not what I've experienced. I was able to use a stap script to change 
> the
> tracing state of events:
> 
>#!/usr/bin/env stap
> 
>%{
>#include 
>%}
> 
>function event:long(cpu:long, addr:long, info:long)
>%{
>char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
> localhost 1234", NULL};
>call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
>STAP_RETURN(0);
>%}
> 
>probe begin {
>printf("hello\n")
>}
>probe 
> process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
>{
>printf("%x %d %d\n", $arg1, $arg2, $arg3)
>event($arg1, $arg2, $arg3)
>exit()
>}
> 
> The only caveat is that you must pass the "-g" argument to stap.
> 
> Also, for some reason the printf in the probe always prints zeros, no matter
> what the actual event receives (I've debugged QEMU down to the call to the
> auto-generated stap functions). Could this be an error in systemtap?

It's strange that arguments do not have valid values.  Debugging the
stap functions is the next step if you want to figure out what happened.
I've never had this issue before so maybe something with Debian
SystemTap userspace probes is broken.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
>> Stefan Hajnoczi writes:
>> 
>> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events 
>> >> (I'm
>> >> only able to see the host kernel events). If someone with more experience 
>> >> on it
>> >> can help me use dtrace with QEMU's events, I'll also add the supporting 
>> >> library
>> >> to let dtrace do the callout to QEMU's moitor interface and control the 
>> >> events,
>> >> and add a prperly useful example of that on the hypertrace docs (which 
>> >> was my
>> >> original intention).
>> 
>> > Which "dtrace" and host OS are you using?
>> 
>> > QEMU builds with static user-space probes.  You need to tell DTrace or
>> > SystemTap to enable those probes in order to record trace data.
>> 
>> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
>> 
>> I just gave it another try, and works if I use probes like:
>> 
>> process("").mark("")
>> 
>> although they don't seem to appear on "stap -l" or anything like that (I 
>> cannot
>> find a "qemu" provider). But I'm still unable to print the event values. 
>> This:

> The following enumerates events for me:

> $ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
> process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long 
> $arg2:long
> process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
> ...

> You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
> I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
> 3.0-3.fc24.

I meant I cannot just use "qemu" as a provider name. Later I discovered that
"process()" does work.


>> Also, I'm still not sure how to interact with QEMU's monitor interface from
>> within the probe code (probes execute in kernel mode, including "guru mode"
>> code).

> When SystemTap is used the QEMU monitor interface does nothing.

That's not what I've experienced. I was able to use a stap script to change the
tracing state of events:

   #!/usr/bin/env stap

   %{
   #include 
   %}

   function event:long(cpu:long, addr:long, info:long)
   %{
   char *argv[4] = {"/bin/sh", "-c", "echo 'trace-event * off' | telnet 
localhost 1234", NULL};
   call_usermodehelper(argv[0], argv, NULL, UMH_WAIT_EXEC);
   STAP_RETURN(0);
   %}

   probe begin {
   printf("hello\n")
   }
   probe 
process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
   {
   printf("%x %d %d\n", $arg1, $arg2, $arg3)
   event($arg1, $arg2, $arg3)
   exit()
   }

The only caveat is that you must pass the "-g" argument to stap.

Also, for some reason the printf in the probe always prints zeros, no matter
what the actual event receives (I've debugged QEMU down to the call to the
auto-generated stap functions). Could this be an error in systemtap?


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Wed, Aug 24, 2016 at 12:25:54PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> >> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events 
> >> (I'm
> >> only able to see the host kernel events). If someone with more experience 
> >> on it
> >> can help me use dtrace with QEMU's events, I'll also add the supporting 
> >> library
> >> to let dtrace do the callout to QEMU's moitor interface and control the 
> >> events,
> >> and add a prperly useful example of that on the hypertrace docs (which was 
> >> my
> >> original intention).
> 
> > Which "dtrace" and host OS are you using?
> 
> > QEMU builds with static user-space probes.  You need to tell DTrace or
> > SystemTap to enable those probes in order to record trace data.
> 
> I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.
> 
> I just gave it another try, and works if I use probes like:
> 
>   process("").mark("")
> 
> although they don't seem to appear on "stap -l" or anything like that (I 
> cannot
> find a "qemu" provider). But I'm still unable to print the event values. This:

The following enumerates events for me:

$ sudo stap -L 'process("/usr/bin/qemu-system-x86_64").mark("*")'
process("/usr/bin/qemu-system-x86_64").mark("alsa_no_frames") $arg1:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_pollout") $arg1:long 
$arg2:long
process("/usr/bin/qemu-system-x86_64").mark("alsa_read_zero") $arg1:long
...

You can also use /usr/share/systemtap/tapset/qemu-system-x86_64.stp.
I'm running Fedora 24 with qemu-system-x86 2.6.0-5.fc24 and SystemTap
3.0-3.fc24.

> Also, I'm still not sure how to interact with QEMU's monitor interface from
> within the probe code (probes execute in kernel mode, including "guru mode"
> code).

When SystemTap is used the QEMU monitor interface does nothing.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
>> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
>> only able to see the host kernel events). If someone with more experience on 
>> it
>> can help me use dtrace with QEMU's events, I'll also add the supporting 
>> library
>> to let dtrace do the callout to QEMU's moitor interface and control the 
>> events,
>> and add a prperly useful example of that on the hypertrace docs (which was my
>> original intention).

> Which "dtrace" and host OS are you using?

> QEMU builds with static user-space probes.  You need to tell DTrace or
> SystemTap to enable those probes in order to record trace data.

I'm using debian on a 4.6.0-1-amd64 kernel with systemtap 3.0.6.

I just gave it another try, and works if I use probes like:

  process("").mark("")

although they don't seem to appear on "stap -l" or anything like that (I cannot
find a "qemu" provider). But I'm still unable to print the event values. This:

  probe 
process("./install/vanilla/bin/qemu-system-i386").mark("guest_mem_before_exec")
  {
  printf("%p %lx %d\n", $arg1, $arg2, $arg3)
  }

always prints "0x0 0x0 0", which is clearly wrong (other backend on the same
build print the correct values).

Also, I'm still not sure how to interact with QEMU's monitor interface from
within the probe code (probes execute in kernel mode, including "guru mode"
code).

If anybody can shed some like into any of this, I'd appreaciate it.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Sun, Aug 21, 2016 at 02:32:34PM +0200, Lluís Vilanova wrote:
> Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
> only able to see the host kernel events). If someone with more experience on 
> it
> can help me use dtrace with QEMU's events, I'll also add the supporting 
> library
> to let dtrace do the callout to QEMU's moitor interface and control the 
> events,
> and add a prperly useful example of that on the hypertrace docs (which was my
> original intention).

Which "dtrace" and host OS are you using?

QEMU builds with static user-space probes.  You need to tell DTrace or
SystemTap to enable those probes in order to record trace data.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) 
>> using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 
>> 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to 
>> easily
>> synchronize or correlate them. You could also modify you guest's tracing 
>> system
>> to emit all events through the hypertrace channel, providing a unified and 
>> fully
>> synchronized trace log. Another use case is timing the performance of guest 
>> code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.

> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.

> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.

> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.

> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.

> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.

As I said, I wanted to implement a simple way to provide common reference points
in guest and host traces. For that, a *synchronous* guest-to-host channel is the
simplest way to do it (otherwise you're back to square one).

Another reason to have this is that I wanted to attach a dtrace/systemtap hook
to the hypertrace event, and from there dynamically control QEMU's tracing. Then
you can insert simple calls to hypertrace on your guest code to tell your dtrace
hooks when to enable/disable tracing of certain QEMU events (like guest instr
traces).

Unfortuntely, I've been unable to to make dtrace recognise QEMU's events (I'm
only able to see the host kernel events). If someone with more experience on it
can help me use dtrace with QEMU's events, I'll also add the supporting library
to let dtrace do the callout to QEMU's moitor interface and control the events,
and add a prperly useful example of that on the hypertrace docs (which was my
original intention).


Thanks,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Luiz Capitulino writes:

> On Thu, 18 Aug 2016 14:53:27 +0100
> Stefan Hajnoczi  wrote:

>> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
>> > Stefan Hajnoczi writes:
>> >   
>> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
>> > >> The hypertrace channel allows guest code to emit events in QEMU (the 
>> > >> host) using
>> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 
>> > >> 'system'
>> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls 
>> > >> are to
>> > >> system calls.
>> > >> 
>> > >> You can use this to emit an event on both guest and QEMU (host) traces 
>> > >> to easily
>> > >> synchronize or correlate them. You could also modify you guest's 
>> > >> tracing system
>> > >> to emit all events through the hypertrace channel, providing a unified 
>> > >> and fully
>> > >> synchronized trace log. Another use case is timing the performance of 
>> > >> guest code
>> > >> when optimizing TCG (QEMU traces have a timestamp).
>> > >> 
>> > >> See first commit for a full description.
>> > >> 
>> > >> Signed-off-by: Lluís Vilanova 
>> > >> ---  
>> >   
>> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
>> > > who have all looked into host/guest tracing solutions.  
>> > [...]
>> > 
>> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
>> 
>> They are working on or have worked on different approaches to host/guest
>> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
>> I know.

> The ftrace solution is documented here:

>  https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

> This traces the guest and host kernels. It supports merging the guest
> and host traces. It's extremely low latency and has helped us to
> find several spikes for real-time KVM (we're talking a few to
> a dozen microseconds at most).

> Now, our stack actually is:

>  - Guest app
>  - Guest kernel
>  - Host kernel
>  - QEMU

> QEMU already has its own tracing (which I don't know how it works).
> If I had to trace the guest app, I'd certainly start off by using
> LTTng. Although, we'd have to write a tool to merge and orchestrate
> (wooo, cloud buzzword!) all those traces (if that's what one wants).
[...]

One of my targets was to simplify the merge by providing known reference points
between guest and host traces.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Fri, 19 Aug 2016 11:02:07 +0100
Stefan Hajnoczi  wrote:


> > [1] 
> > https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no=i:100;=yes=no
> >   
> 
> Cool, that's very interesting!  I'll try to attend.

Good, see you then!

> 
> I guess LinuxCon will post a video of your talk and the slides will be
> available after the talk so people not at LinuxCon will be able to check
> it out too.

Hmm, IIRC the LinuxCon talks are seldom recorded. But the slides will be
available.

-- Steve

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Thu, Aug 18, 2016 at 12:19:50PM -0400, Steven Rostedt wrote:
> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi  wrote:
> 
> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.
> 
> Yes, and I'm currently working on it again. In fact, if any of you will
> be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
> and what I plan to complete in the near future.
> 
> -- Steve
> 
> [1] 
> https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no=i:100;=yes=no

Cool, that's very interesting!  I'll try to attend.

I guess LinuxCon will post a video of your talk and the slides will be
available after the talk so people not at LinuxCon will be able to check
it out too.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Masami Hiramatsu]

On Thu, 18 Aug 2016 09:37:11 -0400
Luiz Capitulino  wrote:

> On Thu, 18 Aug 2016 11:54:24 +0100
> Stefan Hajnoczi  wrote:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > > The hypertrace channel allows guest code to emit events in QEMU (the 
> > > host) using
> > > its tracing infrastructure (see "docs/trace.txt"). This works in both 
> > > 'system'
> > > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are 
> > > to
> > > system calls.
> > > 
> > > You can use this to emit an event on both guest and QEMU (host) traces to 
> > > easily
> > > synchronize or correlate them. You could also modify you guest's tracing 
> > > system
> > > to emit all events through the hypertrace channel, providing a unified 
> > > and fully
> > > synchronized trace log. Another use case is timing the performance of 
> > > guest code
> > > when optimizing TCG (QEMU traces have a timestamp).
> > > 
> > > See first commit for a full description.  
> 
> I hate the be the one asking the stupid questions, but what's
> the problem this series solves? What are the use-cases? Why
> existing solutions don't solve this problem? How does this
> compares to existing solutions?
> 
> > This tracing approach has a high performance overhead, particularly for
> > SMP guests where each trace event requires writing to the global control
> > register.  All CPUs will be hammering this register (heavyweight vmexit)
> > for each trace event.

That reminds me xenprobes.
https://www.usenix.org/legacy/event/usenix07/tech/full_papers/quynh/quynh.pdf

The paper also tried to implement kprobe-like feature for tracing events in
guest. Synchronous tracing maybe easy to read the raw trace data, but cause
big slowdown. Moreover, we can merge and synchronize the data easily after
traced.

> > 
> > I think the folks CCed on this email all take an asynchronous approach
> > to avoid this performance overhead.  Synchronous means taking a VM exit
> > for every event.  Asynchronous means writing trace data to a buffer and
> > later interleaving guest data with host trace data.
> > 
> > LTTng Userspace Tracer is an example of the asynchronous approach.  The
> > trace data buffers are in shared memory.  The LTTng process can grab
> > buffers at appropriate times.
> > 
> > The ftrace virtio-serial approach has been to splice() the ftrace
> > buffers, resulting in efficient I/O.
> 
> Yes. However, note that the virtio-serial device is only used to
> transfer the tracing data to the host. It has no role in the
> tracing process. Also, it's not required. I've been using the
> networking and it works OK as long as your tracing data is not big.

Right, the no-copy virtio-serial is for reducing the overhead as small as
possible but it also requires a special programs in both of guest and host.
(since it uses unix domain socket to transfer data without copy)
In normal case, you can just trace it as like as tracing on a remote machine
via network. That's more scalable and flexible. 


> > Steven is working on a host/guest solution for trace-cmd.  It is also
> > asynchronous.  No new paravirt hardware is needed and it makes me wonder
> > whether the hypertrace PCI device is trying to solve the problem at the
> > wrong layer.

Thanks Steven to continue that!

Regards,

> > 
> > If you want to play around with asynchronous tracing, you could start
> > with trace/simple.c.  It has a trace buffer that is asynchronously
> > written out to file by a dedicated "writer" thread.
> > 
> > The one case where hypertrace makes sense to me is for -user tracing.
> > There QEMU can efficiently interleave guest and QEMU traces, although as
> > mentioned in the patch, I don't think the SIGSEGV approach should be
> > used.
> > 
> > I suggest stripping this series down to focus on -user.  Synchronous
> > tracing is not a good approach for -system emulation.
> > 
> > Stefan
> 


-- 
Masami Hiramatsu 

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Steven Rostedt]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi  wrote:

> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.

Yes, and I'm currently working on it again. In fact, if any of you will
be at LinuxCon in Toronto next week, I'll be presenting[1] what I have
and what I plan to complete in the near future.

-- Steve

[1] 
https://lcccna2016.sched.org/event/7JWL/trace-cmd-virt-server-a-status-update-steven-rostedt-red-hat?iframe=no=i:100;=yes=no

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 14:53:27 +0100
Stefan Hajnoczi  wrote:

> On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> > Stefan Hajnoczi writes:
> >   
> > > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:  
> > >> The hypertrace channel allows guest code to emit events in QEMU (the 
> > >> host) using
> > >> its tracing infrastructure (see "docs/trace.txt"). This works in both 
> > >> 'system'
> > >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are 
> > >> to
> > >> system calls.
> > >> 
> > >> You can use this to emit an event on both guest and QEMU (host) traces 
> > >> to easily
> > >> synchronize or correlate them. You could also modify you guest's tracing 
> > >> system
> > >> to emit all events through the hypertrace channel, providing a unified 
> > >> and fully
> > >> synchronized trace log. Another use case is timing the performance of 
> > >> guest code
> > >> when optimizing TCG (QEMU traces have a timestamp).
> > >> 
> > >> See first commit for a full description.
> > >> 
> > >> Signed-off-by: Lluís Vilanova 
> > >> ---  
> >   
> > > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > > who have all looked into host/guest tracing solutions.  
> > [...]
> > 
> > Oh, I wasn't aware of that. I'm certainly interested in collaborating.  
> 
> They are working on or have worked on different approaches to host/guest
> tracing.  Unfortunately there isn't an out-of-the-box solution as far as
> I know.

The ftrace solution is documented here:

 https://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg00887.html

This traces the guest and host kernels. It supports merging the guest
and host traces. It's extremely low latency and has helped us to
find several spikes for real-time KVM (we're talking a few to
a dozen microseconds at most).

Now, our stack actually is:

 - Guest app
 - Guest kernel
 - Host kernel
 - QEMU

QEMU already has its own tracing (which I don't know how it works).
If I had to trace the guest app, I'd certainly start off by using
LTTng. Although, we'd have to write a tool to merge and orchestrate
(wooo, cloud buzzword!) all those traces (if that's what one wants).

> It would be nice if there was a documented host/guest tracing approach
> that didn't involve much manual setup and handled most use cases.

I'd volunteer to do it, although it will take me weeks to do it.

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Thu, Aug 18, 2016 at 12:22:18PM +0200, Lluís Vilanova wrote:
> Stefan Hajnoczi writes:
> 
> > On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> >> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> >> using
> >> its tracing infrastructure (see "docs/trace.txt"). This works in both 
> >> 'system'
> >> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> >> system calls.
> >> 
> >> You can use this to emit an event on both guest and QEMU (host) traces to 
> >> easily
> >> synchronize or correlate them. You could also modify you guest's tracing 
> >> system
> >> to emit all events through the hypertrace channel, providing a unified and 
> >> fully
> >> synchronized trace log. Another use case is timing the performance of 
> >> guest code
> >> when optimizing TCG (QEMU traces have a timestamp).
> >> 
> >> See first commit for a full description.
> >> 
> >> Signed-off-by: Lluís Vilanova 
> >> ---
> 
> > CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> > who have all looked into host/guest tracing solutions.
> [...]
> 
> Oh, I wasn't aware of that. I'm certainly interested in collaborating.

They are working on or have worked on different approaches to host/guest
tracing.  Unfortunately there isn't an out-of-the-box solution as far as
I know.

It would be nice if there was a documented host/guest tracing approach
that didn't involve much manual setup and handled most use cases.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Luiz Capitulino]

On Thu, 18 Aug 2016 11:54:24 +0100
Stefan Hajnoczi  wrote:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> > The hypertrace channel allows guest code to emit events in QEMU (the host) 
> > using
> > its tracing infrastructure (see "docs/trace.txt"). This works in both 
> > 'system'
> > and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> > system calls.
> > 
> > You can use this to emit an event on both guest and QEMU (host) traces to 
> > easily
> > synchronize or correlate them. You could also modify you guest's tracing 
> > system
> > to emit all events through the hypertrace channel, providing a unified and 
> > fully
> > synchronized trace log. Another use case is timing the performance of guest 
> > code
> > when optimizing TCG (QEMU traces have a timestamp).
> > 
> > See first commit for a full description.  

I hate the be the one asking the stupid questions, but what's
the problem this series solves? What are the use-cases? Why
existing solutions don't solve this problem? How does this
compares to existing solutions?

> This tracing approach has a high performance overhead, particularly for
> SMP guests where each trace event requires writing to the global control
> register.  All CPUs will be hammering this register (heavyweight vmexit)
> for each trace event.
> 
> I think the folks CCed on this email all take an asynchronous approach
> to avoid this performance overhead.  Synchronous means taking a VM exit
> for every event.  Asynchronous means writing trace data to a buffer and
> later interleaving guest data with host trace data.
> 
> LTTng Userspace Tracer is an example of the asynchronous approach.  The
> trace data buffers are in shared memory.  The LTTng process can grab
> buffers at appropriate times.
> 
> The ftrace virtio-serial approach has been to splice() the ftrace
> buffers, resulting in efficient I/O.

Yes. However, note that the virtio-serial device is only used to
transfer the tracing data to the host. It has no role in the
tracing process. Also, it's not required. I've been using the
networking and it works OK as long as your tracing data is not big.

> 
> Steven is working on a host/guest solution for trace-cmd.  It is also
> asynchronous.  No new paravirt hardware is needed and it makes me wonder
> whether the hypertrace PCI device is trying to solve the problem at the
> wrong layer.
> 
> If you want to play around with asynchronous tracing, you could start
> with trace/simple.c.  It has a trace buffer that is asynchronously
> written out to file by a dedicated "writer" thread.
> 
> The one case where hypertrace makes sense to me is for -user tracing.
> There QEMU can efficiently interleave guest and QEMU traces, although as
> mentioned in the patch, I don't think the SIGSEGV approach should be
> used.
> 
> I suggest stripping this series down to focus on -user.  Synchronous
> tracing is not a good approach for -system emulation.
> 
> Stefan

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to 
> easily
> synchronize or correlate them. You could also modify you guest's tracing 
> system
> to emit all events through the hypertrace channel, providing a unified and 
> fully
> synchronized trace log. Another use case is timing the performance of guest 
> code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.

This tracing approach has a high performance overhead, particularly for
SMP guests where each trace event requires writing to the global control
register.  All CPUs will be hammering this register (heavyweight vmexit)
for each trace event.

I think the folks CCed on this email all take an asynchronous approach
to avoid this performance overhead.  Synchronous means taking a VM exit
for every event.  Asynchronous means writing trace data to a buffer and
later interleaving guest data with host trace data.

LTTng Userspace Tracer is an example of the asynchronous approach.  The
trace data buffers are in shared memory.  The LTTng process can grab
buffers at appropriate times.

The ftrace virtio-serial approach has been to splice() the ftrace
buffers, resulting in efficient I/O.

Steven is working on a host/guest solution for trace-cmd.  It is also
asynchronous.  No new paravirt hardware is needed and it makes me wonder
whether the hypertrace PCI device is trying to solve the problem at the
wrong layer.

If you want to play around with asynchronous tracing, you could start
with trace/simple.c.  It has a trace buffer that is asynchronously
written out to file by a dedicated "writer" thread.

The one case where hypertrace makes sense to me is for -user tracing.
There QEMU can efficiently interleave guest and QEMU traces, although as
mentioned in the patch, I don't think the SIGSEGV approach should be
used.

I suggest stripping this series down to focus on -user.  Synchronous
tracing is not a good approach for -system emulation.

Stefan


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Lluís Vilanova]

Stefan Hajnoczi writes:

> On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
>> The hypertrace channel allows guest code to emit events in QEMU (the host) 
>> using
>> its tracing infrastructure (see "docs/trace.txt"). This works in both 
>> 'system'
>> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
>> system calls.
>> 
>> You can use this to emit an event on both guest and QEMU (host) traces to 
>> easily
>> synchronize or correlate them. You could also modify you guest's tracing 
>> system
>> to emit all events through the hypertrace channel, providing a unified and 
>> fully
>> synchronized trace log. Another use case is timing the performance of guest 
>> code
>> when optimizing TCG (QEMU traces have a timestamp).
>> 
>> See first commit for a full description.
>> 
>> Signed-off-by: Lluís Vilanova 
>> ---

> CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
> who have all looked into host/guest tracing solutions.
[...]

Oh, I wasn't aware of that. I'm certainly interested in collaborating.


Cheers,
  Lluis

Re: [Qemu-devel] [PATCH 0/6] hypertrace: Lightweight guest-to-QEMU trace channel

[Stefan Hajnoczi]

On Fri, Aug 05, 2016 at 06:59:23PM +0200, Lluís Vilanova wrote:
> The hypertrace channel allows guest code to emit events in QEMU (the host) 
> using
> its tracing infrastructure (see "docs/trace.txt"). This works in both 'system'
> and 'user' modes. That is, hypertrace is to tracing, what hypercalls are to
> system calls.
> 
> You can use this to emit an event on both guest and QEMU (host) traces to 
> easily
> synchronize or correlate them. You could also modify you guest's tracing 
> system
> to emit all events through the hypertrace channel, providing a unified and 
> fully
> synchronized trace log. Another use case is timing the performance of guest 
> code
> when optimizing TCG (QEMU traces have a timestamp).
> 
> See first commit for a full description.
> 
> Signed-off-by: Lluís Vilanova 
> ---

CCing Steven Rostedt, Masami Hiramatsu, Luiz Capitulino, and LTTng folks
who have all looked into host/guest tracing solutions.

Here is a mailing list archive link to this email thread:
https://lists.gnu.org/archive/html/qemu-devel/2016-08/msg01081.html

> 
> Lluís Vilanova (6):
>   hypertrace: Add documentation
>   hypertrace: Add tracing event "guest_hypertrace"
>   hypertrace: [*-user] Add QEMU-side proxy to "guest_hypertrace" event
>   hypertrace: [softmmu] Add QEMU-side proxy to "guest_hypertrace" event
>   hypertrace: Add guest-side user-level library
>   hypertrace: Add guest-side Linux module
> 
> 
>  Makefile   |8 -
>  Makefile.objs  |6 
>  bsd-user/main.c|   16 +
>  bsd-user/mmap.c|2 
>  bsd-user/syscall.c |4 
>  configure  |   42 +++
>  docs/hypertrace.txt|  141 ++
>  docs/tracing.txt   |3 
>  hypertrace/Makefile.objs   |   20 +
>  hypertrace/guest/linux-module/Kbuild.in|5 
>  hypertrace/guest/linux-module/Makefile |   24 ++
>  .../include/linux/qemu-hypertrace-internal.h   |   46 +++
>  .../linux-module/include/linux/qemu-hypertrace.h   |   71 +
>  hypertrace/guest/linux-module/qemu-hypertrace.c|  133 +
>  hypertrace/guest/user/Makefile |   28 ++
>  hypertrace/guest/user/common.c |  221 +++
>  hypertrace/guest/user/qemu-hypertrace.h|   77 +
>  hypertrace/softmmu.c   |  243 +
>  hypertrace/user.c  |  292 
> 
>  hypertrace/user.h  |   52 
>  include/hw/pci/pci.h   |2 
>  linux-user/main.c  |   19 +
>  linux-user/mmap.c  |2 
>  linux-user/syscall.c   |3 
>  trace/Makefile.objs|2 
>  25 files changed, 1460 insertions(+), 2 deletions(-)
>  create mode 100644 docs/hypertrace.txt
>  create mode 100644 hypertrace/Makefile.objs
>  create mode 100644 hypertrace/guest/linux-module/Kbuild.in
>  create mode 100644 hypertrace/guest/linux-module/Makefile
>  create mode 100644 
> hypertrace/guest/linux-module/include/linux/qemu-hypertrace-internal.h
>  create mode 100644 
> hypertrace/guest/linux-module/include/linux/qemu-hypertrace.h
>  create mode 100644 hypertrace/guest/linux-module/qemu-hypertrace.c
>  create mode 100644 hypertrace/guest/user/Makefile
>  create mode 100644 hypertrace/guest/user/common.c
>  create mode 100644 hypertrace/guest/user/qemu-hypertrace.h
>  create mode 100644 hypertrace/softmmu.c
>  create mode 100644 hypertrace/user.c
>  create mode 100644 hypertrace/user.h
> 
> 
> To: qemu-devel@nongnu.org
> Cc: Stefan Hajnoczi 
> 


signature.asc
Description: PGP signature