Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)
On Thu, Oct 21, 2010 at 07:20:27AM +0200, Christoph Hellwig wrote: On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote: On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote: Once we have -trace events=..., defining the list of active tracepoints before starting qemu will be trivial (e.g. via a config file). Of course, this requires that all tracepoints are built-in... Sorry that I've not been following this very closely, but does this sort of thing allow tracing reads and writes to block devices? Am I right in thinking that if a tracepoint existed in the right place, one could get a log file from that which could be post-processed in another tool? I don't think it does. But I have a patch that I need to update which outputs traces in the form of the Linux blktrace block I/O tracing tool, which does exactly what you need (and more). You can simply use all the blktrace-enabled tools, including seekwatcher on it. It's on the to-do list to use blktrace instead, so a patch which changed qemu to produce blktrace-like output would be useful. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)
On Thu, Oct 21, 2010 at 6:20 AM, Christoph Hellwig h...@lst.de wrote: On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote: On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote: Once we have -trace events=..., defining the list of active tracepoints before starting qemu will be trivial (e.g. via a config file). Of course, this requires that all tracepoints are built-in... Sorry that I've not been following this very closely, but does this sort of thing allow tracing reads and writes to block devices? Am I right in thinking that if a tracepoint existed in the right place, one could get a log file from that which could be post-processed in another tool? I don't think it does. But I have a patch that I need to update which outputs traces in the form of the Linux blktrace block I/O tracing tool, which does exactly what you need (and more). You can simply use all the blktrace-enabled tools, including seekwatcher on it. There are trace events in qemu.git today that let you trace block read/write operations (device, sector, length, r/w information is there). Look at bdrv_aio_readv and bdrv_aio_writev in the trace-events file. I can see the value of using existing tools to parse blktrace data. On the other hand blktrace is taylored to Linux's block layer and QEMU can only map to a subset of it. I'm interested in your patch too, is it a new block protocol or does it hook directly into block.c? Stefan
Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)
On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote: On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote: Once we have -trace events=..., defining the list of active tracepoints before starting qemu will be trivial (e.g. via a config file). Of course, this requires that all tracepoints are built-in... Sorry that I've not been following this very closely, but does this sort of thing allow tracing reads and writes to block devices? Am I right in thinking that if a tracepoint existed in the right place, one could get a log file from that which could be post-processed in another tool? cf: http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content While having a static tracepoint in the right place would be best, it is not strictly neccessary with a tool like DTrace/SystemTAP. With the qemu debuginfo available, those tools can dynamically insert a probe into any QEMU function at any point in the code. So you could easily replace your QEMU patch from that blog post with a simple trace script and get the same info dynamically. The benefit of static markers is that they can provide standard named probe point + args, which are stable long term, even as the code is re-factored/renamed/moved, etc. Regards, Daniel -- |: Red Hat, Engineering, London-o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :| |: http://autobuild.org-o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)
On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote: On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote: Once we have -trace events=..., defining the list of active tracepoints before starting qemu will be trivial (e.g. via a config file). Of course, this requires that all tracepoints are built-in... Sorry that I've not been following this very closely, but does this sort of thing allow tracing reads and writes to block devices? Am I right in thinking that if a tracepoint existed in the right place, one could get a log file from that which could be post-processed in another tool? I don't think it does. But I have a patch that I need to update which outputs traces in the form of the Linux blktrace block I/O tracing tool, which does exactly what you need (and more). You can simply use all the blktrace-enabled tools, including seekwatcher on it.
Re: Tracing block devices (was: Re: [Qemu-devel] Static tracepoint control via trace-event)
On Tue, Oct 19, 2010 at 03:29:51PM +0100, Richard W.M. Jones wrote:
On Tue, Oct 19, 2010 at 03:59:36PM +0200, Jan Kiszka wrote:
Once we have -trace events=..., defining the list of active
tracepoints before starting qemu will be trivial (e.g. via a config
file). Of course, this requires that all tracepoints are built-in...
Sorry that I've not been following this very closely, but does this
sort of thing allow tracing reads and writes to block devices? Am I
right in thinking that if a tracepoint existed in the right place, one
could get a log file from that which could be post-processed in
another tool?
cf:
http://rwmj.wordpress.com/2010/10/05/visualizing-reads-writes-and-alignment/#content
Definitely, here is the commit that added bdrv_aio_writev/bdrv_aio_readv
tracing. bdrv_aio_multiwrite has been traced for a while.
http://patchwork.ozlabs.org/patch/66843/
As an example, I use the following script to find all write requests
that touch a given region. This is very useful for debugging image
corruptions given a trace file:
The usage is:
find_overlapping_io.py bs sector_num nb_sectors
where bs is the block driver state pointer, sector_num is the starting
sector address, and nb_sectors is the number of sectors.
#!/usr/bin/env python
import sys
def trace_filter(fobj, event, keys):
for line in fobj:
fields = line.strip().split()
if fields[0] != event:
continue
attrs = dict([(k, v) for k, v in (x.split('=') for x in fields[2:])])
match = True
for k, v in keys.iteritems():
if k not in attrs:
match = False
break
if attrs[k] != v:
match = False
break
if match:
yield attrs
def intersection(a_sector_num, a_nb_sectors, b_sector_num, b_nb_sectors):
return not (a_sector_num + a_nb_sectors = b_sector_num or \
b_sector_num + b_nb_sectors = a_sector_num)
bs, sector_num, nb_sectors = sys.argv[1:]
sector_num = int(sector_num, 0)
nb_sectors = int(nb_sectors, 0)
for req in trace_filter(sys.stdin, 'bdrv_aio_writev', {'bs': bs}):
if intersection(sector_num, nb_sectors, int(req['sector_num'], 0),
int(req['nb_sectors'], 0)):
print req
Stefan
