Re: [Qemu-devel] [PATCH v4 00/21] Extended I/O accounting
On Wed, Oct 28, 2015 at 05:32:57PM +0200, Alberto Garcia wrote: > Here's v4 of the series that implements extended I/O accounting for > block devices. > > Since part of Max's BlockBackend series has already been merged, this > series can now be applied cleanly on top of the master branch without > additional dependencies. > > Here's the summary of what this series provides: > > - New block_acct_failed() and block_acct_invalid() calls. >We keep track now of the number of successful, failed and invalid >operations (each one separated into read, write and flush). So from >the API point of view, BlockDeviceStats contains 6 new fields for >those. > > - idle_time_ns: time since the last I/O operation. > > - New BlockDeviceTimedStats struct: it has statistics for the I/O >during a given interval of time. It keeps minimum, maximum and >average latencies for read, write and flush operations. > >It also keeps the average read and write queue depths. > > - New 'stats-intervals' option that allows the user to define the >intervals used to keep the aforementioned statistics. An arbitrary >number of intervals can be specified, the length of each one is in >seconds. > >For the API I opted for a colon-separated list of numbers, > > stats-intervals=60:3600:86400 > >I also considered something a different syntax, > > stats-intervals.0.length=60, > stats-intervals.1.length=3600, > stats-intervals.2.length=86400 > >This one could be useful if we want to specify any other attribute >for each interval, but I couldn't come up with any, so I chose the >simpler solution. > > - Two new options, stats-account-invalid and stats-account-failed, >which allow the user to decide whether to count invalid and failed >operations when computing the idle time and total latency. > > Regards, > > Berto > > v4: > - Rebase on top of the current master. This series no longer depends > on any other. > - patch 8: clarify that interval_length is in seconds [Stefan] > - patch 9: rewrite timed_average_sum() so it does not call > qemu_clock_get_ns() twice [Stefan] > > v3: https://lists.gnu.org/archive/html/qemu-block/2015-10/msg00785.html > - Rebased on top of the current master and on Max's BlockBackend > series v7 > - patch 4: minor documentation fixes [Stefan] > - patch 5: s/miliseconds/nanoseconds/ [Stefan] > - patch 6: dropped, there's no "supports_stats" anymore [Stefan] > - patch 7 (now 6): explain why block_acct_invalid() does not update > total_time_ns[] [Stefan] > - patch 12 (now 11): don't initialize BlockAcctCookie to { 0 }, it's >not needed. > > v2: https://lists.gnu.org/archive/html/qemu-block/2015-10/msg00161.html > - First complete implementation of the new statistics > > v1: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg03321.html > - Initial series containing only the timed average infrastructure. > > Alberto Garcia (21): > xen_disk: Account for flush operations > ide: Account for write operations correctly > block: define 'clock_type' for the accounting code > util: Infrastructure for computing recent averages > block: Add idle_time_ns to BlockDeviceStats > block: Add statistics for failed and invalid I/O operations > block: Allow configuring whether to account failed and invalid ops > block: Compute minimum, maximum and average I/O latencies > block: Add average I/O queue depth to BlockDeviceTimedStats > block: New option to define the intervals for collecting I/O > statistics > qemu-io: Account for failed, invalid and flush operations > block: Use QEMU_CLOCK_VIRTUAL for the accounting code in qtest mode > iotests: Add test for the block device statistics > nvme: Account for failed and invalid operations > virtio-blk: Account for failed and invalid operations > xen_disk: Account for failed and invalid operations > atapi: Account for failed and invalid operations > ide: Account for failed and invalid operations > macio: Account for failed operations > scsi-disk: Account for failed operations > block: Update copyright of the accounting code > > block/accounting.c | 123 ++- > block/block-backend.c| 1 + > block/qapi.c | 51 +++ > blockdev.c | 53 +++ > hmp.c| 4 +- > hw/block/nvme.c | 11 +- > hw/block/virtio-blk.c| 4 +- > hw/block/xen_disk.c | 27 +++- > hw/ide/atapi.c | 31 ++-- > hw/ide/core.c| 12 +- > hw/ide/macio.c | 12 +- > hw/scsi/scsi-disk.c | 46 -- > include/block/accounting.h | 28 > include/qemu/timed-average.h | 64 > qapi/block-core.json | 103 - > qemu-io-cmds.c | 9 ++ > qmp-commands.hx | 80 +- > tests/Makefile | 4 + >
[Qemu-devel] [PATCH v4 00/21] Extended I/O accounting
Here's v4 of the series that implements extended I/O accounting for block devices. Since part of Max's BlockBackend series has already been merged, this series can now be applied cleanly on top of the master branch without additional dependencies. Here's the summary of what this series provides: - New block_acct_failed() and block_acct_invalid() calls. We keep track now of the number of successful, failed and invalid operations (each one separated into read, write and flush). So from the API point of view, BlockDeviceStats contains 6 new fields for those. - idle_time_ns: time since the last I/O operation. - New BlockDeviceTimedStats struct: it has statistics for the I/O during a given interval of time. It keeps minimum, maximum and average latencies for read, write and flush operations. It also keeps the average read and write queue depths. - New 'stats-intervals' option that allows the user to define the intervals used to keep the aforementioned statistics. An arbitrary number of intervals can be specified, the length of each one is in seconds. For the API I opted for a colon-separated list of numbers, stats-intervals=60:3600:86400 I also considered something a different syntax, stats-intervals.0.length=60, stats-intervals.1.length=3600, stats-intervals.2.length=86400 This one could be useful if we want to specify any other attribute for each interval, but I couldn't come up with any, so I chose the simpler solution. - Two new options, stats-account-invalid and stats-account-failed, which allow the user to decide whether to count invalid and failed operations when computing the idle time and total latency. Regards, Berto v4: - Rebase on top of the current master. This series no longer depends on any other. - patch 8: clarify that interval_length is in seconds [Stefan] - patch 9: rewrite timed_average_sum() so it does not call qemu_clock_get_ns() twice [Stefan] v3: https://lists.gnu.org/archive/html/qemu-block/2015-10/msg00785.html - Rebased on top of the current master and on Max's BlockBackend series v7 - patch 4: minor documentation fixes [Stefan] - patch 5: s/miliseconds/nanoseconds/ [Stefan] - patch 6: dropped, there's no "supports_stats" anymore [Stefan] - patch 7 (now 6): explain why block_acct_invalid() does not update total_time_ns[] [Stefan] - patch 12 (now 11): don't initialize BlockAcctCookie to { 0 }, it's not needed. v2: https://lists.gnu.org/archive/html/qemu-block/2015-10/msg00161.html - First complete implementation of the new statistics v1: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg03321.html - Initial series containing only the timed average infrastructure. Alberto Garcia (21): xen_disk: Account for flush operations ide: Account for write operations correctly block: define 'clock_type' for the accounting code util: Infrastructure for computing recent averages block: Add idle_time_ns to BlockDeviceStats block: Add statistics for failed and invalid I/O operations block: Allow configuring whether to account failed and invalid ops block: Compute minimum, maximum and average I/O latencies block: Add average I/O queue depth to BlockDeviceTimedStats block: New option to define the intervals for collecting I/O statistics qemu-io: Account for failed, invalid and flush operations block: Use QEMU_CLOCK_VIRTUAL for the accounting code in qtest mode iotests: Add test for the block device statistics nvme: Account for failed and invalid operations virtio-blk: Account for failed and invalid operations xen_disk: Account for failed and invalid operations atapi: Account for failed and invalid operations ide: Account for failed and invalid operations macio: Account for failed operations scsi-disk: Account for failed operations block: Update copyright of the accounting code block/accounting.c | 123 ++- block/block-backend.c| 1 + block/qapi.c | 51 +++ blockdev.c | 53 +++ hmp.c| 4 +- hw/block/nvme.c | 11 +- hw/block/virtio-blk.c| 4 +- hw/block/xen_disk.c | 27 +++- hw/ide/atapi.c | 31 ++-- hw/ide/core.c| 12 +- hw/ide/macio.c | 12 +- hw/scsi/scsi-disk.c | 46 -- include/block/accounting.h | 28 include/qemu/timed-average.h | 64 qapi/block-core.json | 103 - qemu-io-cmds.c | 9 ++ qmp-commands.hx | 80 +- tests/Makefile | 4 + tests/qemu-iotests/136 | 349 +++ tests/qemu-iotests/136.out | 5 + tests/qemu-iotests/group | 1 + tests/test-timed-average.c | 90 +++ util/Makefile.objs | 1 + util/timed-average.c | 231 24