Asias He <as...@redhat.com> wrote on 11/20/2012 03:02:07 AM:
> From: Asias He <as...@redhat.com>
> To: Stefan Hajnoczi <stefa...@redhat.com>,
> Cc: qemu-devel@nongnu.org, Anthony Liguori/Austin/IBM@IBMUS, Paolo
> Bonzini <pbonz...@redhat.com>, Kevin Wolf <kw...@redhat.com>,
> "Michael S. Tsirkin" <m...@redhat.com>, Khoa Huynh/Austin/IBM@IBMUS
> Date: 11/20/2012 03:01 AM
> Subject: Re: [PATCH 0/7] virtio: virtio-blk data plane
>
> Hello Stefan,
>
> On 11/15/2012 11:18 PM, Stefan Hajnoczi wrote:
> > This series adds the -device virtio-blk-pci,x-data-plane=on property
that
> > enables a high performance I/O codepath. A dedicated thread is
> used to process
> > virtio-blk requests outside the global mutex and without going
> through the QEMU
> > block layer.
> >
> > Khoa Huynh <k...@us.ibm.com> reported an increase from 140,000
> IOPS to 600,000
> > IOPS for a single VM using virtio-blk-data-plane in July:
> >
> > http://comments.gmane.org/gmane.comp.emulators.kvm.devel/94580
> >
> > The virtio-blk-data-plane approach was originally presented at
> Linux Plumbers
> > Conference 2010. The following slides contain a brief overview:
> >
> > http://linuxplumbersconf.org/2010/ocw/system/presentations/651/
> original/Optimizing_the_QEMU_Storage_Stack.pdf
> >
> > The basic approach is:
> > 1. Each virtio-blk device has a thread dedicated to handling ioeventfd
> > signalling when the guest kicks the virtqueue.
> > 2. Requests are processed without going through the QEMU block layer
using
> > Linux AIO directly.
> > 3. Completion interrupts are injected via irqfd from the dedicated
thread.
> >
> > To try it out:
> >
> > qemu -drive
if=none,id=drive0,cache=none,aio=native,format=raw,file=...
> > -device virtio-blk-pci,drive=drive0,scsi=off,x-data-plane=on
>
>
> Is this the latest dataplane bits:
> (git://github.com/stefanha/qemu.git virtio-blk-data-plane)
>
> commit 7872075c24fa01c925d4f41faa9d04ce69bf5328
> Author: Stefan Hajnoczi <stefa...@redhat.com>
> Date: Wed Nov 14 15:45:38 2012 +0100
>
> virtio-blk: add x-data-plane=on|off performance feature
>
>
> With this commit on a ramdisk based box, I am seeing about 10K IOPS with
> x-data-plane on and 90K IOPS with x-data-plane off.
>
> Any ideas?
Hi Asias, I'll try this latest patchset next week and will report results
as soon as they are available. I am currently on vacation this week due
to the Thanksgivings holiday in the US.... (Previously, I was able to get
up to 1.33 million IOPS for a single KVM guest using Stefan's previous
code.)
Thanks,
-Khoa
>
> Command line I used:
>
> IMG=/dev/ram0
> x86_64-softmmu/qemu-system-x86_64 \
> -drive file=/root/img/sid.img,if=ide \
> -drive file=${IMG},if=none,cache=none,aio=native,id=disk1 -device
> virtio-blk-pci,x-data-plane=off,drive=disk1,scsi=off \
> -kernel $KERNEL -append "root=/dev/sdb1 console=tty0" \
> -L /tmp/qemu-dataplane/share/qemu/ -nographic -vnc :0 -enable-kvm -m
> 2048 -smp 4 -cpu qemu64,+x2apic -M pc
>
>
> >
> > Limitations:
> > * Only format=raw is supported
> > * Live migration is not supported
> > * Block jobs, hot unplug, and other operations fail with -EBUSY
> > * I/O throttling limits are ignored
> > * Only Linux hosts are supported due to Linux AIO usage
> >
> > The code has reached a stage where I feel it is ready to merge. Users
have
> > been playing with it for some time and want the significant
> performance boost.
> >
> > We are refactoring QEMU to get rid of the global mutex. I believe that
> > virtio-blk-data-plane can eventually become the default mode of
operation.
> >
> > Instead of waiting for global mutex removal efforts to finish, I want
to use
> > virtio-blk-data-plane as an example device for AioContext and threaded
hw
> > dispatch refactoring. This means:
> >
> > 1. When the block layer can bind to an AioContext and execute I/O
> outside the
> > global mutex, virtio-blk-data-plane can use this (and gain image
format
> > support).
> >
> > 2. When hw dispatch no longer needs the global mutex we can use
hw/virtio.c
> > again and perhaps run a pool of iothreads instead of dedicated data
plane
> > threads.
> >
> > But in the meantime, I have cleaned up the virtio-blk-data-plane
> code so that
> > it can be merged as an experimental feature.
> >
> > Changes from the RFC v9:
> > * Add x-data-plane=on|off option and coexist with regular virtio-blk
code
> > * Create thread from BH so it inherits iothread cpusets
> > * Drain requests on vm_stop() so stopped guest does not access image
file
> > * Add migration blocker
> > * Add bdrv_in_use() to prevent block jobs and other operations
> that can interfere
> > * Drop IOQueue request merging for simplicity
> > * Drop ioctl interrupt injection and always use irqfd for simplicity
> > * Major cleanup to split up source files
> > * Rebase from qemu-kvm.git onto qemu.git
> > * Address Michael Tsirkin's review comments
> >
> > Stefan Hajnoczi (7):
> > raw-posix: add raw_get_aio_fd() for virtio-blk-data-plane
> > configure: add CONFIG_VIRTIO_BLK_DATA_PLANE
> > dataplane: add virtqueue vring code
> > dataplane: add event loop
> > dataplane: add Linux AIO request queue
> > dataplane: add virtio-blk data plane code
> > virtio-blk: add x-data-plane=on|off performance feature
> >
> > block.h | 9 +
> > block/raw-posix.c | 34 ++++
> > configure | 21 +++
> > hw/Makefile.objs | 2 +-
> > hw/dataplane/Makefile.objs | 3 +
> > hw/dataplane/event-poll.c | 109 ++++++++++++
> > hw/dataplane/event-poll.h | 40 +++++
> > hw/dataplane/ioq.c | 118 +++++++++++++
> > hw/dataplane/ioq.h | 57 +++++++
> > hw/dataplane/virtio-blk.c | 414 ++++++++++++++++++++++++++++++++
> +++++++++++++
> > hw/dataplane/virtio-blk.h | 41 +++++
> > hw/dataplane/vring.c | 321 +++++++++++++++++++++++++++++++++++
> > hw/dataplane/vring.h | 54 ++++++
> > hw/virtio-blk.c | 59 ++++++-
> > hw/virtio-blk.h | 1 +
> > hw/virtio-pci.c | 3 +
> > trace-events | 9 +
> > 17 files changed, 1293 insertions(+), 2 deletions(-)
> > create mode 100644 hw/dataplane/Makefile.objs
> > create mode 100644 hw/dataplane/event-poll.c
> > create mode 100644 hw/dataplane/event-poll.h
> > create mode 100644 hw/dataplane/ioq.c
> > create mode 100644 hw/dataplane/ioq.h
> > create mode 100644 hw/dataplane/virtio-blk.c
> > create mode 100644 hw/dataplane/virtio-blk.h
> > create mode 100644 hw/dataplane/vring.c
> > create mode 100644 hw/dataplane/vring.h
> >
>
>
> --
> Asias
>
