On Sat, May 28, 2016 at 12:27 AM, Stefan Hajnoczi <stefa...@gmail.com> wrote: > On Fri, May 27, 2016 at 01:55:04PM +0200, Roman Pen wrote: >> Hello, all. >> >> This is RFC because mostly this patch is a quick attempt to get true >> multithreaded multiqueue support for a block device with native AIO. >> The goal is to squeeze everything possible on lockless IO path from >> MQ block on a guest to MQ block on a host. >> >> To avoid any locks in qemu backend and not to introduce thread safety >> into qemu block-layer I open same backend device several times, one >> device per one MQ. e.g. the following is the stack for a virtio-blk >> with num-queues=2: >> >> VirtIOBlock >> / \ >> VirtQueue#0 VirtQueue#1 >> IOThread#0 IOThread#1 >> BH#0 BH#1 >> Backend#0 Backend#1 >> \ / >> /dev/null0 >> >> To group all objects related to one vq new structure is introduced: >> >> typedef struct VirtQueueCtx { >> BlockBackend *blk; >> struct VirtIOBlock *s; >> VirtQueue *vq; >> void *rq; >> QEMUBH *bh; >> QEMUBH *batch_notify_bh; >> IOThread *iothread; >> Notifier insert_notifier; >> Notifier remove_notifier; >> /* Operation blocker on BDS */ >> Error *blocker; >> } VirtQueueCtx; >> >> And VirtIOBlock includes an array of these contexts: >> >> typedef struct VirtIOBlock { >> VirtIODevice parent_obj; >> + VirtQueueCtx mq[VIRTIO_QUEUE_MAX]; >> ... >> >> This patch is based on Stefan's series: "virtio-blk: multiqueue support", >> with minor difference: I reverted "virtio-blk: multiqueue batch notify", >> which does not make a lot sense when each VQ is handled by it's own >> iothread. >> >> The qemu configuration stays the same, i.e. put num-queues=N and N >> iothreads will be started on demand and N drives will be opened: >> >> qemu -device virtio-blk-pci,num-queues=8 >> >> My configuration is the following: >> >> host: >> Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz, >> 8 CPUs, >> /dev/nullb0 as backend with the following parameters: >> $ cat /sys/module/null_blk/parameters/submit_queues >> 8 >> $ cat /sys/module/null_blk/parameters/irqmode >> 1 >> >> guest: >> 8 VCPUs >> >> qemu: >> -object iothread,id=t0 \ >> -drive >> if=none,id=d0,file=/dev/nullb0,format=raw,snapshot=off,cache=none,aio=native >> \ >> -device >> virtio-blk-pci,num-queues=$N,iothread=t0,drive=d0,disable-modern=off,disable-legacy=on >> >> where $N varies during the tests. >> >> fio: >> [global] >> description=Emulation of Storage Server Access Pattern >> bssplit=512/20:1k/16:2k/9:4k/12:8k/19:16k/10:32k/8:64k/4 >> fadvise_hint=0 >> rw=randrw:2 >> direct=1 >> >> ioengine=libaio >> iodepth=64 >> iodepth_batch_submit=64 >> iodepth_batch_complete=64 >> numjobs=8 >> gtod_reduce=1 >> group_reporting=1 >> >> time_based=1 >> runtime=30 >> >> [job] >> filename=/dev/vda >> >> Results: >> num-queues RD bw WR bw >> ---------- ----- ----- >> >> * with 1 iothread * >> >> 1 thr 1 mq 1225MB/s 1221MB/s >> 1 thr 2 mq 1559MB/s 1553MB/s >> 1 thr 4 mq 1729MB/s 1725MB/s >> 1 thr 8 mq 1660MB/s 1655MB/s >> >> * with N iothreads * >> >> 2 thr 2 mq 1845MB/s 1842MB/s >> 4 thr 4 mq 2187MB/s 2183MB/s >> 8 thr 8 mq 1383MB/s 1378MB/s >> >> Obviously, 8 iothreads + 8 vcpu threads is too much for my machine >> with 8 CPUs, but 4 iothreads show quite good result. > > Cool, thanks for trying this experiment and posting results. > > It's encouraging to see the improvement. Did you use any CPU affinity > settings to co-locate vcpu and iothreads onto host CPUs?
No, in these measurements I did not try to pin anything. But the following are results with pinning, take a look: 8 VCPUs, 8 fio jobs =========================================================== o each fio job is pinned to VCPU in 1 to 1 o VCPUs are not pinned o iothreads are not pinned num queues RD bw ---------- -------- * with 1 iothread * 1 thr 1 mq 1096MB/s 1 thr 2 mq 1602MB/s 1 thr 4 mq 1818MB/s 1 thr 8 mq 1860MB/s * with N iothreads * 2 thr 2 mq 2008MB/s 4 thr 4 mq 2267MB/s 8 thr 8 mq 1388MB/s 8 VCPUs, 8 fio jobs =============================================== o each fio job is pinned to VCPU in 1 to 1 o each VCPU is pinned to CPU in 1 to 1 o each iothread is pinned to CPU in 1 to 1 affinity masks: CPUs 01234567 VCPUs XXXXXXXX num queues RD bw iothreads affinity mask ---------- -------- ----------------------- * with 1 iothread * 1 thr 1 mq 997MB/s X------- 1 thr 2 mq 1066MB/s X------- 1 thr 4 mq 969MB/s X------- 1 thr 8 mq 1050MB/s X------- * with N iothreads * 2 thr 2 mq 1597MB/s XX------ 4 thr 4 mq 1985MB/s XXXX---- 8 thr 8 mq 1230MB/s XXXXXXXX 4 VCPUs, 4 fio jobs =============================================== o each fio job is pinned to VCPU in 1 to 1 o VCPUs are not pinned o iothreads are not pinned num queues RD bw ---------- -------- * with 1 iothread * 1 thr 1 mq 1312MB/s 1 thr 2 mq 1445MB/s 1 thr 4 mq 1505MB/s * with N iothreads * 2 thr 2 mq 1710MB/s 4 thr 4 mq 1590MB/s 4 VCPUs, 4 fio jobs =============================================== o each fio job is pinned to VCPU in 1 to 1 o each VCPU is pinned to CPU in 1 to 1 o each iothread is pinned to CPU in 1 to 1 affinity masks: CPUs 01234567 VCPUs XXXX---- num queues RD bw iothreads affinity mask ---------- -------- ----------------------- * with 1 iothread * 1 thr 1 mq 1230MB/s ----X--- 1 thr 2 mq 1357MB/s ----X--- 1 thr 4 mq 1430MB/s ----X--- * with N iothreads * 2 thr 2 mq 1803MB/s ----XX-- 4 thr 4 mq 1673MB/s ----XXXX 4 VCPUs, 4 fio jobs =============================================== o each fio job is pinned to VCPU in 1 to 1 o each VCPU is pinned to 0123 CPUs o each iothread is pinned to 4567 CPUs affinity masks: CPUs 01234567 VCPUs XXXX---- num queues RD bw iothreads affinity mask ---------- -------- ----------------------- * with 1 iothread * 1 thr 1 mq 1213MB/s ----XXXX 1 thr 2 mq 1417MB/s ----XXXX 1 thr 4 mq 1435MB/s ----XXXX * with N iothreads * 2 thr 2 mq 1792MB/s ----XXXX 4 thr 4 mq 1667MB/s ----XXXX SUMMARY: For 8 jobs the only thing I noticed makes sense is fio job pinning. On my machine with 8 CPUs there is no room to optimize execution of 8 jobs. For 4 jobs and 4 VCPUs I tried to pin VCPUs threads and iothreads to different CPUs: VCPUs go to 0123, iothreads go to 4567. And seems that brings something, but not that much. -- Roman