Hello all:
This seires is an update of last version of multiqueue virtio-net support.
This series tries to brings multiqueue support to virtio-net through a
multiqueue support tap backend and multiple vhost threads.
Patch 1 converts bitfield in TAPState to bool. Patch 2 replace assert(0) with
abort() in tap.
To support this, multiqueue nic support were added to qemu. This is done by
introducing an array of NetClientStates in NICState, and make each pair of peers
to be an queue of the nic. This is done in patch 3-9.
Tap were also converted to be able to create a multiple queue
backend. Currently, only linux support this by issuing TUNSETIFF N times with
the same device name to create N queues. Each fd returned by TUNSETIFF were a
queue supported by kernel. Three new command lines were introduced, queues
were used to tell how many queues will be created by qemu; fds were used to
pass multiple pre-created tap file descriptors to qemu; vhostfds were used to
pass multiple pre-created vhost descriptors to qemu. This is done in patch
10-15.
A method of deleting a queue and queue_index were also introduce for virtio,
this is done in patch 16-17.
Vhost were also changed to support multiqueue by introducing a start vq index
which tracks the first virtqueue that will be used by vhost instead of the
assumption that the vhost always use virtqueue from index 0. This is done in
patch 18.
The last part is the multiqueue userspace changes, this is done in patch 19-22.
With this changes, user could start a multiqueue virtio-net device through
./qemu -netdev tap,id=hn0,queues=2,vhost=on -device virtio-net-pci,netdev=hn0
Management tools such as libvirt can pass multiple pre-created fds/vhostfds
through
./qemu -netdev tap,id=hn0,fds=X:Y,vhostfds=M:N -device virtio-net-pci,netdev=hn0
For the one who wants to try, a git tree is available at:
git://github.com/jasowang/qemu.git
Changes from V4:
- fix the conflict with Michael's tree and rebase to the latest (Michael)
Changes from V3:
- convert bitfield to bool in TAPState (Blue)
- use abort() instead of assert(0) in tap code (Blue)
- rebase to the latest
- fix a bug that breaks the non-tap network
Changes from V2:
- Don't start/stop vhost threads when changing queues and simplify the interface
between virtio-net and vhost further.
Changes from V1:
- silent checkpatch (Blue)
- use fds/vhostfds instead of fd/vhostfd (Stefan)
- use fds=X:Y:Z instead of fd=X,fd=Y,fd=Z (Anthony)
- split patches (Stefan)
- typos in commit log (Stefan)
- Warn 'queues=' when fds/vhostfds is used (Stefan)
- rename __net_init_tap to net_init_tap_one (Stefan)
- check the consistency of vnet_hdr of multiple tap fds (Stefan)
- disable multiqueue support for bridge-helper (Stefan)
- rename tap_attach()/tap_detach() to tap_enable()/tap_disable() (Stefan)
- fix booting with legacy guest (WanLong)
- don't bump the version when doing migration (Michael)
- simplify the interface between virtio-net and multiqueue vhost_net (Michael)
- rebase the patches to latest
- re-order the patches that let the net part comes first to simplify the
reviewing
- simplify the interface between virtio-net and multiqueue vhost_net
- move the guest notifiers setup from vhost to vhost_net
- fix a build issue of hw/mcf_fce.c
Changes from RFC v2:
- rebase the codes to latest qemu
- align the multiqueue virtio-net implementation to virtio spec
- split the patches into more smaller patches
- set_link and hotplug support
Changes from RFC V1:
- rebase to the latest
- fix memory leak in parse_netdev
- fix guest notifiers assignment/de-assignment
- changes the command lines to:
qemu -netdev tap,queues=2 -device virtio-net-pci,queues=2
Reference:
V1: http://lists.nongnu.org/archive/html/qemu-devel/2012-12/msg03558.html
RFC v2: http://lists.gnu.org/archive/html/qemu-devel/2012-06/msg04108.html
RFC v1: http://comments.gmane.org/gmane.comp.emulators.qemu/100481
Perf Numbers:
- norm is short for normalize result
- trans.rate is short for transaction rate
Two Intel Xeon 5620 with direct connected intel 82599EB
Host/Guest kernel: David net tree
vhost enabled
- lots of improvents of both latency and cpu utilization in request-reponse test
- get regression of guest sending small packets which because TCP tends to batch
less when the latency were improved
1q/2q/4q
TCP_RR
size #sessions trans.rate norm trans.rate norm trans.rate norm
1 1 9393.26 595.64 9408.18 597.34 9375.19 584.12
1 2072162.1 2214.24 129880.22 2456.13 196949.81 2298.13
1 50107513.38 2653.99 139721.93 2490.58 259713.82 2873.57
1 100 126734.63 2676.54 145553.5 2406.63 265252.68 2943
64 19453.42 632.33 9371.37 616.13 9338.19 615.97
64 20 70620.03 2093.68 125155.75 2409.15 191239.91 2253.32
64 50 1069662448.29 146518.67 2514.47 242134.07 2720.91
64 100 117046.35 2394.56 190153.09 2696.82 238881.29 2704.41
256 1 8733.29 736.36 8701.07 680.83 8608.92 530.1
256 20 69279.89 2274.45 115103.07
