What's the reason for using qemu as a middleman? Couldn't the new PMD itself open /dev/vhost-net or the vhost-user socket and send the commands to set up virtqueues? That was the approach taken by Jianfeng's earlier RFC.
On Thu, Nov 19, 2015 at 2:57 AM, Tetsuya Mukawa <mukawa at igel.co.jp> wrote: > THIS IS A PoC IMPLEMENATION. > > [Abstraction] > > Normally, virtio-net PMD only works on VM, because there is no virtio-net > device on host. > This RFC patch extends virtio-net PMD to be able to work on host as > virtual PMD. > But we didn't implement virtio-net device as a part of virtio-net PMD. > To prepare virtio-net device for the PMD, start QEMU process with special > QTest mode, then connect it from virtio-net PMD through unix domain socket. > > The PMD can connect to anywhere QEMU virtio-net device can. > For example, the PMD can connects to vhost-net kernel module and > vhost-user backend application. > Similar to virtio-net PMD on QEMU, application memory that uses virtio-net > PMD will be shared between vhost backend application. > But vhost backend application memory will not be shared. > > Main target of this PMD is container like docker, rkt, lxc and etc. > We can isolate related processes(virtio-net PMD process, QEMU and > vhost-user backend process) by container. > But, to communicate through unix domain socket, shared directory will be > needed. > > > [How to use] > > So far, we need QEMU patch to connect to vhost-user backend. > Please check known issue in later section. > Because of this, I will describe example of using vhost-net kernel module. > > - Compile > Set "CONFIG_RTE_VIRTIO_VDEV=y" in config/common_linux. > Then compile it. > > - Start QEMU like below. > $ sudo qemu-system-x86_64 -qtest unix:/tmp/qtest0,server -machine > accel=qtest \ > -display none -qtest-log /dev/null \ > -netdev > type=tap,script=/etc/qemu-ifup,id=net0,vhost=on \ > -device virtio-net-pci,netdev=net0 \ > -chardev > socket,id=chr1,path=/tmp/ivshmem0,server \ > -device ivshmem,size=1G,chardev=chr1,vectors=1 > > - Start DPDK application like below > $ sudo ./testpmd -c f -n 1 -m 1024 --shm \ > > --vdev="eth_cvio0,qtest=/tmp/qtest0,ivshmem=/tmp/ivshmem0" -- \ > --disable-hw-vlan --txqflags=0xf00 -i > > - Check created tap device. > > (*1) Please Specify same memory size in QEMU and DPDK command line. > > > [Detailed Description] > > - virtio-net device implementation > The PMD uses QEMU virtio-net device. To do that, QEMU QTest functionality > is used. > QTest is a test framework of QEMU devices. It allows us to implement a > device driver outside of QEMU. > With QTest, we can implement DPDK application and virtio-net PMD as > standalone process on host. > When QEMU is invoked as QTest mode, any guest code will not run. > To know more about QTest, see below. > http://wiki.qemu.org/Features/QTest > > - probing devices > QTest provides a unix domain socket. Through this socket, driver process > can access to I/O port and memory of QEMU virtual machine. > The PMD will send I/O port accesses to probe pci devices. > If we can find virtio-net and ivshmem device, initialize the devices. > Also, I/O port accesses of virtio-net PMD will be sent through socket, and > virtio-net PMD can initialize vitio-net device on QEMU correctly. > > - ivshmem device to share memory > To share memory that virtio-net PMD process uses, ivshmem device will be > used. > Because ivshmem device can only handle one file descriptor, shared memory > should be consist of one file. > To allocate such a memory, EAL has new option called "--shm". > If the option is specified, EAL will open a file and allocate memory from > hugepages. > While initializing ivshmem device, we can set BAR(Base Address Register). > It represents which memory QEMU vcpu can access to this shared memory. > We will specify host physical address of shared memory as this address. > It is very useful because we don't need to apply patch to QEMU to > calculate address offset. > (For example, if virtio-net PMD process will allocate memory from shared > memory, then specify the physical address of it to virtio-net register, > QEMU virtio-net device can understand it without calculating address > offset.) > > - Known limitation > So far, the PMD doesn't handle interrupts from QEMU devices. > Because of this, VIRTIO_NET_F_STATUS functionality is dropped. > But without it, we can use all virtio-net functions. > > - Known issues > So far, to use vhost-user, we need to apply vhost-user patch to QEMU and > DPDK vhost library. > This is because, QEMU will not send memory information and file descriptor > of ivshmem device to vhost-user backend. > (Anyway, vhost-net kernel module can receive the information. So > vhost-user behavior will not be correct. I will submit the patch to QEMU > soon) > Also, we may have an issue in DPDK vhost library to handle kickfd and > callfd. The patch for it is needed. > (Let me check it more) > If someone wants to check vhost-user behavior, I will describe it more in > later email. > > > [Addition] > > We can apply same manner to handle any kind of QEMU devices from DPDK > application. > So far, I don't have any ideas except for virtio-net device. But someone > would have. > > > Tetsuya Mukawa (2): > EAL: Add new EAL "--shm" option. > virtio: Extend virtio-net PMD to support container environment > > config/common_linuxapp | 5 + > drivers/net/virtio/Makefile | 4 + > drivers/net/virtio/qtest.c | 590 > +++++++++++++++++++++++++++++ > drivers/net/virtio/virtio_ethdev.c | 214 ++++++++++- > drivers/net/virtio/virtio_ethdev.h | 16 + > drivers/net/virtio/virtio_pci.h | 25 ++ > lib/librte_eal/common/eal_common_options.c | 5 + > lib/librte_eal/common/eal_internal_cfg.h | 1 + > lib/librte_eal/common/eal_options.h | 2 + > lib/librte_eal/common/include/rte_memory.h | 5 + > lib/librte_eal/linuxapp/eal/eal_memory.c | 71 ++++ > 11 files changed, 917 insertions(+), 21 deletions(-) > create mode 100644 drivers/net/virtio/qtest.c > > -- > 2.1.4 > >