Hi,
> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Tuesday, September 25, 2018 8:07 PM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coque...@redhat.com>; Bie,
> Tiwei <tiwei....@intel.com>; Wang, Zhihong <zhihong.w...@intel.com>
> Cc: Wang, Xiao W <xiao.w.w...@intel.com>; Rami Rosen
> <roszenr...@gmail.com>; Wang, Haiyue <haiyue.w...@intel.com>; Ye,
> Xiaolong <xiaolong...@intel.com>
> Subject: [PATCH v5 2/2] examples/vdpa: introduce a new sample for vDPA
>
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
>
> Signed-off-by: Xiao Wang <xiao.w.w...@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong...@intel.com>
> ---
> MAINTAINERS | 2 +
> doc/guides/rel_notes/release_18_11.rst | 8 +
> doc/guides/sample_app_ug/index.rst | 1 +
> doc/guides/sample_app_ug/vdpa.rst | 118 +++++++
> examples/Makefile | 2 +-
> examples/vdpa/Makefile | 32 ++
> examples/vdpa/main.c | 466 +++++++++++++++++++++++++
> examples/vdpa/meson.build | 16 +
> 8 files changed, 644 insertions(+), 1 deletion(-)
> create mode 100644 doc/guides/sample_app_ug/vdpa.rst
> create mode 100644 examples/vdpa/Makefile
> create mode 100644 examples/vdpa/main.c
> create mode 100644 examples/vdpa/meson.build
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
> F: examples/vhost_scsi/
> F: doc/guides/sample_app_ug/vhost_scsi.rst
> F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
>
> Vhost PMD
> M: Maxime Coquelin <maxime.coque...@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst
> b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
> SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
> vdev_netvsc, tap, and failsafe drivers combination.
>
> +* **Add a new sample for vDPA**
> +
> + The vdpa sample application creates vhost-user sockets by using the
> + vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> + virtio ring compatible devices to serve virtio driver directly to enable
> + datapath acceleration. As vDPA driver can help to set up vhost datapath,
> + this application doesn't need to launch dedicated worker threads for vhost
> + enqueue/dequeue operations.
>
> API Changes
> -----------
> diff --git a/doc/guides/sample_app_ug/index.rst
> b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
> vhost
> vhost_scsi
> vhost_crypto
> + vdpa
> netmap_compatibility
> ip_pipeline
> test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst
> b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..a089393c0
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,118 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> + ./vdpa [EAL options] -- [--client] [--interactive|-i] or [--iface
> SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU
> needs
> + to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> + /tmp/vhost-user-, then the socket files will be named as
> /tmp/vhost-user-<n>
> + (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> + internal cmds are supported:
> +
> + 1. help: show help message
> + 2. list: list all available vdpa devices
> + 3. create: create a new vdpa port with socket file and vdpa device address
> + 4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> + ./vdpa --log-level=9 -c 0x2 -n 4 --socket-mem 1024,1024 \
> + -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> + -- --interactive
DPDK has log level 1-8, setting the log-level=9 looks strange.
Here you can add description about the whitelisted device, since they are
virtio ring compatible devices.
> +
> +.. note::
> + We need to bind vfio-pci to VFs before running vdpa sample.
> +
> + * modprobe vfio-pci
> + * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> + vdpa> list
> + device id device address queue num supported features
> + 0 0000:06:00.3 1 0x5572362272
> + 1 0000:06:00.4 1 0x5572362272
> +
> + vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> + vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> + qemu-system-x86_64 -cpu host -enable-kvm \
> + <snip>
> + -mem-prealloc \
> + -chardev socket,id=char0,path=<socket_file created in above steps> \
> + -netdev type=vhost-user,id=vdpa,chardev=char0 \
> + -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
In order to leverage EPT, add parameter "page-per-vq=on" for the virtio-net-pci
deivce.
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> + Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost
> backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume
> A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM
> on A,
> + in migration-listen mode:
> +
> +.. code-block:: console
> +
> + B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> + A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> + A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
> DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
> endif
> DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
> ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
> DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
> endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..7acda67c4
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,466 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +
[...]
> +};
> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> + cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> + __attribute__((unused)) void *parsed_result,
> + struct cmdline *cl,
> + __attribute__((unused)) void *data)
> +{
> + int did;
> + uint32_t queue_num;
> + uint64_t features;
> + struct rte_vdpa_device *vdev;
> + struct rte_pci_addr addr;
> +
> + cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
> features\n");
> + for (did = 0; did < dev_total; did++) {
> + vdev = rte_vdpa_get_device(did);
> + if (!vdev)
> + continue;
> + if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> + RTE_LOG(ERR, VDPA,
> + "failed to get vdpa queue number "
> + "for device id %d.\n", did);
> + continue;
> + }
> + if (vdev->ops->get_features(did, &features) < 0) {
> + RTE_LOG(ERR, VDPA,
> + "failed to get vdpa features "
> + "for device id %d.\n", did);
> + continue;
> + }
> + addr = vdev->addr.pci_addr;
> + cmdline_printf(cl,
> + "%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIu64
> "\n", did,
s/PRIu64/PRIx64
> + addr.domain, addr.bus, addr.devid,
> + addr.function, queue_num, features);
> + }
> +}
BRs,
Xiao
