V6 -> V7:
- Addressed Philippe Mathieu-Daudé comments
- modified pow2roundup32 signature
- added his RB tag (thanks)
- Addressed Corenlia Huck comments:
- Compiled the pvrdma for all archs and not only x86/arm (thanks)
- Fixed typo in documentation
- Rebased on latest master
V5 -> V6:
- Found a ppc machine and solved ppc compilation issues
- Tried to fix the s390x issue (still looking of a machine)
V4 -> V5:
- Fixed (at least tried to) compilation issues
V3 -> V4:
- Fixed documentation (added more impl details)
- Fixed compilation errors discovered by patchew.
- Addressed Michael S. Tsirkin comments:
- Removed unnecessary typedefs and replace them with
macros in VMware header files, together with explanations.
- Moved more code from vmw specific to rdma generic code.
- Added page size limitations to the documentation.
V2 -> V3:
- Addressed Michael S. Tsirkin and Philippe Mathieu-Daudé comments:
- Moved the device to hw/rdma
- Addressed Michael S. Tsirkin comments:
- Split the code into generic (hw/rdma) and VMWare
specific (hw/rdma/vmw)
- Added more details to documentation - VMware guest-host protocol.
- Remove mad processing
- limited the memory the Guest can pin.
- Addressed Philippe Mathieu-Daudé comment:
- s/roundup_pow_of_two/pow2roundup32 and move it to qemu/host-utils.h
- Added Shamit Rabinovici's review to documentation
- Rebased to latest master
RFC -> V2:
- Full implementation of the pvrdma device
- Backend is an ibdevice interface, no need for the KDBR module
General description
===================
PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
It works with its Linux Kernel driver AS IS, no need for any special guest
modifications.
While it complies with the VMware device, it can also communicate with bare
metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
can work with Soft-RoCE (rxe).
It does not require the whole guest RAM to be pinned allowing memory
over-commit and, even if not implemented yet, migration support will be
possible with some HW assistance.
Design
======
- Follows the behavior of VMware's pvrdma device, however is not tightly
coupled with it and most of the code can be reused if we decide to
continue to a Virtio based RDMA device.
- It exposes 3 BARs:
BAR 0 - MSIX, utilize 3 vectors for command ring, async events and
completions
BAR 1 - Configuration of registers
BAR 2 - UAR, used to pass HW commands from driver.
- The device performs internal management of the RDMA
resources (PDs, CQs, QPs, ...), meaning the objects
are not directly coupled to a physical RDMA device resources.
The pvrdma backend is an ibdevice interface that can be exposed
either by a Soft-RoCE(rxe) device on machines with no RDMA device,
or an HCA SRIOV function(VF/PF).
Note that ibdevice interfaces can't be shared between pvrdma devices,
each one requiring a separate instance (rxe or SRIOV VF).
Tests and performance
=====================
Tested with SoftRoCE backend (rxe)/Mellanox ConnectX3,
and Mellanox ConnectX4 HCAs with:
- VMs in the same host
- VMs in different hosts
- VMs to bare metal.
The best performance achieved with ConnectX HCAs and buffer size
bigger than 1MB which was the line rate ~ 50Gb/s.
The conclusion is that using the PVRDMA device there are no
actual performance penalties compared to bare metal for big enough
buffers (which is quite common when using RDMA), while allowing
memory overcommit.
Marcel Apfelbaum (3):
mem: add share parameter to memory-backend-ram
docs: add pvrdma device documentation.
MAINTAINERS: add entry for hw/rdma
Yuval Shaia (2):
pci/shpc: Move function to generic header file
pvrdma: initial implementation
MAINTAINERS | 8 +
Makefile.objs | 2 +
backends/hostmem-file.c | 25 +-
backends/hostmem-ram.c | 4 +-
backends/hostmem.c | 21 ++
configure | 9 +-
docs/pvrdma.txt | 254 +++++++++++++
exec.c | 26 +-
hw/Makefile.objs | 1 +
hw/pci/shpc.c | 13 +-
hw/rdma/Makefile.objs | 6 +
hw/rdma/rdma_backend.c | 815 ++++++++++++++++++++++++++++++++++++++++++
hw/rdma/rdma_backend.h | 92 +++++
hw/rdma/rdma_backend_defs.h | 62 ++++
hw/rdma/rdma_rm.c | 619 ++++++++++++++++++++++++++++++++
hw/rdma/rdma_rm.h | 69 ++++
hw/rdma/rdma_rm_defs.h | 106 ++++++
hw/rdma/rdma_utils.c | 52 +++
hw/rdma/rdma_utils.h | 43 +++
hw/rdma/trace-events | 5 +
hw/rdma/vmw/pvrdma.h | 122 +++++++
hw/rdma/vmw/pvrdma_cmd.c | 679 +++++++++++++++++++++++++++++++++++
hw/rdma/vmw/pvrdma_dev_api.h | 602 +++++++++++++++++++++++++++++++
hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++++++
hw/rdma/vmw/pvrdma_dev_ring.h | 42 +++
hw/rdma/vmw/pvrdma_ib_verbs.h | 433 ++++++++++++++++++++++
hw/rdma/vmw/pvrdma_main.c | 644 +++++++++++++++++++++++++++++++++
hw/rdma/vmw/pvrdma_qp_ops.c | 212 +++++++++++
hw/rdma/vmw/pvrdma_qp_ops.h | 27 ++
hw/rdma/vmw/pvrdma_ring.h | 134 +++++++
hw/rdma/vmw/trace-events | 5 +
hw/rdma/vmw/vmw_pvrdma-abi.h | 311 ++++++++++++++++
include/exec/memory.h | 23 ++
include/exec/ram_addr.h | 3 +-
include/hw/pci/pci_ids.h | 3 +
include/qemu/host-utils.h | 10 +
include/qemu/osdep.h | 2 +-
include/sysemu/hostmem.h | 2 +-
include/sysemu/kvm.h | 2 +-
memory.c | 16 +-
target/s390x/kvm.c | 4 +-
util/oslib-posix.c | 4 +-
util/oslib-win32.c | 2 +-
43 files changed, 5589 insertions(+), 64 deletions(-)
create mode 100644 docs/pvrdma.txt
create mode 100644 hw/rdma/Makefile.objs
create mode 100644 hw/rdma/rdma_backend.c
create mode 100644 hw/rdma/rdma_backend.h
create mode 100644 hw/rdma/rdma_backend_defs.h
create mode 100644 hw/rdma/rdma_rm.c
create mode 100644 hw/rdma/rdma_rm.h
create mode 100644 hw/rdma/rdma_rm_defs.h
create mode 100644 hw/rdma/rdma_utils.c
create mode 100644 hw/rdma/rdma_utils.h
create mode 100644 hw/rdma/trace-events
create mode 100644 hw/rdma/vmw/pvrdma.h
create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
create mode 100644 hw/rdma/vmw/pvrdma_main.c
create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
create mode 100644 hw/rdma/vmw/pvrdma_ring.h
create mode 100644 hw/rdma/vmw/trace-events
create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h
--
2.13.5
