Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
On Wed, 10 Jan 2018 11:28:59 +0200 Marcel Apfelbaum wrote: > On 09/01/2018 14:51, Cornelia Huck wrote: > > On Tue, 9 Jan 2018 13:08:33 +0200 > > Yuval Shaia wrote: > > > >> On Tue, Jan 09, 2018 at 11:39:11AM +0100, Cornelia Huck wrote: > >>> On Sun, 7 Jan 2018 14:32:23 +0200 > >>> Marcel Apfelbaum wrote: > > > diff --git a/hw/rdma/vmw/pvrdma_dev_api.h b/hw/rdma/vmw/pvrdma_dev_api.h > new file mode 100644 > index 00..bf1986a976 > --- /dev/null > +++ b/hw/rdma/vmw/pvrdma_dev_api.h > @@ -0,0 +1,602 @@ > +/* > + * QEMU VMWARE paravirtual RDMA device definitions > + * > + * Copyright (C) 2018 Oracle > + * Copyright (C) 2018 Red Hat Inc > + * > + * Authors: > + * Yuval Shaia > + * Marcel Apfelbaum > + * > + * This work is licensed under the terms of the GNU GPL, version 2. > + * See the COPYING file in the top-level directory. > + * > + */ > + > +#ifndef PVRDMA_DEV_API_H > +#define PVRDMA_DEV_API_H > + > +/* > + * Following is an interface definition for PVRDMA device as provided by > + * VMWARE. > + * See original copyright from Linux kernel v4.14.5 header file > + * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h > >>> > >>> Could that file be exported as UAPI in the kernel and added to the > >>> linux-headers script? > >> > >> We took this approach as apposed to kernel-headers with the following on > >> our mind: > >> (1) This is the convention used in vmxnet3. > >> (2) vmw_pvrdma was introduced only lately, taking the kernel-headers > >> approach will force specific kernel on a host in order to compile QEMU. > > > > qemu will get the kernel headers once from the upstream kernel and then > > will be able to be built everywhere. > > > >> (3) To support VMWare's pvrdma device we took a snapshot of existing > >> driver/device settings and breezed there. This is driver/device API and we > >> can't allow our self to chase VMWare's tail whenever they are changing the > >> API. Just consider a case where they will change for example the ARM bit. > > > > But as want to enable the existing device driver, you'll want to be > > able to produce a compatible device anyway, don't you? Also, wouldn't > > VMWare break older kernels if they suddenly changed the api? > > > > I think it was a missunderstanding here :) > > We don't actually need the VMware headers here in order to compile the device. > Requesting the pvrdma headers to be present really limits the hosts we could > compile QEMU without an actual benefit. > > The headers describe the data structures passed by the Guest driver to the > PVRDMA device. > Having them on the host should not be a requirement, we could simply define > our own > structs but it would be harder to maintain. (diffing with linux headers to > see what's new) > > Note we don't care about ABI changes, we emulate a PCI device, not > a para-virt one, the driver has the contract to pass the same data > definitions always. > If the data changes, there are 2 options: > 1. Pass another "version nr" on the command channel (DSR) and we declare as > not supported > by this device and fail gracefully. > 2. Use another device id. OK, thanks for the clarification, that makes sense. > > We looked for what are the options for *not requiring* the headers to be > present > and we saw two in QEMU: > 1. Adding it to include/standard-headers/linux/ > But it seemed to much, only our device need them. > 2. Adding it to the hw/rdma/vwm directory > Please look at: (hw/net/vmxnet3.h) > > The second option seemed better, but we could go with "1." > if I got it right and you think is cleaner. > But again, the first option kind of requires updating the > linux headers always, but we want quite the opposite, to not > be connected to the changes. > I hope is more clear now. OK, that makes sense as well, if you intend this to be a one-time effort. "Copy a header from the Linux kernel" just raised kind of a flag for me. > > > > [Also, is there a canonical reference for this API?] > > > > Linux Kernel PVRDMA driver implementation... :) > > >> > >> Just IMHO. > >> > > Thanks a lot for your inputs, much appreciated! NP. I'd like to do more, as this is interesting stuff, but you know how it goes...
Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
On 09/01/2018 14:51, Cornelia Huck wrote: On Tue, 9 Jan 2018 13:08:33 +0200 Yuval Shaia wrote: On Tue, Jan 09, 2018 at 11:39:11AM +0100, Cornelia Huck wrote: On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum wrote: diff --git a/hw/rdma/vmw/pvrdma_dev_api.h b/hw/rdma/vmw/pvrdma_dev_api.h new file mode 100644 index 00..bf1986a976 --- /dev/null +++ b/hw/rdma/vmw/pvrdma_dev_api.h @@ -0,0 +1,602 @@ +/* + * QEMU VMWARE paravirtual RDMA device definitions + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef PVRDMA_DEV_API_H +#define PVRDMA_DEV_API_H + +/* + * Following is an interface definition for PVRDMA device as provided by + * VMWARE. + * See original copyright from Linux kernel v4.14.5 header file + * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h Could that file be exported as UAPI in the kernel and added to the linux-headers script? We took this approach as apposed to kernel-headers with the following on our mind: (1) This is the convention used in vmxnet3. (2) vmw_pvrdma was introduced only lately, taking the kernel-headers approach will force specific kernel on a host in order to compile QEMU. qemu will get the kernel headers once from the upstream kernel and then will be able to be built everywhere. (3) To support VMWare's pvrdma device we took a snapshot of existing driver/device settings and breezed there. This is driver/device API and we can't allow our self to chase VMWare's tail whenever they are changing the API. Just consider a case where they will change for example the ARM bit. But as want to enable the existing device driver, you'll want to be able to produce a compatible device anyway, don't you? Also, wouldn't VMWare break older kernels if they suddenly changed the api? I think it was a missunderstanding here :) We don't actually need the VMware headers here in order to compile the device. Requesting the pvrdma headers to be present really limits the hosts we could compile QEMU without an actual benefit. The headers describe the data structures passed by the Guest driver to the PVRDMA device. Having them on the host should not be a requirement, we could simply define our own structs but it would be harder to maintain. (diffing with linux headers to see what's new) Note we don't care about ABI changes, we emulate a PCI device, not a para-virt one, the driver has the contract to pass the same data definitions always. If the data changes, there are 2 options: 1. Pass another "version nr" on the command channel (DSR) and we declare as not supported by this device and fail gracefully. 2. Use another device id. We looked for what are the options for *not requiring* the headers to be present and we saw two in QEMU: 1. Adding it to include/standard-headers/linux/ But it seemed to much, only our device need them. 2. Adding it to the hw/rdma/vwm directory Please look at: (hw/net/vmxnet3.h) The second option seemed better, but we could go with "1." if I got it right and you think is cleaner. But again, the first option kind of requires updating the linux headers always, but we want quite the opposite, to not be connected to the changes. I hope is more clear now. [Also, is there a canonical reference for this API?] Linux Kernel PVRDMA driver implementation... Just IMHO. Thanks a lot for your inputs, much appreciated! Marcel (...) diff --git a/hw/rdma/vmw/pvrdma_ib_verbs.h b/hw/rdma/vmw/pvrdma_ib_verbs.h new file mode 100644 index 00..cf1430024b --- /dev/null +++ b/hw/rdma/vmw/pvrdma_ib_verbs.h @@ -0,0 +1,433 @@ +/* + * QEMU VMWARE paravirtual RDMA device definitions + * + * Copyright (C) 2018 Oracle + * Copyright (C) 2018 Red Hat Inc + * + * Authors: + * Yuval Shaia + * Marcel Apfelbaum + * + * This work is licensed under the terms of the GNU GPL, version 2. + * See the COPYING file in the top-level directory. + * + */ + +#ifndef PVRDMA_IB_VERBS_H +#define PVRDMA_IB_VERBS_H + +/* + * VMWARE headers we got from Linux kernel do not fully comply QEMU coding + * standards in sense of types and defines used. + * Since we didn't want to change VMWARE code, following set of typedefs + * and defines needed to compile these headers with QEMU introduced. + */ + +#define u8 uint8_t +#define u16unsigned short +#define u32uint32_t +#define u64uint64_t I think the headers update already takes care of some conversions. Otherwise, same comment as for the header above. Sorry, i'm not following, can you elaborate on that? There's some post processing done by the headers update. But I think you'd need to use __u8 and friends for an exported header anyway. + +/* + * Following is an interface definition for PVRDMA device as provided by + * VMWARE. + * See original copyright from Linux
Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
Hi Cornelia, On 09/01/2018 12:39, Cornelia Huck wrote: On Sun, 7 Jan 2018 14:32:23 +0200 Marcel Apfelbaum wrote: From: Yuval Shaia PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. It works with its Linux Kernel driver AS IS, no need for any special guest modifications. While it complies with the VMware device, it can also communicate with bare metal RDMA-enabled machines and does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe). It does not require the whole guest RAM to be pinned allowing memory over-commit and, even if not implemented yet, migration support will be possible with some HW assistance. Signed-off-by: Yuval Shaia Signed-off-by: Marcel Apfelbaum --- Makefile.objs | 2 + configure | 9 +- default-configs/arm-softmmu.mak| 1 + default-configs/i386-softmmu.mak | 1 + default-configs/x86_64-softmmu.mak | 1 + hw/Makefile.objs | 1 + hw/rdma/Makefile.objs | 6 + hw/rdma/rdma_backend.c | 815 + hw/rdma/rdma_backend.h | 92 + hw/rdma/rdma_backend_defs.h| 62 +++ hw/rdma/rdma_rm.c | 619 hw/rdma/rdma_rm.h | 69 hw/rdma/rdma_rm_defs.h | 106 + hw/rdma/rdma_utils.c | 52 +++ hw/rdma/rdma_utils.h | 43 ++ hw/rdma/trace-events | 5 + hw/rdma/vmw/pvrdma.h | 122 ++ hw/rdma/vmw/pvrdma_cmd.c | 679 ++ hw/rdma/vmw/pvrdma_dev_api.h | 602 +++ hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++ hw/rdma/vmw/pvrdma_dev_ring.h | 42 ++ hw/rdma/vmw/pvrdma_ib_verbs.h | 433 hw/rdma/vmw/pvrdma_main.c | 644 + hw/rdma/vmw/pvrdma_qp_ops.c| 212 ++ hw/rdma/vmw/pvrdma_qp_ops.h| 27 ++ hw/rdma/vmw/pvrdma_ring.h | 134 ++ hw/rdma/vmw/trace-events | 5 + hw/rdma/vmw/vmw_pvrdma-abi.h | 311 ++ include/hw/pci/pci_ids.h | 3 + 29 files changed, 5233 insertions(+), 4 deletions(-) create mode 100644 hw/rdma/Makefile.objs create mode 100644 hw/rdma/rdma_backend.c create mode 100644 hw/rdma/rdma_backend.h create mode 100644 hw/rdma/rdma_backend_defs.h create mode 100644 hw/rdma/rdma_rm.c create mode 100644 hw/rdma/rdma_rm.h create mode 100644 hw/rdma/rdma_rm_defs.h create mode 100644 hw/rdma/rdma_utils.c create mode 100644 hw/rdma/rdma_utils.h create mode 100644 hw/rdma/trace-events create mode 100644 hw/rdma/vmw/pvrdma.h create mode 100644 hw/rdma/vmw/pvrdma_cmd.c create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h create mode 100644 hw/rdma/vmw/pvrdma_main.c create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h create mode 100644 hw/rdma/vmw/pvrdma_ring.h create mode 100644 hw/rdma/vmw/trace-events create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h (...) diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak index b0d6e65038..0e7a3c1700 100644 --- a/default-configs/arm-softmmu.mak +++ b/default-configs/arm-softmmu.mak @@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y CONFIG_MSF2=y CONFIG_FW_CFG_DMA=y CONFIG_XILINX_AXI=y +CONFIG_PVRDMA=y diff --git a/default-configs/i386-softmmu.mak b/default-configs/i386-softmmu.mak index 95ac4b464a..88298e4ef5 100644 --- a/default-configs/i386-softmmu.mak +++ b/default-configs/i386-softmmu.mak @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) CONFIG_PXB=y CONFIG_ACPI_VMGENID=y CONFIG_FW_CFG_DMA=y +CONFIG_PVRDMA=y diff --git a/default-configs/x86_64-softmmu.mak b/default-configs/x86_64-softmmu.mak index 0221236825..f571da36eb 100644 --- a/default-configs/x86_64-softmmu.mak +++ b/default-configs/x86_64-softmmu.mak @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) CONFIG_PXB=y CONFIG_ACPI_VMGENID=y CONFIG_FW_CFG_DMA=y +CONFIG_PVRDMA=y Any reason you did not add this to other architectures? We simply didn't test other architectures, but it could work. I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at least builds (did not try to actually get it to work, although I don't see any immediate blocker for that). We will add it to s390, sure. Thanks, Marcel (...) [...]
Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
On Tue, 9 Jan 2018 13:08:33 +0200 Yuval Shaia wrote: > On Tue, Jan 09, 2018 at 11:39:11AM +0100, Cornelia Huck wrote: > > On Sun, 7 Jan 2018 14:32:23 +0200 > > Marcel Apfelbaum wrote: > > > diff --git a/hw/rdma/vmw/pvrdma_dev_api.h b/hw/rdma/vmw/pvrdma_dev_api.h > > > new file mode 100644 > > > index 00..bf1986a976 > > > --- /dev/null > > > +++ b/hw/rdma/vmw/pvrdma_dev_api.h > > > @@ -0,0 +1,602 @@ > > > +/* > > > + * QEMU VMWARE paravirtual RDMA device definitions > > > + * > > > + * Copyright (C) 2018 Oracle > > > + * Copyright (C) 2018 Red Hat Inc > > > + * > > > + * Authors: > > > + * Yuval Shaia > > > + * Marcel Apfelbaum > > > + * > > > + * This work is licensed under the terms of the GNU GPL, version 2. > > > + * See the COPYING file in the top-level directory. > > > + * > > > + */ > > > + > > > +#ifndef PVRDMA_DEV_API_H > > > +#define PVRDMA_DEV_API_H > > > + > > > +/* > > > + * Following is an interface definition for PVRDMA device as provided by > > > + * VMWARE. > > > + * See original copyright from Linux kernel v4.14.5 header file > > > + * drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h > > > > Could that file be exported as UAPI in the kernel and added to the > > linux-headers script? > > We took this approach as apposed to kernel-headers with the following on > our mind: > (1) This is the convention used in vmxnet3. > (2) vmw_pvrdma was introduced only lately, taking the kernel-headers > approach will force specific kernel on a host in order to compile QEMU. qemu will get the kernel headers once from the upstream kernel and then will be able to be built everywhere. > (3) To support VMWare's pvrdma device we took a snapshot of existing > driver/device settings and breezed there. This is driver/device API and we > can't allow our self to chase VMWare's tail whenever they are changing the > API. Just consider a case where they will change for example the ARM bit. But as want to enable the existing device driver, you'll want to be able to produce a compatible device anyway, don't you? Also, wouldn't VMWare break older kernels if they suddenly changed the api? [Also, is there a canonical reference for this API?] > > Just IMHO. > > > > > (...) > > > > > diff --git a/hw/rdma/vmw/pvrdma_ib_verbs.h b/hw/rdma/vmw/pvrdma_ib_verbs.h > > > new file mode 100644 > > > index 00..cf1430024b > > > --- /dev/null > > > +++ b/hw/rdma/vmw/pvrdma_ib_verbs.h > > > @@ -0,0 +1,433 @@ > > > +/* > > > + * QEMU VMWARE paravirtual RDMA device definitions > > > + * > > > + * Copyright (C) 2018 Oracle > > > + * Copyright (C) 2018 Red Hat Inc > > > + * > > > + * Authors: > > > + * Yuval Shaia > > > + * Marcel Apfelbaum > > > + * > > > + * This work is licensed under the terms of the GNU GPL, version 2. > > > + * See the COPYING file in the top-level directory. > > > + * > > > + */ > > > + > > > +#ifndef PVRDMA_IB_VERBS_H > > > +#define PVRDMA_IB_VERBS_H > > > + > > > +/* > > > + * VMWARE headers we got from Linux kernel do not fully comply QEMU > > > coding > > > + * standards in sense of types and defines used. > > > + * Since we didn't want to change VMWARE code, following set of typedefs > > > + * and defines needed to compile these headers with QEMU introduced. > > > + */ > > > + > > > +#define u8 uint8_t > > > +#define u16unsigned short > > > +#define u32uint32_t > > > +#define u64uint64_t > > > > I think the headers update already takes care of some conversions. > > Otherwise, same comment as for the header above. > > Sorry, i'm not following, can you elaborate on that? There's some post processing done by the headers update. But I think you'd need to use __u8 and friends for an exported header anyway. > > > > > > + > > > +/* > > > + * Following is an interface definition for PVRDMA device as provided by > > > + * VMWARE. > > > + * See original copyright from Linux kernel v4.14.5 header file > > > + * drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h > > > + */ > > > > (...) > > > > > diff --git a/hw/rdma/vmw/vmw_pvrdma-abi.h b/hw/rdma/vmw/vmw_pvrdma-abi.h > > > new file mode 100644 > > > index 00..8cfb9d7745 > > > --- /dev/null > > > +++ b/hw/rdma/vmw/vmw_pvrdma-abi.h > > > @@ -0,0 +1,311 @@ > > > +/* > > > + * QEMU VMWARE paravirtual RDMA device definitions > > > + * > > > + * Copyright (C) 2018 Oracle > > > + * Copyright (C) 2018 Red Hat Inc > > > + * > > > + * Authors: > > > + * Yuval Shaia > > > + * Marcel Apfelbaum > > > + * > > > + * This work is licensed under the terms of the GNU GPL, version 2. > > > + * See the COPYING file in the top-level directory. > > > + * > > > + */ > > > + > > > +#ifndef VMW_PVRDMA_ABI_H > > > +#define VMW_PVRDMA_ABI_H > > > + > > > +/* > > > + * Following is an interface definition for PVRDMA device as provided by > > > + * VMWARE. > > > + * See original copyright from Linux kernel v4.14.5 header file > > > + * include/uapi/rdma/vmw_pvrdma-abi.h > > > + */ > > > > This o
Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
On Tue, Jan 09, 2018 at 11:39:11AM +0100, Cornelia Huck wrote: > On Sun, 7 Jan 2018 14:32:23 +0200 > Marcel Apfelbaum wrote: > > > From: Yuval Shaia > > > > PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device. > > It works with its Linux Kernel driver AS IS, no need for any special guest > > modifications. > > > > While it complies with the VMware device, it can also communicate with bare > > metal RDMA-enabled machines and does not require an RDMA HCA in the host, it > > can work with Soft-RoCE (rxe). > > > > It does not require the whole guest RAM to be pinned allowing memory > > over-commit and, even if not implemented yet, migration support will be > > possible with some HW assistance. > > > > Signed-off-by: Yuval Shaia > > Signed-off-by: Marcel Apfelbaum > > --- > > Makefile.objs | 2 + > > configure | 9 +- > > default-configs/arm-softmmu.mak| 1 + > > default-configs/i386-softmmu.mak | 1 + > > default-configs/x86_64-softmmu.mak | 1 + > > hw/Makefile.objs | 1 + > > hw/rdma/Makefile.objs | 6 + > > hw/rdma/rdma_backend.c | 815 > > + > > hw/rdma/rdma_backend.h | 92 + > > hw/rdma/rdma_backend_defs.h| 62 +++ > > hw/rdma/rdma_rm.c | 619 > > hw/rdma/rdma_rm.h | 69 > > hw/rdma/rdma_rm_defs.h | 106 + > > hw/rdma/rdma_utils.c | 52 +++ > > hw/rdma/rdma_utils.h | 43 ++ > > hw/rdma/trace-events | 5 + > > hw/rdma/vmw/pvrdma.h | 122 ++ > > hw/rdma/vmw/pvrdma_cmd.c | 679 ++ > > hw/rdma/vmw/pvrdma_dev_api.h | 602 +++ > > hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++ > > hw/rdma/vmw/pvrdma_dev_ring.h | 42 ++ > > hw/rdma/vmw/pvrdma_ib_verbs.h | 433 > > hw/rdma/vmw/pvrdma_main.c | 644 + > > hw/rdma/vmw/pvrdma_qp_ops.c| 212 ++ > > hw/rdma/vmw/pvrdma_qp_ops.h| 27 ++ > > hw/rdma/vmw/pvrdma_ring.h | 134 ++ > > hw/rdma/vmw/trace-events | 5 + > > hw/rdma/vmw/vmw_pvrdma-abi.h | 311 ++ > > include/hw/pci/pci_ids.h | 3 + > > 29 files changed, 5233 insertions(+), 4 deletions(-) > > create mode 100644 hw/rdma/Makefile.objs > > create mode 100644 hw/rdma/rdma_backend.c > > create mode 100644 hw/rdma/rdma_backend.h > > create mode 100644 hw/rdma/rdma_backend_defs.h > > create mode 100644 hw/rdma/rdma_rm.c > > create mode 100644 hw/rdma/rdma_rm.h > > create mode 100644 hw/rdma/rdma_rm_defs.h > > create mode 100644 hw/rdma/rdma_utils.c > > create mode 100644 hw/rdma/rdma_utils.h > > create mode 100644 hw/rdma/trace-events > > create mode 100644 hw/rdma/vmw/pvrdma.h > > create mode 100644 hw/rdma/vmw/pvrdma_cmd.c > > create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h > > create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c > > create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h > > create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h > > create mode 100644 hw/rdma/vmw/pvrdma_main.c > > create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c > > create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h > > create mode 100644 hw/rdma/vmw/pvrdma_ring.h > > create mode 100644 hw/rdma/vmw/trace-events > > create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h > > (...) > > > diff --git a/default-configs/arm-softmmu.mak > > b/default-configs/arm-softmmu.mak > > index b0d6e65038..0e7a3c1700 100644 > > --- a/default-configs/arm-softmmu.mak > > +++ b/default-configs/arm-softmmu.mak > > @@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y > > CONFIG_MSF2=y > > CONFIG_FW_CFG_DMA=y > > CONFIG_XILINX_AXI=y > > +CONFIG_PVRDMA=y > > diff --git a/default-configs/i386-softmmu.mak > > b/default-configs/i386-softmmu.mak > > index 95ac4b464a..88298e4ef5 100644 > > --- a/default-configs/i386-softmmu.mak > > +++ b/default-configs/i386-softmmu.mak > > @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) > > CONFIG_PXB=y > > CONFIG_ACPI_VMGENID=y > > CONFIG_FW_CFG_DMA=y > > +CONFIG_PVRDMA=y > > diff --git a/default-configs/x86_64-softmmu.mak > > b/default-configs/x86_64-softmmu.mak > > index 0221236825..f571da36eb 100644 > > --- a/default-configs/x86_64-softmmu.mak > > +++ b/default-configs/x86_64-softmmu.mak > > @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM) > > CONFIG_PXB=y > > CONFIG_ACPI_VMGENID=y > > CONFIG_FW_CFG_DMA=y > > +CONFIG_PVRDMA=y > > Any reason you did not add this to other architectures? > > I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at > least builds (did not try to actually get it to work, although I don't > see any immediate blocker for that). > > (...) > > > diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c > > new file mod
Re: [Qemu-devel] [PATCH V6 4/5] pvrdma: initial implementation
On Sun, 7 Jan 2018 14:32:23 +0200
Marcel Apfelbaum wrote:
> From: Yuval Shaia
>
> PVRDMA is the QEMU implementation of VMware's paravirtualized RDMA device.
> It works with its Linux Kernel driver AS IS, no need for any special guest
> modifications.
>
> While it complies with the VMware device, it can also communicate with bare
> metal RDMA-enabled machines and does not require an RDMA HCA in the host, it
> can work with Soft-RoCE (rxe).
>
> It does not require the whole guest RAM to be pinned allowing memory
> over-commit and, even if not implemented yet, migration support will be
> possible with some HW assistance.
>
> Signed-off-by: Yuval Shaia
> Signed-off-by: Marcel Apfelbaum
> ---
> Makefile.objs | 2 +
> configure | 9 +-
> default-configs/arm-softmmu.mak| 1 +
> default-configs/i386-softmmu.mak | 1 +
> default-configs/x86_64-softmmu.mak | 1 +
> hw/Makefile.objs | 1 +
> hw/rdma/Makefile.objs | 6 +
> hw/rdma/rdma_backend.c | 815
> +
> hw/rdma/rdma_backend.h | 92 +
> hw/rdma/rdma_backend_defs.h| 62 +++
> hw/rdma/rdma_rm.c | 619
> hw/rdma/rdma_rm.h | 69
> hw/rdma/rdma_rm_defs.h | 106 +
> hw/rdma/rdma_utils.c | 52 +++
> hw/rdma/rdma_utils.h | 43 ++
> hw/rdma/trace-events | 5 +
> hw/rdma/vmw/pvrdma.h | 122 ++
> hw/rdma/vmw/pvrdma_cmd.c | 679 ++
> hw/rdma/vmw/pvrdma_dev_api.h | 602 +++
> hw/rdma/vmw/pvrdma_dev_ring.c | 139 +++
> hw/rdma/vmw/pvrdma_dev_ring.h | 42 ++
> hw/rdma/vmw/pvrdma_ib_verbs.h | 433
> hw/rdma/vmw/pvrdma_main.c | 644 +
> hw/rdma/vmw/pvrdma_qp_ops.c| 212 ++
> hw/rdma/vmw/pvrdma_qp_ops.h| 27 ++
> hw/rdma/vmw/pvrdma_ring.h | 134 ++
> hw/rdma/vmw/trace-events | 5 +
> hw/rdma/vmw/vmw_pvrdma-abi.h | 311 ++
> include/hw/pci/pci_ids.h | 3 +
> 29 files changed, 5233 insertions(+), 4 deletions(-)
> create mode 100644 hw/rdma/Makefile.objs
> create mode 100644 hw/rdma/rdma_backend.c
> create mode 100644 hw/rdma/rdma_backend.h
> create mode 100644 hw/rdma/rdma_backend_defs.h
> create mode 100644 hw/rdma/rdma_rm.c
> create mode 100644 hw/rdma/rdma_rm.h
> create mode 100644 hw/rdma/rdma_rm_defs.h
> create mode 100644 hw/rdma/rdma_utils.c
> create mode 100644 hw/rdma/rdma_utils.h
> create mode 100644 hw/rdma/trace-events
> create mode 100644 hw/rdma/vmw/pvrdma.h
> create mode 100644 hw/rdma/vmw/pvrdma_cmd.c
> create mode 100644 hw/rdma/vmw/pvrdma_dev_api.h
> create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.c
> create mode 100644 hw/rdma/vmw/pvrdma_dev_ring.h
> create mode 100644 hw/rdma/vmw/pvrdma_ib_verbs.h
> create mode 100644 hw/rdma/vmw/pvrdma_main.c
> create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.c
> create mode 100644 hw/rdma/vmw/pvrdma_qp_ops.h
> create mode 100644 hw/rdma/vmw/pvrdma_ring.h
> create mode 100644 hw/rdma/vmw/trace-events
> create mode 100644 hw/rdma/vmw/vmw_pvrdma-abi.h
(...)
> diff --git a/default-configs/arm-softmmu.mak b/default-configs/arm-softmmu.mak
> index b0d6e65038..0e7a3c1700 100644
> --- a/default-configs/arm-softmmu.mak
> +++ b/default-configs/arm-softmmu.mak
> @@ -132,3 +132,4 @@ CONFIG_GPIO_KEY=y
> CONFIG_MSF2=y
> CONFIG_FW_CFG_DMA=y
> CONFIG_XILINX_AXI=y
> +CONFIG_PVRDMA=y
> diff --git a/default-configs/i386-softmmu.mak
> b/default-configs/i386-softmmu.mak
> index 95ac4b464a..88298e4ef5 100644
> --- a/default-configs/i386-softmmu.mak
> +++ b/default-configs/i386-softmmu.mak
> @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
> CONFIG_PXB=y
> CONFIG_ACPI_VMGENID=y
> CONFIG_FW_CFG_DMA=y
> +CONFIG_PVRDMA=y
> diff --git a/default-configs/x86_64-softmmu.mak
> b/default-configs/x86_64-softmmu.mak
> index 0221236825..f571da36eb 100644
> --- a/default-configs/x86_64-softmmu.mak
> +++ b/default-configs/x86_64-softmmu.mak
> @@ -61,3 +61,4 @@ CONFIG_HYPERV_TESTDEV=$(CONFIG_KVM)
> CONFIG_PXB=y
> CONFIG_ACPI_VMGENID=y
> CONFIG_FW_CFG_DMA=y
> +CONFIG_PVRDMA=y
Any reason you did not add this to other architectures?
I added "CONFIG_PVRDMA=$(CONFIG_PCI)" to s390x-softmmu.mak, and it at
least builds (did not try to actually get it to work, although I don't
see any immediate blocker for that).
(...)
> diff --git a/hw/rdma/rdma_backend.c b/hw/rdma/rdma_backend.c
> new file mode 100644
> index 00..dcb799f49b
> --- /dev/null
> +++ b/hw/rdma/rdma_backend.c
(...)
> +static void poll_cq(RdmaDeviceResources *rdma_dev_res, struct ibv_cq *ibcq,
> +bool one_poll)
> +{
> +int i, ne;
> +BackendCtx *bctx;
> +struct ibv_wc wc[2];
> +
> +p
