[RFC v4 2/3] virtio: Introduce Vdmabuf driver
This driver "transfers" a dmabuf created on the Guest to the Host. A common use-case for such a transfer includes sharing the scanout buffer created by a display server or a compositor running in the Guest with Qemu UI -- running on the Host. The "transfer" is accomplished by sharing the PFNs of all the pages associated with the dmabuf and having a new dmabuf created on the Host that is backed up by the pages mapped from the Guest. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 1105 +++ include/linux/virtio_vdmabuf.h | 287 +++ include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 87 +++ 6 files changed, 1489 insertions(+) create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 7b41130d3f35..e563c12f711e 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -139,4 +139,12 @@ config VIRTIO_DMA_SHARED_BUFFER This option adds a flavor of dma buffers that are backed by virtio resources. +config VIRTIO_VDMABUF + bool "Enables Vdmabuf driver in guest os" + default n + depends on VIRTIO + help +This driver provides a way to share the dmabufs created in +the Guest with the Host. + endif # VIRTIO_MENU diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 591e6f72aa54..b4bb0738009c 100644 --- a/drivers/virtio/Makefile +++ b/drivers/virtio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o obj-$(CONFIG_VIRTIO_DMA_SHARED_BUFFER) += virtio_dma_buf.o +obj-$(CONFIG_VIRTIO_VDMABUF) += virtio_vdmabuf.o diff --git a/drivers/virtio/virtio_vdmabuf.c b/drivers/virtio/virtio_vdmabuf.c new file mode 100644 index ..803b7398d3ed --- /dev/null +++ b/drivers/virtio/virtio_vdmabuf.c @@ -0,0 +1,1105 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + * Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define VIRTIO_VDMABUF_MAX_ID INT_MAX +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) +#define NEW_BUF_ID_GEN(vmid, cnt) (((vmid & 0x) << 32) | \ + ((cnt) & 0x)) + +/* one global drv object */ +static struct virtio_vdmabuf_info *drv_info; + +struct virtio_vdmabuf { + /* virtio device structure */ + struct virtio_device *vdev; + + /* virtual queue array */ + struct virtqueue *vqs[VDMABUF_VQ_MAX]; + + /* ID of guest OS */ + u64 vmid; + + /* spin lock that needs to be acquired before accessing +* virtual queue +*/ + spinlock_t vq_lock; + struct mutex recv_lock; + struct mutex send_lock; + + struct list_head msg_list; + + /* workqueue */ + struct workqueue_struct *wq; + struct work_struct recv_work; + struct work_struct send_work; + struct work_struct send_msg_work; + + struct virtio_vdmabuf_event_queue *evq; +}; + +static virtio_vdmabuf_buf_id_t get_buf_id(struct virtio_vdmabuf *vdmabuf) +{ + virtio_vdmabuf_buf_id_t buf_id = {0, {0, 0} }; + static int count = 0; + + count = count < VIRTIO_VDMABUF_MAX_ID ? count + 1 : 0; + buf_id.
[RFC v4 3/3] vhost: Add Vdmabuf backend
This backend acts as the counterpart to the Vdmabuf Virtio frontend. When it receives a new export event from the frontend, it raises an event to alert the Qemu UI/userspace. Qemu then "imports" this buffer using the Unique ID. As part of the import step, a new dmabuf is created on the Host using the page information obtained from the Guest. The fd associated with this dmabuf is made available to Qemu UI/userspace which then creates a texture from it for the purpose of displaying it. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c| 1372 include/uapi/linux/vhost.h |3 + 4 files changed, 1387 insertions(+) create mode 100644 drivers/vhost/vdmabuf.c diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig index 587fbae06182..9a99cc2611ca 100644 --- a/drivers/vhost/Kconfig +++ b/drivers/vhost/Kconfig @@ -89,4 +89,13 @@ config VHOST_CROSS_ENDIAN_LEGACY If unsure, say "N". +config VHOST_VDMABUF + bool "Vhost backend for the Vdmabuf driver" + depends on KVM && EVENTFD + select VHOST + default n + help + This driver works in pair with the Virtio Vdmabuf frontend. It can + be used to create a dmabuf using the pages shared by the Guest. + endif diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile index f3e1897cce85..5c2cea4a7eaf 100644 --- a/drivers/vhost/Makefile +++ b/drivers/vhost/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_VHOST) += vhost.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o vhost_iotlb-y := iotlb.o + +obj-$(CONFIG_VHOST_VDMABUF) += vhost_vdmabuf.o +vhost_vdmabuf-y := vdmabuf.o diff --git a/drivers/vhost/vdmabuf.c b/drivers/vhost/vdmabuf.c new file mode 100644 index ..fe0efe82683d --- /dev/null +++ b/drivers/vhost/vdmabuf.c @@ -0,0 +1,1372 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + *Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vhost.h" + +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) + +enum { + VHOST_VDMABUF_FEATURES = VHOST_FEATURES, +}; + +static struct virtio_vdmabuf_info *drv_info; + +struct kvm_instance { + struct kvm *kvm; + struct list_head link; +}; + +struct vhost_vdmabuf { + struct vhost_dev dev; + struct vhost_virtqueue vqs[VDMABUF_VQ_MAX]; + struct vhost_work send_work; + struct virtio_vdmabuf_event_queue *evq; + u64 vmid; + + struct list_head msg_list; + struct list_head list; + struct kvm *kvm; +}; + +static inline void vhost_vdmabuf_add(struct vhost_vdmabuf *new) +{ + list_add_tail(&new->list, &drv_info->head_vdmabuf_list); +} + +static inline struct vhost_vdmabuf *vhost_vdmabuf_find(u64 vmid) +{ + struct vhost_vdmabuf *found; + + list_for_each_entry(found, &drv_info->head_vdmabuf_list, list) + if (found->vmid == vmid) + return found; + + return NULL; +} + +static inline bool vhost_vdmabuf_del(struct vhost_vdmabuf *vdmabuf) +{ + struct vhost_vdmabuf *iter, *temp; + + list_for_each_entry_safe(iter, temp, +&drv_info->head_vdmabuf_list, +list) + if (iter == vdmabuf) { + list_del(&iter->list); + return true; + } + + return false; +} + +static inline void vhost_vdmabuf
[RFC v4 0/3] Introduce Virtio based Dmabuf driver(s)
The Virtual Dmabuf or Virtio based Dmabuf (Vdmabuf) driver can be used to "transfer" a page-backed dmabuf created in the Guest to the Host without making any copies. This is mostly accomplished by recreating the dmabuf on the Host using the PFNs and other meta-data shared by the guest. A use-case where this driver would be a good fit is a multi-GPU system (perhaps one discrete and one integrated) where one of the GPUs does not have access to the display/connectors/outputs. This could be an embedded system design decision or a restriction made at the firmware/BIOS level or perhaps the device is setup in UPT (Universal Passthrough) mode. When such a GPU is passthrough'd to a Guest OS, this driver can help in transferring the scanout buffer(s) (rendered using the native rendering stack) to the Host for the purpose of displaying them. Or, quite simply, this driver can be used to transfer a dmabuf created by an application running on the Guest to another application running on the Host. The userspace component running in the Guest that transfers the dmabuf is referred to as the producer or exporter and its counterpart running in the Host is referred to as importer or consumer. For instance, a Wayland compositor would potentially be a producer and Qemu UI would be a consumer. It is the producer's responsibility to not reuse or destroy the shared buffer while it is still being used by the consumer. The consumer would send a release cmd indicating that it is done after which the shared buffer can be safely used again by the producer. One way the producer can prevent accidental re-use of the shared buffer is to lock the buffer when it exports it and unlock it after it gets a release cmd. As an example, the GBM API provides a simple way to lock and unlock a surface's buffers. For each dmabuf that is to be shared with the Host, a 128-bit unique ID is generated that identifies this buffer across the whole system. This ID is a combination of the Qemu process ID, a counter and a randomizer. We could potentially use UUID API but we currently use the above mentioned combination to identify the source of the buffer at any given time for potential bookkeeping. A typical cycle starts with the producer or exporter calling the alloc_fd IOCTL to get a new fd/dmabuf from Vdmabuf. It would then import and render to it and finally exports it by calling the export IOCTL. A new unique ID is generated for this buffer and it gets registered with the Host. The Host then alerts the consumer or importer by raising an event and shares the ID. In response, the consumer calls the import IOCTL using the ID and gets a newly created dmabuf fd in return. After it is done using the dmabuf, the consumer finally calls the release IOCTL and the Guest is notified which in turn notifies the producer letting it know that the buffer is now safe to reuse. v2: - Added a notifier mechanism for getting the kvm pointer. - Added start and stop routines in the Vhost backend. - Augmented the cover letter and made some minor improvements. v3: - Refactored the code to make it similar to vsock. - Used two virtqueues instead of one for efficient two-way communication. v4: - Made Vdmabuf guest driver allocate the dma-buf and backing storage and exported it to Weston to be used as a render target. (Gerd) The Vdmabuf driver was tested using Weston (headless) and Qemu from here: https://gitlab.freedesktop.org/Vivek/weston/-/blob/vdmabuf/libweston/backend-headless/headless.c#L522 https://lists.nongnu.org/archive/html/qemu-devel/2021-02/msg02976.html TODO: - Use dma_fences to improve synchronization for multiple importers. - Ensure that a process other than Qemu can also be the importer on Host. Other Considerations: - Should virtio-gpu be augmented to provide the same functionality as vdmabuf? - How can virtio-gpu/vdmabuf work with a Windows Guest? - Should there be a Vhost backend for virtio-gpu to reduce overhead? - Should a transfer of a dma-buf from Guest to Host be dependent on a DRM driver (virtio-gpu)? Vivek Kasireddy (3): kvm: Add a notifier for create and destroy VM events virtio: Introduce Vdmabuf driver vhost: Add Vdmabuf backend drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c | 1372 +++ drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 1105 + include/linux/kvm_host.h|5 + include/linux/virtio_vdmabuf.h | 287 ++ include/uapi/linux/vhost.h |3 + include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 87 ++ virt/kvm/kvm_main.c | 20 +- 12 files changed, 2899 insertions(+), 2 deletions(-) create mode 100644 drivers/vhost/vdmabuf.c create mode 100644 drivers/virtio/virtio_vdmabuf.c create
[RFC v4 1/3] kvm: Add a notifier for create and destroy VM events
After registering with this notifier, other drivers that are dependent on KVM can get notified whenever a VM is created or destroyed. This also provides a way for sharing the KVM instance pointer with other drivers. Signed-off-by: Vivek Kasireddy --- include/linux/kvm_host.h | 5 + virt/kvm/kvm_main.c | 20 ++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f3b1013fb22c..fc1a688301a0 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -88,6 +88,9 @@ #define KVM_PFN_ERR_HWPOISON (KVM_PFN_ERR_MASK + 1) #define KVM_PFN_ERR_RO_FAULT (KVM_PFN_ERR_MASK + 2) +#define KVM_EVENT_CREATE_VM 0 +#define KVM_EVENT_DESTROY_VM 1 + /* * error pfns indicate that the gfn is in slot but faild to * translate it to pfn on host. @@ -1494,5 +1497,7 @@ static inline void kvm_handle_signal_exit(struct kvm_vcpu *vcpu) /* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +int kvm_vm_register_notifier(struct notifier_block *nb); +int kvm_vm_unregister_notifier(struct notifier_block *nb); #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 5f260488e999..8a0e8bb02a5f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -101,6 +101,8 @@ DEFINE_MUTEX(kvm_lock); static DEFINE_RAW_SPINLOCK(kvm_count_lock); LIST_HEAD(vm_list); +static struct blocking_notifier_head kvm_vm_notifier; + static cpumask_var_t cpus_hardware_enabled; static int kvm_usage_count; static atomic_t hardware_enable_failed; @@ -148,12 +150,20 @@ static void kvm_io_bus_destroy(struct kvm_io_bus *bus); __visible bool kvm_rebooting; EXPORT_SYMBOL_GPL(kvm_rebooting); -#define KVM_EVENT_CREATE_VM 0 -#define KVM_EVENT_DESTROY_VM 1 static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm); static unsigned long long kvm_createvm_count; static unsigned long long kvm_active_vms; +inline int kvm_vm_register_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&kvm_vm_notifier, nb); +} + +inline int kvm_vm_unregister_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_unregister(&kvm_vm_notifier, nb); +} + __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, unsigned long start, unsigned long end) { @@ -808,6 +818,8 @@ static struct kvm *kvm_create_vm(unsigned long type) preempt_notifier_inc(); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_CREATE_VM, kvm); return kvm; out_err: @@ -886,6 +898,8 @@ static void kvm_destroy_vm(struct kvm *kvm) preempt_notifier_dec(); hardware_disable_all(); mmdrop(mm); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_DESTROY_VM, kvm); } void kvm_get_kvm(struct kvm *kvm) @@ -4968,6 +4982,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, r = kvm_vfio_ops_init(); WARN_ON(r); + BLOCKING_INIT_NOTIFIER_HEAD(&kvm_vm_notifier); + return 0; out_unreg: -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC v3 2/3] virtio: Introduce Vdmabuf driver
This driver "transfers" a dmabuf created on the Guest to the Host. A common use-case for such a transfer includes sharing the scanout buffer created by a display server or a compositor running in the Guest with Qemu UI -- running on the Host. The "transfer" is accomplished by sharing the PFNs of all the pages associated with the dmabuf and having a new dmabuf created on the Host that is backed up by the pages mapped from the Guest. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 1090 +++ include/linux/virtio_vdmabuf.h | 271 +++ include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 99 +++ 6 files changed, 1470 insertions(+) create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 7b41130d3f35..e563c12f711e 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -139,4 +139,12 @@ config VIRTIO_DMA_SHARED_BUFFER This option adds a flavor of dma buffers that are backed by virtio resources. +config VIRTIO_VDMABUF + bool "Enables Vdmabuf driver in guest os" + default n + depends on VIRTIO + help +This driver provides a way to share the dmabufs created in +the Guest with the Host. + endif # VIRTIO_MENU diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 591e6f72aa54..b4bb0738009c 100644 --- a/drivers/virtio/Makefile +++ b/drivers/virtio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o obj-$(CONFIG_VIRTIO_DMA_SHARED_BUFFER) += virtio_dma_buf.o +obj-$(CONFIG_VIRTIO_VDMABUF) += virtio_vdmabuf.o diff --git a/drivers/virtio/virtio_vdmabuf.c b/drivers/virtio/virtio_vdmabuf.c new file mode 100644 index ..c28f144eb126 --- /dev/null +++ b/drivers/virtio/virtio_vdmabuf.c @@ -0,0 +1,1090 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + * Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define VIRTIO_VDMABUF_MAX_ID INT_MAX +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) +#define NEW_BUF_ID_GEN(vmid, cnt) (((vmid & 0x) << 32) | \ + ((cnt) & 0x)) + +/* one global drv object */ +static struct virtio_vdmabuf_info *drv_info; + +struct virtio_vdmabuf { + /* virtio device structure */ + struct virtio_device *vdev; + + /* virtual queue array */ + struct virtqueue *vqs[VDMABUF_VQ_MAX]; + + /* ID of guest OS */ + u64 vmid; + + /* spin lock that needs to be acquired before accessing +* virtual queue +*/ + spinlock_t vq_lock; + struct mutex recv_lock; + struct mutex send_lock; + + struct list_head msg_list; + + /* workqueue */ + struct workqueue_struct *wq; + struct work_struct recv_work; + struct work_struct send_work; + struct work_struct send_msg_work; + + struct virtio_vdmabuf_event_queue *evq; +}; + +static virtio_vdmabuf_buf_id_t get_buf_id(struct virtio_vdmabuf *vdmabuf) +{ + virtio_vdmabuf_buf_id_t buf_id = {0, {0, 0} }; + static int count = 0; + + count = count < VIRTIO_VDMABUF_MAX_ID ? count + 1 : 0; + buf_id.
[RFC v3 3/3] vhost: Add Vdmabuf backend
This backend acts as the counterpart to the Vdmabuf Virtio frontend. When it receives a new export event from the frontend, it raises an event to alert the Qemu UI/userspace. Qemu then "imports" this buffer using the Unique ID. As part of the import step, a new dmabuf is created on the Host using the page information obtained from the Guest. The fd associated with this dmabuf is made available to Qemu UI/userspace which then creates a texture from it for the purpose of displaying it. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c| 1446 include/uapi/linux/vhost.h |3 + 4 files changed, 1461 insertions(+) create mode 100644 drivers/vhost/vdmabuf.c diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig index 587fbae06182..9a99cc2611ca 100644 --- a/drivers/vhost/Kconfig +++ b/drivers/vhost/Kconfig @@ -89,4 +89,13 @@ config VHOST_CROSS_ENDIAN_LEGACY If unsure, say "N". +config VHOST_VDMABUF + bool "Vhost backend for the Vdmabuf driver" + depends on KVM && EVENTFD + select VHOST + default n + help + This driver works in pair with the Virtio Vdmabuf frontend. It can + be used to create a dmabuf using the pages shared by the Guest. + endif diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile index f3e1897cce85..5c2cea4a7eaf 100644 --- a/drivers/vhost/Makefile +++ b/drivers/vhost/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_VHOST) += vhost.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o vhost_iotlb-y := iotlb.o + +obj-$(CONFIG_VHOST_VDMABUF) += vhost_vdmabuf.o +vhost_vdmabuf-y := vdmabuf.o diff --git a/drivers/vhost/vdmabuf.c b/drivers/vhost/vdmabuf.c new file mode 100644 index ..1d6e9bcf6648 --- /dev/null +++ b/drivers/vhost/vdmabuf.c @@ -0,0 +1,1446 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + *Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vhost.h" + +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) + +enum { + VHOST_VDMABUF_FEATURES = VHOST_FEATURES, +}; + +static struct virtio_vdmabuf_info *drv_info; + +struct kvm_instance { + struct kvm *kvm; + struct list_head link; +}; + +struct vhost_vdmabuf { + struct vhost_dev dev; + struct vhost_virtqueue vqs[VDMABUF_VQ_MAX]; + struct vhost_work send_work; + struct virtio_vdmabuf_event_queue *evq; + u64 vmid; + + struct list_head msg_list; + struct list_head list; + struct kvm *kvm; +}; + +static inline void vhost_vdmabuf_add(struct vhost_vdmabuf *new) +{ + list_add_tail(&new->list, &drv_info->head_vdmabuf_list); +} + +static inline struct vhost_vdmabuf *vhost_vdmabuf_find(u64 vmid) +{ + struct vhost_vdmabuf *found; + + list_for_each_entry(found, &drv_info->head_vdmabuf_list, list) + if (found->vmid == vmid) + return found; + + return NULL; +} + +static inline bool vhost_vdmabuf_del(struct vhost_vdmabuf *vdmabuf) +{ + struct vhost_vdmabuf *iter, *temp; + + list_for_each_entry_safe(iter, temp, +&drv_info->head_vdmabuf_list, +list) + if (iter == vdmabuf) { + list_del(&iter->list); + return true; + } + + return false; +} + +static inline void vhost_vdmabuf
[RFC v3 1/3] kvm: Add a notifier for create and destroy VM events
After registering with this notifier, other drivers that are dependent on KVM can get notified whenever a VM is created or destroyed. This also provides a way for sharing the KVM instance pointer with other drivers. Signed-off-by: Vivek Kasireddy --- include/linux/kvm_host.h | 5 + virt/kvm/kvm_main.c | 20 ++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f3b1013fb22c..fc1a688301a0 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -88,6 +88,9 @@ #define KVM_PFN_ERR_HWPOISON (KVM_PFN_ERR_MASK + 1) #define KVM_PFN_ERR_RO_FAULT (KVM_PFN_ERR_MASK + 2) +#define KVM_EVENT_CREATE_VM 0 +#define KVM_EVENT_DESTROY_VM 1 + /* * error pfns indicate that the gfn is in slot but faild to * translate it to pfn on host. @@ -1494,5 +1497,7 @@ static inline void kvm_handle_signal_exit(struct kvm_vcpu *vcpu) /* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +int kvm_vm_register_notifier(struct notifier_block *nb); +int kvm_vm_unregister_notifier(struct notifier_block *nb); #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 5f260488e999..8a0e8bb02a5f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -101,6 +101,8 @@ DEFINE_MUTEX(kvm_lock); static DEFINE_RAW_SPINLOCK(kvm_count_lock); LIST_HEAD(vm_list); +static struct blocking_notifier_head kvm_vm_notifier; + static cpumask_var_t cpus_hardware_enabled; static int kvm_usage_count; static atomic_t hardware_enable_failed; @@ -148,12 +150,20 @@ static void kvm_io_bus_destroy(struct kvm_io_bus *bus); __visible bool kvm_rebooting; EXPORT_SYMBOL_GPL(kvm_rebooting); -#define KVM_EVENT_CREATE_VM 0 -#define KVM_EVENT_DESTROY_VM 1 static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm); static unsigned long long kvm_createvm_count; static unsigned long long kvm_active_vms; +inline int kvm_vm_register_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&kvm_vm_notifier, nb); +} + +inline int kvm_vm_unregister_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_unregister(&kvm_vm_notifier, nb); +} + __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, unsigned long start, unsigned long end) { @@ -808,6 +818,8 @@ static struct kvm *kvm_create_vm(unsigned long type) preempt_notifier_inc(); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_CREATE_VM, kvm); return kvm; out_err: @@ -886,6 +898,8 @@ static void kvm_destroy_vm(struct kvm *kvm) preempt_notifier_dec(); hardware_disable_all(); mmdrop(mm); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_DESTROY_VM, kvm); } void kvm_get_kvm(struct kvm *kvm) @@ -4968,6 +4982,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, r = kvm_vfio_ops_init(); WARN_ON(r); + BLOCKING_INIT_NOTIFIER_HEAD(&kvm_vm_notifier); + return 0; out_unreg: -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC v3 0/3] Introduce Virtio based Dmabuf driver
The Virtual Dmabuf or Virtio based Dmabuf (Vdmabuf) driver can be used to "transfer" a page-backed dmabuf created in the Guest to the Host without making any copies. This is mostly accomplished by recreating the dmabuf on the Host using the PFNs and other meta-data shared by the guest. A use-case where this driver would be a good fit is a multi-GPU system (perhaps one discrete and one integrated) where one of the GPUs does not have access to the display/connectors/outputs. This could be an embedded system design decision or a restriction made at the firmware/BIOS level or perhaps the device is setup in UPT (Universal Passthrough) mode. When such a GPU is passthrough'd to a Guest OS, this driver can help in transferring the scanout buffer(s) (rendered using the native rendering stack) to the Host for the purpose of displaying them. Or, quite simply, this driver can be used to transfer a dmabuf created by an application running on the Guest to another application running on the Host. The userspace component running in the Guest that transfers the dmabuf is referred to as the producer or exporter and its counterpart running in the Host is referred to as importer or consumer. For instance, a Wayland compositor would potentially be a producer and Qemu UI would be a consumer. It is the producer's responsibility to not reuse or destroy the shared buffer while it is still being used by the consumer. The consumer would send a release cmd indicating that it is done after which the shared buffer can be safely used again by the producer. One way the producer can prevent accidental re-use of the shared buffer is to lock the buffer when it exports it and unlock it after it gets a release cmd. As an example, the GBM API provides a simple way to lock and unlock a surface's buffers. For each dmabuf that is to be shared with the Host, a 128-bit unique ID is generated that identifies this buffer across the whole system. This ID is a combination of the Qemu process ID, a counter and a randomizer. We could potentially use UUID API but we currently use the above mentioned combination to identify the source of the buffer at any given time for potential bookkeeping. A typical cycle starts with the producer or exporter calling the export IOCTL to export a dmabuf; a new unique ID is generated for this buffer and it gets registered with the Host. The Host then alerts the consumer or importer by raising an event and shares the ID. In response, the consumer calls the import IOCTL using the ID and gets a newly created dmabuf fd in return. After it is done using the dmabuf, the consumer finally calls the release IOCTL and the Guest is notified which in turn notifies the producer letting it know that the buffer is now safe to reuse. v2: - Added a notifier mechanism for getting the kvm pointer. - Added start and stop routines in the Vhost backend. - Augmented the cover letter and made some minor improvements. v3: - Refactored the code to make it similar to vsock Vivek Kasireddy (3): kvm: Add a notifier for create and destroy VM events virtio: Introduce Vdmabuf driver vhost: Add Vdmabuf backend drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c | 1446 +++ drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 1090 include/linux/kvm_host.h|5 + include/linux/virtio_vdmabuf.h | 271 + include/uapi/linux/vhost.h |3 + include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 99 ++ virt/kvm/kvm_main.c | 20 +- 12 files changed, 2954 insertions(+), 2 deletions(-) create mode 100644 drivers/vhost/vdmabuf.c create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC v2 3/3] vhost: Add Vdmabuf backend
This backend acts as the counterpart to the Vdmabuf Virtio frontend. When it receives a new export event from the frontend, it raises an event to alert the Qemu UI/userspace. Qemu then "imports" this buffer using the Unique ID. As part of the import step, a new dmabuf is created on the Host using the page information obtained from the Guest. The fd associated with this dmabuf is made available to Qemu UI/userspace which then creates a texture from it for the purpose of displaying it. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c| 1407 include/uapi/linux/vhost.h |3 + 4 files changed, 1422 insertions(+) create mode 100644 drivers/vhost/vdmabuf.c diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig index 587fbae06182..9a99cc2611ca 100644 --- a/drivers/vhost/Kconfig +++ b/drivers/vhost/Kconfig @@ -89,4 +89,13 @@ config VHOST_CROSS_ENDIAN_LEGACY If unsure, say "N". +config VHOST_VDMABUF + bool "Vhost backend for the Vdmabuf driver" + depends on KVM && EVENTFD + select VHOST + default n + help + This driver works in pair with the Virtio Vdmabuf frontend. It can + be used to create a dmabuf using the pages shared by the Guest. + endif diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile index f3e1897cce85..5c2cea4a7eaf 100644 --- a/drivers/vhost/Makefile +++ b/drivers/vhost/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_VHOST) += vhost.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o vhost_iotlb-y := iotlb.o + +obj-$(CONFIG_VHOST_VDMABUF) += vhost_vdmabuf.o +vhost_vdmabuf-y := vdmabuf.o diff --git a/drivers/vhost/vdmabuf.c b/drivers/vhost/vdmabuf.c new file mode 100644 index ..2a8a1d852e93 --- /dev/null +++ b/drivers/vhost/vdmabuf.c @@ -0,0 +1,1407 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + *Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vhost.h" + +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) + +static struct virtio_vdmabuf_info *drv_info; + +struct kvm_instance { + struct kvm *kvm; + struct list_head link; +}; + +struct vhost_vdmabuf { + struct vhost_dev dev; + struct vhost_virtqueue vq; + struct vhost_work tx_work; + struct virtio_vdmabuf_event_queue *evq; + u64 vmid; + + /* synchronization between transmissions */ + struct mutex tx_mutex; + /* synchronization on tx and rx */ + struct mutex vq_mutex; + + struct virtio_vdmabuf_txmsg next; + struct list_head list; + struct kvm *kvm; +}; + +static inline void vhost_vdmabuf_add(struct vhost_vdmabuf *new) +{ + list_add_tail(&new->list, &drv_info->head_vdmabuf_list); +} + +static inline struct vhost_vdmabuf *vhost_vdmabuf_find(u64 vmid) +{ + struct vhost_vdmabuf *found; + + list_for_each_entry(found, &drv_info->head_vdmabuf_list, list) + if (found->vmid == vmid) + return found; + + return NULL; +} + +static inline bool vhost_vdmabuf_del(struct vhost_vdmabuf *vdmabuf) +{ + struct vhost_vdmabuf *iter, *temp; + + list_for_each_entry_safe(iter, temp, +&drv_info->head_vdmabuf_list, +list) + if (iter == vdmabuf) { + list_del(&iter->list); +
[RFC v2 2/3] virtio: Introduce Vdmabuf driver
This driver "transfers" a dmabuf created on the Guest to the Host. A common use-case for such a transfer includes sharing the scanout buffer created by a display server or a compositor running in the Guest with Qemu UI -- running on the Host. The "transfer" is accomplished by sharing the PFNs of all the pages associated with the dmabuf and having a new dmabuf created on the Host that is backed up by the pages mapped from the Guest. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/virtio/Kconfig | 8 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_vdmabuf.c | 986 include/linux/virtio_vdmabuf.h | 272 include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_vdmabuf.h | 99 +++ 6 files changed, 1367 insertions(+) create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 7b41130d3f35..e563c12f711e 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -139,4 +139,12 @@ config VIRTIO_DMA_SHARED_BUFFER This option adds a flavor of dma buffers that are backed by virtio resources. +config VIRTIO_VDMABUF + bool "Enables Vdmabuf driver in guest os" + default n + depends on VIRTIO + help +This driver provides a way to share the dmabufs created in +the Guest with the Host. + endif # VIRTIO_MENU diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 591e6f72aa54..b4bb0738009c 100644 --- a/drivers/virtio/Makefile +++ b/drivers/virtio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o obj-$(CONFIG_VIRTIO_DMA_SHARED_BUFFER) += virtio_dma_buf.o +obj-$(CONFIG_VIRTIO_VDMABUF) += virtio_vdmabuf.o diff --git a/drivers/virtio/virtio_vdmabuf.c b/drivers/virtio/virtio_vdmabuf.c new file mode 100644 index ..0b40ea4fd6f1 --- /dev/null +++ b/drivers/virtio/virtio_vdmabuf.c @@ -0,0 +1,986 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + * Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define VIRTIO_VDMABUF_MAX_ID INT_MAX +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) +#define NEW_BUF_ID_GEN(vmid, cnt) (((vmid & 0x) << 32) | \ + ((cnt) & 0x)) + +/* one global drv object */ +static struct virtio_vdmabuf_info *drv_info; + +struct virtio_vdmabuf { + /* virtio device structure */ + struct virtio_device *vdev; + + /* virtual queue array */ + struct virtqueue *vq; + + /* ID of guest OS */ + u64 vmid; + + /* spin lock that needs to be acquired before accessing +* virtual queue +*/ + spinlock_t vq_lock; + struct mutex rx_lock; + + /* workqueue */ + struct workqueue_struct *wq; + struct work_struct rx_work; + struct virtio_vdmabuf_event_queue *evq; +}; + +static virtio_vdmabuf_buf_id_t get_buf_id(struct virtio_vdmabuf *vdmabuf) +{ + virtio_vdmabuf_buf_id_t buf_id = {0, {0, 0} }; + static int count = 0; + + count = count < VIRTIO_VDMABUF_MAX_ID ? count + 1 : 0; + buf_id.id = NEW_BUF_ID_GEN(vdmabuf->vmid, count); + + /* random data embedded in the id for security */ + get_random_bytes(&buf_id.rng_key[0], 8); + + return buf_
[RFC v2 1/3] kvm: Add a notifier for create and destroy VM events
After registering with this notifier, other drivers that are dependent on KVM can get notified whenever a VM is created or destroyed. This also provides a way for sharing the KVM instance pointer with other drivers. Signed-off-by: Vivek Kasireddy --- include/linux/kvm_host.h | 5 + virt/kvm/kvm_main.c | 20 ++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index f3b1013fb22c..fc1a688301a0 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -88,6 +88,9 @@ #define KVM_PFN_ERR_HWPOISON (KVM_PFN_ERR_MASK + 1) #define KVM_PFN_ERR_RO_FAULT (KVM_PFN_ERR_MASK + 2) +#define KVM_EVENT_CREATE_VM 0 +#define KVM_EVENT_DESTROY_VM 1 + /* * error pfns indicate that the gfn is in slot but faild to * translate it to pfn on host. @@ -1494,5 +1497,7 @@ static inline void kvm_handle_signal_exit(struct kvm_vcpu *vcpu) /* Max number of entries allowed for each kvm dirty ring */ #define KVM_DIRTY_RING_MAX_ENTRIES 65536 +int kvm_vm_register_notifier(struct notifier_block *nb); +int kvm_vm_unregister_notifier(struct notifier_block *nb); #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 5f260488e999..8a0e8bb02a5f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -101,6 +101,8 @@ DEFINE_MUTEX(kvm_lock); static DEFINE_RAW_SPINLOCK(kvm_count_lock); LIST_HEAD(vm_list); +static struct blocking_notifier_head kvm_vm_notifier; + static cpumask_var_t cpus_hardware_enabled; static int kvm_usage_count; static atomic_t hardware_enable_failed; @@ -148,12 +150,20 @@ static void kvm_io_bus_destroy(struct kvm_io_bus *bus); __visible bool kvm_rebooting; EXPORT_SYMBOL_GPL(kvm_rebooting); -#define KVM_EVENT_CREATE_VM 0 -#define KVM_EVENT_DESTROY_VM 1 static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm); static unsigned long long kvm_createvm_count; static unsigned long long kvm_active_vms; +inline int kvm_vm_register_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_register(&kvm_vm_notifier, nb); +} + +inline int kvm_vm_unregister_notifier(struct notifier_block *nb) +{ + return blocking_notifier_chain_unregister(&kvm_vm_notifier, nb); +} + __weak void kvm_arch_mmu_notifier_invalidate_range(struct kvm *kvm, unsigned long start, unsigned long end) { @@ -808,6 +818,8 @@ static struct kvm *kvm_create_vm(unsigned long type) preempt_notifier_inc(); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_CREATE_VM, kvm); return kvm; out_err: @@ -886,6 +898,8 @@ static void kvm_destroy_vm(struct kvm *kvm) preempt_notifier_dec(); hardware_disable_all(); mmdrop(mm); + blocking_notifier_call_chain(&kvm_vm_notifier, +KVM_EVENT_DESTROY_VM, kvm); } void kvm_get_kvm(struct kvm *kvm) @@ -4968,6 +4982,8 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, r = kvm_vfio_ops_init(); WARN_ON(r); + BLOCKING_INIT_NOTIFIER_HEAD(&kvm_vm_notifier); + return 0; out_unreg: -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC v2 0/3] Introduce Vdmabuf driver
The Virtual dmabuf or Virtio based dmabuf (Vdmabuf) driver can be used to "transfer" a page-backed dmabuf created in the Guest to the Host without making any copies. This is mostly accomplished by recreating the dmabuf on the Host using the PFNs and other meta-data shared by the guest. A use-case where this driver would be a good fit is a multi-GPU system (perhaps one discrete and one integrated) where one of the GPUs does not have access to the display/connectors/outputs. This could be an embedded system design decision or a restriction made at the firmware/BIOS level or perhaps the device is setup in UPT (Universal Passthrough) mode. When such a GPU is passthrough'd to a Guest OS, this driver can help in transferring the scanout buffer(s) (rendered using the native rendering stack) to the Host for the purpose of displaying them. The userspace component running in the Guest that transfers the dmabuf is referred to as the producer or exporter and its counterpart running in the Host is referred to as importer or consumer. For instance, a Wayland compositor would potentially be a producer and Qemu UI would be a consumer. It is the producer's responsibility to not reuse or destroy the shared buffer while it is still being used by the consumer. The consumer would send a release cmd indicating that it is done after which the shared buffer can be safely used again by the producer. One way the producer can prevent accidental re-use of the shared buffer is to lock the buffer when it exports it and unlock it after it gets a release cmd. As an example, the GBM API provides a simple way to lock and unlock a surface's buffers. For each dmabuf that is to be shared with the Host, a 128-bit unique ID is generated that identifies this buffer across the whole system. This ID is a combination of the Qemu process ID, a counter and a randomizer. We could potentially use UUID API but we currently use the above mentioned combination to identify the source of the buffer at any given time for bookkeeping. v2: - Added a notifier mechanism for getting the kvm pointer instead of sharing it via VFIO. - Added start and stop routines in the Vhost backend. - Augmented the cover letter and made some minor improvements. Vivek Kasireddy (3): kvm: Add a notifier for create and destroy VM events virtio: Introduce Vdmabuf driver vhost: Add Vdmabuf backend drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c | 1407 +++ drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 986 +++ include/linux/kvm_host.h|5 + include/linux/virtio_vdmabuf.h | 272 ++ include/uapi/linux/vhost.h |3 + include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 99 ++ virt/kvm/kvm_main.c | 20 +- 12 files changed, 2812 insertions(+), 2 deletions(-) create mode 100644 drivers/vhost/vdmabuf.c create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC 1/3] virtio: Introduce Vdmabuf driver
This driver "transfers" a dmabuf created on the Guest to the Host. A common use-case for such a transfer includes sharing the scanout buffer created by a display server or a compositor running in the Guest with Qemu UI -- running on the Host. The "transfer" is accomplished by sharing the PFNs of all the pages associated with the dmabuf and having a new dmabuf created on the Host that is backed up by the pages mapped from the Guest. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/virtio/Kconfig | 8 + drivers/virtio/Makefile | 1 + drivers/virtio/virtio_vdmabuf.c | 973 include/linux/virtio_vdmabuf.h | 271 include/uapi/linux/virtio_ids.h | 1 + include/uapi/linux/virtio_vdmabuf.h | 99 +++ 6 files changed, 1353 insertions(+) create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig index 7b41130d3f35..e563c12f711e 100644 --- a/drivers/virtio/Kconfig +++ b/drivers/virtio/Kconfig @@ -139,4 +139,12 @@ config VIRTIO_DMA_SHARED_BUFFER This option adds a flavor of dma buffers that are backed by virtio resources. +config VIRTIO_VDMABUF + bool "Enables Vdmabuf driver in guest os" + default n + depends on VIRTIO + help +This driver provides a way to share the dmabufs created in +the Guest with the Host. + endif # VIRTIO_MENU diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile index 591e6f72aa54..b4bb0738009c 100644 --- a/drivers/virtio/Makefile +++ b/drivers/virtio/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_VIRTIO_INPUT) += virtio_input.o obj-$(CONFIG_VIRTIO_VDPA) += virtio_vdpa.o obj-$(CONFIG_VIRTIO_MEM) += virtio_mem.o obj-$(CONFIG_VIRTIO_DMA_SHARED_BUFFER) += virtio_dma_buf.o +obj-$(CONFIG_VIRTIO_VDMABUF) += virtio_vdmabuf.o diff --git a/drivers/virtio/virtio_vdmabuf.c b/drivers/virtio/virtio_vdmabuf.c new file mode 100644 index ..e377114c2a2b --- /dev/null +++ b/drivers/virtio/virtio_vdmabuf.c @@ -0,0 +1,973 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + * Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define VIRTIO_VDMABUF_MAX_ID INT_MAX +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) +#define NEW_BUF_ID_GEN(vmid, cnt) (((vmid & 0x) << 32) | \ + ((cnt) & 0x)) + +/* one global drv object */ +static struct virtio_vdmabuf_info *drv_info; + +struct virtio_vdmabuf { + /* virtio device structure */ + struct virtio_device *vdev; + + /* virtual queue array */ + struct virtqueue *vq; + + /* ID of guest OS */ + u64 vmid; + + /* spin lock that needs to be acquired before accessing +* virtual queue +*/ + spinlock_t vq_lock; + struct mutex rx_lock; + + /* workqueue */ + struct workqueue_struct *wq; + struct work_struct rx_work; + struct virtio_vdmabuf_event_queue *evq; +}; + +static virtio_vdmabuf_buf_id_t get_buf_id(void) +{ + struct virtio_vdmabuf *vdmabuf = drv_info->priv; + virtio_vdmabuf_buf_id_t buf_id = {0, {0, 0} }; + static int count = 0; + + count = count < VIRTIO_VDMABUF_MAX_ID ? count + 1 : 0; + buf_id.id = NEW_BUF_ID_GEN(vdmabuf->vmid, count); + + /* random data embedded in the id for security */ + get_random_bytes(&buf_id
[RFC 2/3] vhost: Add Vdmabuf backend
This backend acts as the counterpart to the Vdmabuf Virtio frontend. When it receives a new export event from the frontend, it raises an event to alert the Qemu UI/userspace. Qemu then "imports" this buffer using the Unique ID. As part of the import step, a new dmabuf is created on the Host using the page information obtained from the Guest. The fd associated with this dmabuf is made available to Qemu UI/userspace which then creates a texture from it for the purpose of displaying it. Signed-off-by: Dongwon Kim Signed-off-by: Vivek Kasireddy --- drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c | 1332 +++ 3 files changed, 1344 insertions(+) create mode 100644 drivers/vhost/vdmabuf.c diff --git a/drivers/vhost/Kconfig b/drivers/vhost/Kconfig index 587fbae06182..1f1c51c4499e 100644 --- a/drivers/vhost/Kconfig +++ b/drivers/vhost/Kconfig @@ -89,4 +89,13 @@ config VHOST_CROSS_ENDIAN_LEGACY If unsure, say "N". +config VHOST_VDMABUF + bool "Vhost backend for the Vdmabuf driver" + depends on EVENTFD + select VHOST + default n + help + This driver works in pair with the Virtio Vdmabuf frontend. It can + be used to create a dmabuf using the pages shared by the Guest. + endif diff --git a/drivers/vhost/Makefile b/drivers/vhost/Makefile index f3e1897cce85..5c2cea4a7eaf 100644 --- a/drivers/vhost/Makefile +++ b/drivers/vhost/Makefile @@ -17,3 +17,6 @@ obj-$(CONFIG_VHOST) += vhost.o obj-$(CONFIG_VHOST_IOTLB) += vhost_iotlb.o vhost_iotlb-y := iotlb.o + +obj-$(CONFIG_VHOST_VDMABUF) += vhost_vdmabuf.o +vhost_vdmabuf-y := vdmabuf.o diff --git a/drivers/vhost/vdmabuf.c b/drivers/vhost/vdmabuf.c new file mode 100644 index ..7e2576fc2c0d --- /dev/null +++ b/drivers/vhost/vdmabuf.c @@ -0,0 +1,1332 @@ +// SPDX-License-Identifier: (MIT OR GPL-2.0) + +/* + * Copyright © 2021 Intel Corporation + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice (including the next + * paragraph) shall be included in all copies or substantial portions of the + * Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING + * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS + * IN THE SOFTWARE. + * + * Authors: + * Dongwon Kim + *Mateusz Polrola + *Vivek Kasireddy + * + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vhost.h" + +#define REFS_PER_PAGE (PAGE_SIZE/sizeof(long)) + +static struct virtio_vdmabuf_info *drv_info; + +struct kvm_instance { + struct kvm *kvm; + struct list_head link; +}; + +struct vhost_vdmabuf { + struct vhost_dev dev; + struct vhost_virtqueue vq; + struct vhost_work tx_work; + struct virtio_vdmabuf_event_queue *evq; + u64 vmid; + + /* synchronization between transmissions */ + struct mutex tx_mutex; + /* synchronization on tx and rx */ + struct mutex vq_mutex; + + struct virtio_vdmabuf_txmsg next; + struct list_head list; + struct kvm *kvm; +}; + +static inline void vhost_vdmabuf_add(struct vhost_vdmabuf *new) +{ + list_add_tail(&new->list, &drv_info->head_vdmabuf_list); +} + +static inline struct vhost_vdmabuf *vhost_vdmabuf_find(u64 vmid) +{ + struct vhost_vdmabuf *found; + + list_for_each_entry(found, &drv_info->head_vdmabuf_list, list) + if (found->vmid == vmid) + return found; + + return NULL; +} + +static inline bool vhost_vdmabuf_del(struct vhost_vdmabuf *vdmabuf) +{ + struct vhost_vdmabuf *iter, *temp; + + list_for_each_entry_safe(iter, temp, +&drv_info->head_vdmabuf_list, +list) + if (iter == vdmabuf) { + list_del(&iter->list); + return true; + } + + return false;
[RFC 3/3] vfio: Share the KVM instance with Vdmabuf
Getting a copy of the KVM instance is necessary for mapping Guest pages in the Host. TODO: Instead of invoking the symbol directly, there needs to be a better way of getting a copy of the KVM instance probably by using other notifiers. However, currently, KVM shares its instance only with VFIO and therefore we are compelled to bind the passthrough'd device to vfio-pci. Signed-off-by: Vivek Kasireddy --- drivers/vfio/vfio.c | 9 + 1 file changed, 9 insertions(+) diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c index 4ad8a35667a7..9fb11b1ad3cd 100644 --- a/drivers/vfio/vfio.c +++ b/drivers/vfio/vfio.c @@ -2213,11 +2213,20 @@ static int vfio_unregister_iommu_notifier(struct vfio_group *group, return ret; } +extern void vhost_vdmabuf_get_kvm(unsigned long action, void *data); void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm) { + void (*fn)(unsigned long, void *); + group->kvm = kvm; blocking_notifier_call_chain(&group->notifier, VFIO_GROUP_NOTIFY_SET_KVM, kvm); + + fn = symbol_get(vhost_vdmabuf_get_kvm); + if (fn) { + fn(VFIO_GROUP_NOTIFY_SET_KVM, kvm); + symbol_put(vhost_vdmabuf_get_kvm); + } } EXPORT_SYMBOL_GPL(vfio_group_set_kvm); -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC 0/3] Introduce Vdmabuf driver
The Virtual dmabuf or Virtio based dmabuf (Vdmabuf) driver can be used to "transfer" a page-backed dmabuf created in the Guest to the Host without making any copies. A use-case where this driver would be a good fit is a multi-GPU system (perhaps one discrete and one integrated) where one of the GPUs does not have access to the display/connectors/outputs. This could be a embedded system design decision or a restriction made at the firmware/BIOS level. When such a GPU is passthrough'd to a Guest OS, this driver can help in transferring the scanout buffer(s) (rendered using the native rendering stack) to the Host for the purpose of displaying them. The userspace component running in the Guest that transfers the dmabuf is referred to as the producer or exporter and its counterpart running in the Host is referred to as importer or consumer. For instance, a Wayland compositor would potentially be a producer and Qemu UI would be a consumer. It is the producer's responsibility to not reuse or destroy the shared buffer while it is still being used by the consumer. The consumer would send a release cmd indicating that it is done after which the shared buffer can be safely used again by the producer. One way the producer can prevent accidental re-use of the shared buffer is to lock the buffer when it exports it and unlock it after it gets a release cmd. As an example, the GBM API provides a simple way to lock and unlock a surface's buffers. For each dmabuf that is to be shared with the Host, a 128-bit unique ID is generated that identifies this buffer across the whole system. This ID is a combination of the Qemu process ID, a counter and a randomizer. We could potentially use UUID API but we currently use the above mentioned combination to identify the source of the buffer at any given time for bookkeeping. Vivek Kasireddy (3): virtio: Introduce Vdmabuf driver vhost: Add Vdmabuf backend vfio: Share the KVM instance with Vdmabuf drivers/vfio/vfio.c |9 + drivers/vhost/Kconfig |9 + drivers/vhost/Makefile |3 + drivers/vhost/vdmabuf.c | 1332 +++ drivers/virtio/Kconfig |8 + drivers/virtio/Makefile |1 + drivers/virtio/virtio_vdmabuf.c | 973 +++ include/linux/virtio_vdmabuf.h | 271 ++ include/uapi/linux/virtio_ids.h |1 + include/uapi/linux/virtio_vdmabuf.h | 99 ++ 10 files changed, 2706 insertions(+) create mode 100644 drivers/vhost/vdmabuf.c create mode 100644 drivers/virtio/virtio_vdmabuf.c create mode 100644 include/linux/virtio_vdmabuf.h create mode 100644 include/uapi/linux/virtio_vdmabuf.h -- 2.26.2 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization