Re: [RFC v2 PATCH] vdpa_sim: implement .reset_map support
On 10/19/2023 2:29 AM, Stefano Garzarella wrote: On Wed, Oct 18, 2023 at 04:47:48PM -0700, Si-Wei Liu wrote: On 10/18/2023 1:05 AM, Stefano Garzarella wrote: On Tue, Oct 17, 2023 at 10:11:33PM -0700, Si-Wei Liu wrote: RFC only. Not tested on vdpa-sim-blk with user virtual address. Works fine with vdpa-sim-net which uses physical address to map. This patch is based on top of [1]. [1] https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei@oracle.com/ Signed-off-by: Si-Wei Liu --- RFC v2: - initialize iotlb to passthrough mode in device add I tested this version and I didn't see any issue ;-) Great, thank you so much for your help on testing my patch, Stefano! You're welcome :-) Just for my own interest/curiosity, currently there's no vhost-vdpa backend client implemented for vdpa-sim-blk Yep, we developed libblkio [1]. libblkio exposes common API to access block devices in userspace. It supports several drivers. The one useful for this use case is `virtio-blk-vhost-vdpa`. Here [2] some examples on how to use the libblkio test suite with the vdpa-sim-blk. Since QEMU 7.2, it supports libblkio drivers, so you can use the following options to attach a vdpa-blk device to a VM: -blockdev node-name=drive_src1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \ -device virtio-blk-pci,id=src1,bootindex=2,drive=drive_src1 \ For now only what we called slow-path [3][4] is supported, since the VQs are not directly exposed to the guest, but QEMU allocates other VQs (similar to shadow VQs for net) to support live-migration and QEMU storage features. Fast-path is on the agenda, but on pause for now. or any vdpa block device in userspace as yet, correct? Do you mean with VDUSE? In this case, yes, qemu-storage-daemon supports it, and can implement a virtio-blk in user space, exposing a disk image thorough VDUSE. There is an example in libblkio as well [5] on how to start it. So there was no test specific to vhost-vdpa that needs to be exercised, right? I hope I answered above :-) Definitely! This is exactly what I needed, it's really useful! Much appreciated for the detailed information! I hadn't been aware of the latest status on libblkio drivers and qemu support since I last checked it (it was at some point right after KVM 2022, sorry my knowledge too outdated). I followed your links below and checked a few things, looks my change shouldn't affect anything. Good to see all the desired pieces landed to QEMU and libblkio already as planned, great job done! Cheers, -Siwei This reminded me that I need to write a blog post with all this information, I hope to do that soon! Stefano [1] https://gitlab.com/libblkio/libblkio [2] https://gitlab.com/libblkio/libblkio/-/blob/main/tests/meson.build?ref_type=heads#L42 [3] https://kvmforum2022.sched.com/event/15jK5/qemu-storage-daemon-and-libblkio-exploring-new-shores-for-the-qemu-block-layer-kevin-wolf-stefano-garzarella-red-hat [4] https://kvmforum2021.sched.com/event/ke3a/vdpa-blk-unified-hardware-and-software-offload-for-virtio-blk-stefano-garzarella-red-hat [5] https://gitlab.com/libblkio/libblkio/-/blob/main/tests/meson.build?ref_type=heads#L58 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC v2 PATCH] vdpa_sim: implement .reset_map support
On Wed, Oct 18, 2023 at 04:47:48PM -0700, Si-Wei Liu wrote: On 10/18/2023 1:05 AM, Stefano Garzarella wrote: On Tue, Oct 17, 2023 at 10:11:33PM -0700, Si-Wei Liu wrote: RFC only. Not tested on vdpa-sim-blk with user virtual address. Works fine with vdpa-sim-net which uses physical address to map. This patch is based on top of [1]. [1] https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei@oracle.com/ Signed-off-by: Si-Wei Liu --- RFC v2: - initialize iotlb to passthrough mode in device add I tested this version and I didn't see any issue ;-) Great, thank you so much for your help on testing my patch, Stefano! You're welcome :-) Just for my own interest/curiosity, currently there's no vhost-vdpa backend client implemented for vdpa-sim-blk Yep, we developed libblkio [1]. libblkio exposes common API to access block devices in userspace. It supports several drivers. The one useful for this use case is `virtio-blk-vhost-vdpa`. Here [2] some examples on how to use the libblkio test suite with the vdpa-sim-blk. Since QEMU 7.2, it supports libblkio drivers, so you can use the following options to attach a vdpa-blk device to a VM: -blockdev node-name=drive_src1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \ -device virtio-blk-pci,id=src1,bootindex=2,drive=drive_src1 \ For now only what we called slow-path [3][4] is supported, since the VQs are not directly exposed to the guest, but QEMU allocates other VQs (similar to shadow VQs for net) to support live-migration and QEMU storage features. Fast-path is on the agenda, but on pause for now. or any vdpa block device in userspace as yet, correct? Do you mean with VDUSE? In this case, yes, qemu-storage-daemon supports it, and can implement a virtio-blk in user space, exposing a disk image thorough VDUSE. There is an example in libblkio as well [5] on how to start it. So there was no test specific to vhost-vdpa that needs to be exercised, right? I hope I answered above :-) This reminded me that I need to write a blog post with all this information, I hope to do that soon! Stefano [1] https://gitlab.com/libblkio/libblkio [2] https://gitlab.com/libblkio/libblkio/-/blob/main/tests/meson.build?ref_type=heads#L42 [3] https://kvmforum2022.sched.com/event/15jK5/qemu-storage-daemon-and-libblkio-exploring-new-shores-for-the-qemu-block-layer-kevin-wolf-stefano-garzarella-red-hat [4] https://kvmforum2021.sched.com/event/ke3a/vdpa-blk-unified-hardware-and-software-offload-for-virtio-blk-stefano-garzarella-red-hat [5] https://gitlab.com/libblkio/libblkio/-/blob/main/tests/meson.build?ref_type=heads#L58 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC v2 PATCH] vdpa_sim: implement .reset_map support
On 10/18/2023 1:05 AM, Stefano Garzarella wrote:
On Tue, Oct 17, 2023 at 10:11:33PM -0700, Si-Wei Liu wrote:
RFC only. Not tested on vdpa-sim-blk with user virtual address.
Works fine with vdpa-sim-net which uses physical address to map.
This patch is based on top of [1].
[1]
https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei@oracle.com/
Signed-off-by: Si-Wei Liu
---
RFC v2:
- initialize iotlb to passthrough mode in device add
I tested this version and I didn't see any issue ;-)
Great, thank you so much for your help on testing my patch, Stefano!
Just for my own interest/curiosity, currently there's no vhost-vdpa
backend client implemented for vdpa-sim-blk or any vdpa block device in
userspace as yet, correct? So there was no test specific to vhost-vdpa
that needs to be exercised, right?
Thanks,
-Siwei
Tested-by: Stefano Garzarella
---
drivers/vdpa/vdpa_sim/vdpa_sim.c | 34
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c
b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 76d41058add9..2a0a6042d61d 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -151,13 +151,6 @@ static void vdpasim_do_reset(struct vdpasim
*vdpasim)
>iommu_lock);
}
- for (i = 0; i < vdpasim->dev_attr.nas; i++) {
- vhost_iotlb_reset(>iommu[i]);
- vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX,
- 0, VHOST_MAP_RW);
- vdpasim->iommu_pt[i] = true;
- }
-
vdpasim->running = true;
spin_unlock(>iommu_lock);
@@ -259,8 +252,12 @@ struct vdpasim *vdpasim_create(struct
vdpasim_dev_attr *dev_attr,
if (!vdpasim->iommu_pt)
goto err_iommu;
- for (i = 0; i < vdpasim->dev_attr.nas; i++)
+ for (i = 0; i < vdpasim->dev_attr.nas; i++) {
vhost_iotlb_init(>iommu[i], max_iotlb_entries, 0);
+ vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX, 0,
+ VHOST_MAP_RW);
+ vdpasim->iommu_pt[i] = true;
+ }
for (i = 0; i < dev_attr->nvqs; i++)
vringh_set_iotlb(>vqs[i].vring, >iommu[0],
@@ -637,6 +634,25 @@ static int vdpasim_set_map(struct vdpa_device
*vdpa, unsigned int asid,
return ret;
}
+static int vdpasim_reset_map(struct vdpa_device *vdpa, unsigned int
asid)
+{
+ struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+
+ if (asid >= vdpasim->dev_attr.nas)
+ return -EINVAL;
+
+ spin_lock(>iommu_lock);
+ if (vdpasim->iommu_pt[asid])
+ goto out;
+ vhost_iotlb_reset(>iommu[asid]);
+ vhost_iotlb_add_range(>iommu[asid], 0, ULONG_MAX,
+ 0, VHOST_MAP_RW);
+ vdpasim->iommu_pt[asid] = true;
+out:
+ spin_unlock(>iommu_lock);
+ return 0;
+}
+
static int vdpasim_bind_mm(struct vdpa_device *vdpa, struct mm_struct
*mm)
{
struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -759,6 +775,7 @@ static const struct vdpa_config_ops
vdpasim_config_ops = {
.set_group_asid = vdpasim_set_group_asid,
.dma_map = vdpasim_dma_map,
.dma_unmap = vdpasim_dma_unmap,
+ .reset_map = vdpasim_reset_map,
.bind_mm = vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
@@ -796,6 +813,7 @@ static const struct vdpa_config_ops
vdpasim_batch_config_ops = {
.get_iova_range = vdpasim_get_iova_range,
.set_group_asid = vdpasim_set_group_asid,
.set_map = vdpasim_set_map,
+ .reset_map = vdpasim_reset_map,
.bind_mm = vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC v2 PATCH] vdpa_sim: implement .reset_map support
On Tue, Oct 17, 2023 at 10:11:33PM -0700, Si-Wei Liu wrote:
RFC only. Not tested on vdpa-sim-blk with user virtual address.
Works fine with vdpa-sim-net which uses physical address to map.
This patch is based on top of [1].
[1]
https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei@oracle.com/
Signed-off-by: Si-Wei Liu
---
RFC v2:
- initialize iotlb to passthrough mode in device add
I tested this version and I didn't see any issue ;-)
Tested-by: Stefano Garzarella
---
drivers/vdpa/vdpa_sim/vdpa_sim.c | 34
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 76d41058add9..2a0a6042d61d 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -151,13 +151,6 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
>iommu_lock);
}
- for (i = 0; i < vdpasim->dev_attr.nas; i++) {
- vhost_iotlb_reset(>iommu[i]);
- vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX,
- 0, VHOST_MAP_RW);
- vdpasim->iommu_pt[i] = true;
- }
-
vdpasim->running = true;
spin_unlock(>iommu_lock);
@@ -259,8 +252,12 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr
*dev_attr,
if (!vdpasim->iommu_pt)
goto err_iommu;
- for (i = 0; i < vdpasim->dev_attr.nas; i++)
+ for (i = 0; i < vdpasim->dev_attr.nas; i++) {
vhost_iotlb_init(>iommu[i], max_iotlb_entries, 0);
+ vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX, 0,
+ VHOST_MAP_RW);
+ vdpasim->iommu_pt[i] = true;
+ }
for (i = 0; i < dev_attr->nvqs; i++)
vringh_set_iotlb(>vqs[i].vring, >iommu[0],
@@ -637,6 +634,25 @@ static int vdpasim_set_map(struct vdpa_device *vdpa,
unsigned int asid,
return ret;
}
+static int vdpasim_reset_map(struct vdpa_device *vdpa, unsigned int asid)
+{
+ struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+
+ if (asid >= vdpasim->dev_attr.nas)
+ return -EINVAL;
+
+ spin_lock(>iommu_lock);
+ if (vdpasim->iommu_pt[asid])
+ goto out;
+ vhost_iotlb_reset(>iommu[asid]);
+ vhost_iotlb_add_range(>iommu[asid], 0, ULONG_MAX,
+ 0, VHOST_MAP_RW);
+ vdpasim->iommu_pt[asid] = true;
+out:
+ spin_unlock(>iommu_lock);
+ return 0;
+}
+
static int vdpasim_bind_mm(struct vdpa_device *vdpa, struct mm_struct *mm)
{
struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -759,6 +775,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
.set_group_asid = vdpasim_set_group_asid,
.dma_map= vdpasim_dma_map,
.dma_unmap = vdpasim_dma_unmap,
+ .reset_map = vdpasim_reset_map,
.bind_mm= vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
@@ -796,6 +813,7 @@ static const struct vdpa_config_ops
vdpasim_batch_config_ops = {
.get_iova_range = vdpasim_get_iova_range,
.set_group_asid = vdpasim_set_group_asid,
.set_map= vdpasim_set_map,
+ .reset_map = vdpasim_reset_map,
.bind_mm= vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC v2 PATCH] vdpa_sim: implement .reset_map support
RFC only. Not tested on vdpa-sim-blk with user virtual address.
Works fine with vdpa-sim-net which uses physical address to map.
This patch is based on top of [1].
[1]
https://lore.kernel.org/virtualization/1696928580-7520-1-git-send-email-si-wei@oracle.com/
Signed-off-by: Si-Wei Liu
---
RFC v2:
- initialize iotlb to passthrough mode in device add
---
drivers/vdpa/vdpa_sim/vdpa_sim.c | 34
1 file changed, 26 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 76d41058add9..2a0a6042d61d 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -151,13 +151,6 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
>iommu_lock);
}
- for (i = 0; i < vdpasim->dev_attr.nas; i++) {
- vhost_iotlb_reset(>iommu[i]);
- vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX,
- 0, VHOST_MAP_RW);
- vdpasim->iommu_pt[i] = true;
- }
-
vdpasim->running = true;
spin_unlock(>iommu_lock);
@@ -259,8 +252,12 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr
*dev_attr,
if (!vdpasim->iommu_pt)
goto err_iommu;
- for (i = 0; i < vdpasim->dev_attr.nas; i++)
+ for (i = 0; i < vdpasim->dev_attr.nas; i++) {
vhost_iotlb_init(>iommu[i], max_iotlb_entries, 0);
+ vhost_iotlb_add_range(>iommu[i], 0, ULONG_MAX, 0,
+ VHOST_MAP_RW);
+ vdpasim->iommu_pt[i] = true;
+ }
for (i = 0; i < dev_attr->nvqs; i++)
vringh_set_iotlb(>vqs[i].vring, >iommu[0],
@@ -637,6 +634,25 @@ static int vdpasim_set_map(struct vdpa_device *vdpa,
unsigned int asid,
return ret;
}
+static int vdpasim_reset_map(struct vdpa_device *vdpa, unsigned int asid)
+{
+ struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+
+ if (asid >= vdpasim->dev_attr.nas)
+ return -EINVAL;
+
+ spin_lock(>iommu_lock);
+ if (vdpasim->iommu_pt[asid])
+ goto out;
+ vhost_iotlb_reset(>iommu[asid]);
+ vhost_iotlb_add_range(>iommu[asid], 0, ULONG_MAX,
+ 0, VHOST_MAP_RW);
+ vdpasim->iommu_pt[asid] = true;
+out:
+ spin_unlock(>iommu_lock);
+ return 0;
+}
+
static int vdpasim_bind_mm(struct vdpa_device *vdpa, struct mm_struct *mm)
{
struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -759,6 +775,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
.set_group_asid = vdpasim_set_group_asid,
.dma_map= vdpasim_dma_map,
.dma_unmap = vdpasim_dma_unmap,
+ .reset_map = vdpasim_reset_map,
.bind_mm= vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
@@ -796,6 +813,7 @@ static const struct vdpa_config_ops
vdpasim_batch_config_ops = {
.get_iova_range = vdpasim_get_iova_range,
.set_group_asid = vdpasim_set_group_asid,
.set_map= vdpasim_set_map,
+ .reset_map = vdpasim_reset_map,
.bind_mm= vdpasim_bind_mm,
.unbind_mm = vdpasim_unbind_mm,
.free = vdpasim_free,
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
