Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device
On 10/16/2023 4:52 PM, Michael S. Tsirkin wrote: On Mon, Oct 16, 2023 at 04:33:10PM +0800, Zhu, Lingshan wrote: On 10/13/2023 9:50 PM, Michael S. Tsirkin wrote: On Fri, Oct 13, 2023 at 06:28:34PM +0800, Zhu, Lingshan wrote: On 10/12/2023 9:27 PM, Jason Gunthorpe wrote: On Thu, Oct 12, 2023 at 06:29:47PM +0800, Zhu, Lingshan wrote: sorry for the late reply, we have discussed this for weeks in virtio mailing list. I have proposed a live migration solution which is a config space solution. I'm sorry that can't be a serious proposal - config space can't do DMA, it is not suitable. config space only controls the live migration process and config the related facilities. We don't use config space to transfer data. The new added registers work like queue_enable or features. For example, we use DMA to report dirty pages and MMIO to fetch the dirty data. I remember in another thread you said:"you can't use DMA for any migration flows" And I agree to that statement, so we use config space registers to control the flow. Thanks, Zhu Lingshan Jason If you are using dma then I don't see what's wrong with admin vq. dma is all it does. dma != admin vq, Well they share the same issue that they don't work for nesting because DMA can not be intercepted. (hope this is not a spam to virtualization list and I try to keep this short) only use dma for host memory access, e.g., dirty page bitmap, no need to intercepted. and I think we have discussed many details in pros and cons in admin vq live migration proposal in virtio-comment. I am not sure we should span the discussions here, repeat them over again. Thanks Yea let's not. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device
On 10/13/2023 9:50 PM, Michael S. Tsirkin wrote: On Fri, Oct 13, 2023 at 06:28:34PM +0800, Zhu, Lingshan wrote: On 10/12/2023 9:27 PM, Jason Gunthorpe wrote: On Thu, Oct 12, 2023 at 06:29:47PM +0800, Zhu, Lingshan wrote: sorry for the late reply, we have discussed this for weeks in virtio mailing list. I have proposed a live migration solution which is a config space solution. I'm sorry that can't be a serious proposal - config space can't do DMA, it is not suitable. config space only controls the live migration process and config the related facilities. We don't use config space to transfer data. The new added registers work like queue_enable or features. For example, we use DMA to report dirty pages and MMIO to fetch the dirty data. I remember in another thread you said:"you can't use DMA for any migration flows" And I agree to that statement, so we use config space registers to control the flow. Thanks, Zhu Lingshan Jason If you are using dma then I don't see what's wrong with admin vq. dma is all it does. dma != admin vq, and I think we have discussed many details in pros and cons in admin vq live migration proposal in virtio-comment. I am not sure we should span the discussions here, repeat them over again. Thanks ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device
On 10/12/2023 9:27 PM, Jason Gunthorpe wrote: On Thu, Oct 12, 2023 at 06:29:47PM +0800, Zhu, Lingshan wrote: sorry for the late reply, we have discussed this for weeks in virtio mailing list. I have proposed a live migration solution which is a config space solution. I'm sorry that can't be a serious proposal - config space can't do DMA, it is not suitable. config space only controls the live migration process and config the related facilities. We don't use config space to transfer data. The new added registers work like queue_enable or features. For example, we use DMA to report dirty pages and MMIO to fetch the dirty data. I remember in another thread you said:"you can't use DMA for any migration flows" And I agree to that statement, so we use config space registers to control the flow. Thanks, Zhu Lingshan Jason ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device
On 10/11/2023 2:59 PM, Christoph Hellwig wrote: On Wed, Oct 11, 2023 at 02:43:37AM -0400, Michael S. Tsirkin wrote: Btw, what is that intel thing everyone is talking about? And why would the virtio core support vendor specific behavior like that? It's not a thing it's Zhu Lingshan :) intel is just one of the vendors that implemented vdpa support and so Zhu Lingshan from intel is working on vdpa and has also proposed virtio spec extensions for migration. intel's driver is called ifcvf. vdpa composes all this stuff that is added to vfio in userspace, so it's a different approach. Well, so let's call it virtio live migration instead of intel. And please work all together in the virtio committee that you have one way of communication between controlling and controlled functions. If one extension does it one way and the other a different way that's just creating a giant mess. I hope so, Jason Wang has proposed a solution to cooperate, but sadly rejected... ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 10/11] vfio/virtio: Expose admin commands over virtio device
On 10/11/2023 4:00 PM, Parav Pandit via Virtualization wrote: Hi Christoph, From: Christoph Hellwig Sent: Wednesday, October 11, 2023 12:29 PM On Wed, Oct 11, 2023 at 02:43:37AM -0400, Michael S. Tsirkin wrote: Btw, what is that intel thing everyone is talking about? And why would the virtio core support vendor specific behavior like that? It's not a thing it's Zhu Lingshan :) intel is just one of the vendors that implemented vdpa support and so Zhu Lingshan from intel is working on vdpa and has also proposed virtio spec extensions for migration. intel's driver is called ifcvf. vdpa composes all this stuff that is added to vfio in userspace, so it's a different approach. Well, so let's call it virtio live migration instead of intel. And please work all together in the virtio committee that you have one way of communication between controlling and controlled functions. If one extension does it one way and the other a different way that's just creating a giant mess. We in virtio committee are working on VF device migration where: VF = controlled function PF = controlling function The second proposal is what Michael mentioned from Intel that somehow combine controlled and controlling function as single entity on VF. The main reasons I find it weird are: 1. it must always need to do mediation to do fake the device reset, and flr flows 2. dma cannot work as you explained for complex device state 3. it needs constant knowledge of each tiny things for each virtio device type Such single entity appears a bit very weird to me but maybe it is just me. sorry for the late reply, we have discussed this for weeks in virtio mailing list. I have proposed a live migration solution which is a config space solution. We(me, Jason and Eugenio) have been working on this solution for more than two years and we are implementing virtio live migration basic facilities. The implementation is transport specific, e.g., for PCI we implement new or extend registers which work as other config space registers do. The reason we are arguing is: I am not sure admin vq based live migration solution is a good choice, because: 1) it does not work for nested 2) it does not work for bare metal 3) QOS problem 4) security leaks. Sorry to span the discussions here. Thanks, Zhu Lingshan ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 11/11] vfio/virtio: Introduce a vfio driver over virtio devices
On 9/26/2023 2:36 AM, Michael S. Tsirkin wrote:
On Mon, Sep 25, 2023 at 08:26:33AM +, Parav Pandit wrote:
From: Jason Wang
Sent: Monday, September 25, 2023 8:00 AM
On Fri, Sep 22, 2023 at 8:25 PM Parav Pandit wrote:
From: Jason Gunthorpe
Sent: Friday, September 22, 2023 5:53 PM
And what's more, using MMIO BAR0 then it can work for legacy.
Oh? How? Our team didn't think so.
It does not. It was already discussed.
The device reset in legacy is not synchronous.
How do you know this?
Not sure the motivation of same discussion done in the OASIS with you and
others in past.
Anyways, please find the answer below.
About reset,
The legacy device specification has not enforced below cited 1.0 driver
requirement of 1.0.
"The driver SHOULD consider a driver-initiated reset complete when it reads device
status as 0."
[1] https://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf
Basically, I think any drivers that did not read status (linux pre 2011)
before freeing memory under DMA have a reset path that is racy wrt DMA, since
memory writes are posted and IO writes while not posted have completion
that does not order posted transactions, e.g. from pci express spec:
D2b
An I/O or Configuration Write Completion 37 is permitted to pass a
Posted Request.
having said that there were a ton of driver races discovered on this
path in the years since, I suspect if one cares about this then
just avoiding stress on reset is wise.
The drivers do not wait for reset to complete; it was written for the sw
backend.
Do you see there's a flush after reset in the legacy driver?
Yes. it only flushes the write by reading it. The driver does not get _wait_
for the reset to complete within the device like above.
One can thinkably do that wait in hardware, though. Just defer completion until
read is done.
I agree with MST. At least Intel devices work fine with vfio-pci and
legacy driver without any changes.
So far so good.
Thanks
Zhu Lingshan
Please see the reset flow of 1.x device as below.
In fact the comment of the 1.x device also needs to be updated to indicate that
driver need to wait for the device to finish the reset.
I will send separate patch for improving this comment of vp_reset() to match
the spec.
static void vp_reset(struct virtio_device *vdev)
{
struct virtio_pci_device *vp_dev = to_vp_device(vdev);
struct virtio_pci_modern_device *mdev = &vp_dev->mdev;
/* 0 status means a reset. */
vp_modern_set_status(mdev, 0);
/* After writing 0 to device_status, the driver MUST wait for a read of
* device_status to return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);
/* Flush pending VQ/configuration callbacks. */
vp_synchronize_vectors(vdev);
}
static void vp_reset(struct virtio_device *vdev) {
struct virtio_pci_device *vp_dev = to_vp_device(vdev);
/* 0 status means a reset. */
vp_legacy_set_status(&vp_dev->ldev, 0);
/* Flush out the status write, and flush in device writes,
* including MSi-X interrupts, if any. */
vp_legacy_get_status(&vp_dev->ldev);
/* Flush pending VQ/configuration callbacks. */
vp_synchronize_vectors(vdev);
}
Thanks
Hence MMIO BAR0 is not the best option in real implementations.
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 11/11] vfio/virtio: Introduce a vfio driver over virtio devices
On 9/22/2023 4:55 AM, Michael S. Tsirkin wrote: On Thu, Sep 21, 2023 at 04:51:15PM -0300, Jason Gunthorpe wrote: On Thu, Sep 21, 2023 at 03:17:25PM -0400, Michael S. Tsirkin wrote: On Thu, Sep 21, 2023 at 03:39:26PM -0300, Jason Gunthorpe wrote: What is the huge amount of work am I asking to do? You are asking us to invest in the complexity of VDPA through out (keep it working, keep it secure, invest time in deploying and debugging in the field) I'm asking you to do nothing of the kind - I am saying that this code will have to be duplicated in vdpa, Why would that be needed? For the same reason it was developed in the 1st place - presumably because it adds efficient legacy guest support with the right card? I get it, you specifically don't need VDPA functionality, but I don't see why is this universal, or common. and so I am asking what exactly is missing to just keep it all there. VFIO. Seriously, we don't want unnecessary mediation in this path at all. But which mediation is necessary is exactly up to the specific use-case. I have no idea why would you want all of VFIO to e.g. pass access to random config registers to the guest when it's a virtio device and the config registers are all nicely listed in the spec. I know nvidia hardware is so great, it has super robust cards with less security holes than the vdpa driver, but I very much doubt this is universal for all virtio offload cards. I agree with MST. note I didn't ask you to add iommufd to vdpa though that would be nice ;) I did once send someone to look.. It didn't succeed :( Jason Pity. Maybe there's some big difficulty blocking this? I'd like to know. ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH vfio 11/11] vfio/virtio: Introduce a vfio driver over virtio devices
On 9/22/2023 2:39 AM, Jason Gunthorpe wrote: On Thu, Sep 21, 2023 at 12:53:04PM -0400, Michael S. Tsirkin wrote: vdpa is not vfio, I don't know how you can suggest vdpa is a replacement for a vfio driver. They are completely different things. Each side has its own strengths, and vfio especially is accelerating in its capability in way that vpda is not. eg if an iommufd conversion had been done by now for vdpa I might be more sympathetic. Yea, I agree iommufd is a big problem with vdpa right now. Cindy was sick and I didn't know and kept assuming she's working on this. I don't think it's a huge amount of work though. I'll take a look. Is there anything else though? Do tell. Confidential compute will never work with VDPA's approach. I don't understand why vDPA can not and will never support Confidential computing? Do you see any blockers? There are a bunch of things that I think are important for virtio that are completely out of scope for vfio, such as migrating cross-vendor. VFIO supports migration, if you want to have cross-vendor migration then make a standard that describes the VFIO migration data format for virtio devices. What is the huge amount of work am I asking to do? You are asking us to invest in the complexity of VDPA through out (keep it working, keep it secure, invest time in deploying and debugging in the field) When it doesn't provide *ANY* value to the solution. The starting point is a completely working vfio PCI function and the end goal is to put that function into a VM. That is VFIO, not VDPA. VPDA is fine for what it does, but it is not a reasonable replacement for VFIO. Jason ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] vp_vdpa: synchronize irq when free irq
On 6/30/2023 4:02 PM, Jason Wang wrote:
On Fri, Jun 30, 2023 at 10:37 AM Zhu Lingshan wrote:
This commits synchronizes irqs when free them
Need to explain why it is needed.
Signed-off-by: Zhu Lingshan
Tested-by: Cindy Lu
---
drivers/vdpa/virtio_pci/vp_vdpa.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c
b/drivers/vdpa/virtio_pci/vp_vdpa.c
index 281287fae89f..d8ee3e68cd2d 100644
--- a/drivers/vdpa/virtio_pci/vp_vdpa.c
+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
@@ -116,6 +116,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
for (i = 0; i < vp_vdpa->queues; i++) {
if (vp_vdpa->vring[i].irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_queue_vector(mdev, i, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->vring[i].irq);
Interestingly, we don't do this in virtio_pci_common.c and we only
synchronize during reset there.
Any reason makes vp_vdpa different?
As Michael point out, free_irq calls synchronize_irq internally.
And this vp_vdpa_free irq is called in the reset routine.
So I think we don't need to sync it here.
Thanks
devm_free_irq(&pdev->dev, vp_vdpa->vring[i].irq,
&vp_vdpa->vring[i]);
vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
@@ -124,6 +125,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
if (vp_vdpa->config_irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_config_vector(mdev, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->config_irq);
devm_free_irq(&pdev->dev, vp_vdpa->config_irq, vp_vdpa);
vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
}
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH] vp_vdpa: synchronize irq when free irq
On 6/30/2023 4:34 PM, Michael S. Tsirkin wrote:
On Fri, Jun 30, 2023 at 06:36:56PM +0800, Zhu Lingshan wrote:
This commits synchronizes irqs when free them
Signed-off-by: Zhu Lingshan
Tested-by: Cindy Lu
---
drivers/vdpa/virtio_pci/vp_vdpa.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c
b/drivers/vdpa/virtio_pci/vp_vdpa.c
index 281287fae89f..d8ee3e68cd2d 100644
--- a/drivers/vdpa/virtio_pci/vp_vdpa.c
+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
@@ -116,6 +116,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
for (i = 0; i < vp_vdpa->queues; i++) {
if (vp_vdpa->vring[i].irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_queue_vector(mdev, i, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->vring[i].irq);
devm_free_irq(&pdev->dev, vp_vdpa->vring[i].irq,
&vp_vdpa->vring[i]);
vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
@@ -124,6 +125,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
if (vp_vdpa->config_irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_config_vector(mdev, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->config_irq);
devm_free_irq(&pdev->dev, vp_vdpa->config_irq, vp_vdpa);
vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
Weird, doesn't free_irq synchronize automatically?
Oh, I see free_irq calls __syncronize_hardirq internally
Thanks
}
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH] vp_vdpa: synchronize irq when free irq
This commits synchronizes irqs when free them
Signed-off-by: Zhu Lingshan
Tested-by: Cindy Lu
---
drivers/vdpa/virtio_pci/vp_vdpa.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c
b/drivers/vdpa/virtio_pci/vp_vdpa.c
index 281287fae89f..d8ee3e68cd2d 100644
--- a/drivers/vdpa/virtio_pci/vp_vdpa.c
+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
@@ -116,6 +116,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
for (i = 0; i < vp_vdpa->queues; i++) {
if (vp_vdpa->vring[i].irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_queue_vector(mdev, i, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->vring[i].irq);
devm_free_irq(&pdev->dev, vp_vdpa->vring[i].irq,
&vp_vdpa->vring[i]);
vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
@@ -124,6 +125,7 @@ static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
if (vp_vdpa->config_irq != VIRTIO_MSI_NO_VECTOR) {
vp_modern_config_vector(mdev, VIRTIO_MSI_NO_VECTOR);
+ synchronize_irq(vp_vdpa->config_irq);
devm_free_irq(&pdev->dev, vp_vdpa->config_irq, vp_vdpa);
vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
}
--
2.39.3
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 1/3] vDPA/ifcvf: dynamic allocate vq data stores
On 6/26/2023 10:49 AM, Jason Wang wrote:
On Mon, Jun 26, 2023 at 10:38 AM Zhu, Lingshan wrote:
On 6/26/2023 10:32 AM, Jason Wang wrote:
On Mon, Jun 12, 2023 at 3:14 PM Zhu Lingshan wrote:
This commit dynamically allocates the data
stores for the virtqueues based on
virtio_pci_common_cfg.num_queues.
While at it, it's better to allocate vring_lm_cfg as well and drop
IFCVF_MAX_QUEUES.
Yes, this has been done in 3/3 patch in this series.
Ok, yes, but it seems patch 3 implements a lot of logic so I suggest
moving it to patch 1.
Not sure it's too late since I see the patch has been merged by Michael.
I am not sure, I think we need two patches handle their respective
changes anyway,
and can not drop IFCVF_MAX_QUEUES in the first one.
Thanks
Thanks
Zhu Lingshan
Thanks
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 3 +++
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 2 ++
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 1b5da11f5403..f86495ace825 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -134,6 +134,9 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
}
hw->nr_vring = vp_ioread16(&hw->common_cfg->num_queues);
+ hw->vring = kzalloc(sizeof(struct vring_info) * hw->nr_vring,
GFP_KERNEL);
+ if (!hw->vring)
+ return -ENOMEM;
for (i = 0; i < hw->nr_vring; i++) {
vp_iowrite16(i, &hw->common_cfg->queue_select);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 3110ffc50caf..fa797184056b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -74,7 +74,7 @@ struct ifcvf_hw {
u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
- struct vring_info vring[IFCVF_MAX_QUEUES];
+ struct vring_info *vring;
void __iomem * const *base;
char config_msix_name[256];
struct vdpa_callback config_cb;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 6e47ac2c669a..2af0de771b49 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -830,6 +830,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
return 0;
err:
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
return ret;
}
@@ -840,6 +841,7 @@ static void ifcvf_remove(struct pci_dev *pdev)
ifcvf_mgmt_dev = pci_get_drvdata(pdev);
vdpa_mgmtdev_unregister(&ifcvf_mgmt_dev->mdev);
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 1/3] vDPA/ifcvf: dynamic allocate vq data stores
On 6/26/2023 10:32 AM, Jason Wang wrote:
On Mon, Jun 12, 2023 at 3:14 PM Zhu Lingshan wrote:
This commit dynamically allocates the data
stores for the virtqueues based on
virtio_pci_common_cfg.num_queues.
While at it, it's better to allocate vring_lm_cfg as well and drop
IFCVF_MAX_QUEUES.
Yes, this has been done in 3/3 patch in this series.
Thanks
Zhu Lingshan
Thanks
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 3 +++
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 2 ++
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 1b5da11f5403..f86495ace825 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -134,6 +134,9 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
}
hw->nr_vring = vp_ioread16(&hw->common_cfg->num_queues);
+ hw->vring = kzalloc(sizeof(struct vring_info) * hw->nr_vring,
GFP_KERNEL);
+ if (!hw->vring)
+ return -ENOMEM;
for (i = 0; i < hw->nr_vring; i++) {
vp_iowrite16(i, &hw->common_cfg->queue_select);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 3110ffc50caf..fa797184056b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -74,7 +74,7 @@ struct ifcvf_hw {
u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
- struct vring_info vring[IFCVF_MAX_QUEUES];
+ struct vring_info *vring;
void __iomem * const *base;
char config_msix_name[256];
struct vdpa_callback config_cb;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 6e47ac2c669a..2af0de771b49 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -830,6 +830,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
return 0;
err:
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
return ret;
}
@@ -840,6 +841,7 @@ static void ifcvf_remove(struct pci_dev *pdev)
ifcvf_mgmt_dev = pci_get_drvdata(pdev);
vdpa_mgmtdev_unregister(&ifcvf_mgmt_dev->mdev);
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 2/3] vDPA/ifcvf: detect and report max allowed vq size
Rather than a hardcode, this commit detects
and reports the max value of allowed size
of the virtqueues
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 31 +++
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +++-
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index f86495ace825..f4d7d96c4c86 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -69,6 +69,37 @@ static int ifcvf_read_config_range(struct pci_dev *dev,
return 0;
}
+static u16 ifcvf_get_vq_size(struct ifcvf_hw *hw, u16 qid)
+{
+ u16 queue_size;
+
+ vp_iowrite16(qid, &hw->common_cfg->queue_select);
+ queue_size = vp_ioread16(&hw->common_cfg->queue_size);
+
+ return queue_size;
+}
+
+/* This function returns the max allowed safe size for
+ * all virtqueues. It is the minimal size that can be
+ * suppprted by all virtqueues.
+ */
+u16 ifcvf_get_max_vq_size(struct ifcvf_hw *hw)
+{
+ u16 queue_size, max_size, qid;
+
+ max_size = ifcvf_get_vq_size(hw, 0);
+ for (qid = 1; qid < hw->nr_vring; qid++) {
+ queue_size = ifcvf_get_vq_size(hw, qid);
+ /* 0 means the queue is unavailable */
+ if (!queue_size)
+ continue;
+
+ max_size = min(queue_size, max_size);
+ }
+
+ return max_size;
+}
+
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
{
struct virtio_pci_cap cap;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index fa797184056b..30935a95b672 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -28,7 +28,6 @@
#define IFCVF_MAX_QUEUES 17
#define IFCVF_QUEUE_ALIGNMENT PAGE_SIZE
-#define IFCVF_QUEUE_MAX32768
#define IFCVF_PCI_MAX_RESOURCE 6
#define IFCVF_LM_CFG_SIZE 0x40
@@ -138,4 +137,5 @@ bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features);
u64 ifcvf_get_driver_features(struct ifcvf_hw *hw);
+u16 ifcvf_get_max_vq_size(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 2af0de771b49..c3ece395caf7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -451,7 +451,9 @@ static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_dev)
static u16 ifcvf_vdpa_get_vq_num_max(struct vdpa_device *vdpa_dev)
{
- return IFCVF_QUEUE_MAX;
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+
+ return ifcvf_get_max_vq_size(vf);
}
static int ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 3/3] vDPA/ifcvf: implement new accessors for vq_state
This commit implements a better layout of the
live migration bar, therefore the accessors for virtqueue
state have been refactored.
This commit also add a comment to the probing-ids list,
indicating this driver drives F2000X-PL virtio-net
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 21 +
drivers/vdpa/ifcvf/ifcvf_base.h | 25 +
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +++-
3 files changed, 17 insertions(+), 33 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index f4d7d96c4c86..060f837a4f9f 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -328,30 +328,19 @@ void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64
features)
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
- struct ifcvf_lm_cfg __iomem *ifcvf_lm;
- void __iomem *avail_idx_addr;
+ struct ifcvf_lm_cfg __iomem *lm_cfg = hw->lm_cfg;
u16 last_avail_idx;
- u32 q_pair_id;
- ifcvf_lm = (struct ifcvf_lm_cfg __iomem *)hw->lm_cfg;
- q_pair_id = qid / 2;
- avail_idx_addr = &ifcvf_lm->vring_lm_cfg[q_pair_id].idx_addr[qid % 2];
- last_avail_idx = vp_ioread16(avail_idx_addr);
+ last_avail_idx = vp_ioread16(&lm_cfg->vq_state_region + qid * 2);
return last_avail_idx;
}
int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num)
{
- struct ifcvf_lm_cfg __iomem *ifcvf_lm;
- void __iomem *avail_idx_addr;
- u32 q_pair_id;
-
- ifcvf_lm = (struct ifcvf_lm_cfg __iomem *)hw->lm_cfg;
- q_pair_id = qid / 2;
- avail_idx_addr = &ifcvf_lm->vring_lm_cfg[q_pair_id].idx_addr[qid % 2];
- hw->vring[qid].last_avail_idx = num;
- vp_iowrite16(num, avail_idx_addr);
+ struct ifcvf_lm_cfg __iomem *lm_cfg = hw->lm_cfg;
+
+ vp_iowrite16(num, &lm_cfg->vq_state_region + qid * 2);
return 0;
}
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 30935a95b672..b57849c643f6 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -24,14 +24,9 @@
#define N3000_DEVICE_ID0x1041
#define N3000_SUBSYS_DEVICE_ID 0x001A
-/* Max 8 data queue pairs(16 queues) and one control vq for now. */
-#define IFCVF_MAX_QUEUES 17
-
#define IFCVF_QUEUE_ALIGNMENT PAGE_SIZE
#define IFCVF_PCI_MAX_RESOURCE 6
-#define IFCVF_LM_CFG_SIZE 0x40
-#define IFCVF_LM_RING_STATE_OFFSET 0x20
#define IFCVF_LM_BAR 4
#define IFCVF_ERR(pdev, fmt, ...) dev_err(&pdev->dev, fmt, ##__VA_ARGS__)
@@ -54,10 +49,18 @@ struct vring_info {
char msix_name[256];
};
+struct ifcvf_lm_cfg {
+ __le64 control;
+ __le64 status;
+ __le64 lm_mem_log_start_addr;
+ __le64 lm_mem_log_end_addr;
+ __le16 vq_state_region;
+};
+
struct ifcvf_hw {
u8 __iomem *isr;
/* Live migration */
- u8 __iomem *lm_cfg;
+ struct ifcvf_lm_cfg __iomem *lm_cfg;
/* Notification bar number */
u8 notify_bar;
u8 msix_vector_status;
@@ -92,16 +95,6 @@ struct ifcvf_adapter {
struct ifcvf_hw *vf;
};
-struct ifcvf_vring_lm_cfg {
- u32 idx_addr[2];
- u8 reserved[IFCVF_LM_CFG_SIZE - 8];
-};
-
-struct ifcvf_lm_cfg {
- u8 reserved[IFCVF_LM_RING_STATE_OFFSET];
- struct ifcvf_vring_lm_cfg vring_lm_cfg[IFCVF_MAX_QUEUES];
-};
-
struct ifcvf_vdpa_mgmt_dev {
struct vdpa_mgmt_dev mdev;
struct ifcvf_hw vf;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index c3ece395caf7..e98fa8100f3c 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -853,7 +853,9 @@ static struct pci_device_id ifcvf_pci_ids[] = {
N3000_DEVICE_ID,
PCI_VENDOR_ID_INTEL,
N3000_SUBSYS_DEVICE_ID) },
- /* C5000X-PL network device */
+ /* C5000X-PL network device
+* F2000X-PL network device
+*/
{ PCI_DEVICE_SUB(PCI_VENDOR_ID_REDHAT_QUMRANET,
VIRTIO_TRANS_ID_NET,
PCI_VENDOR_ID_INTEL,
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 1/3] vDPA/ifcvf: dynamic allocate vq data stores
This commit dynamically allocates the data
stores for the virtqueues based on
virtio_pci_common_cfg.num_queues.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 3 +++
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 2 ++
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 1b5da11f5403..f86495ace825 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -134,6 +134,9 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
}
hw->nr_vring = vp_ioread16(&hw->common_cfg->num_queues);
+ hw->vring = kzalloc(sizeof(struct vring_info) * hw->nr_vring,
GFP_KERNEL);
+ if (!hw->vring)
+ return -ENOMEM;
for (i = 0; i < hw->nr_vring; i++) {
vp_iowrite16(i, &hw->common_cfg->queue_select);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 3110ffc50caf..fa797184056b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -74,7 +74,7 @@ struct ifcvf_hw {
u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
- struct vring_info vring[IFCVF_MAX_QUEUES];
+ struct vring_info *vring;
void __iomem * const *base;
char config_msix_name[256];
struct vdpa_callback config_cb;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 6e47ac2c669a..2af0de771b49 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -830,6 +830,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
return 0;
err:
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
return ret;
}
@@ -840,6 +841,7 @@ static void ifcvf_remove(struct pci_dev *pdev)
ifcvf_mgmt_dev = pci_get_drvdata(pdev);
vdpa_mgmtdev_unregister(&ifcvf_mgmt_dev->mdev);
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 0/3] vDPA/ifcvf: enable virtio-net on Intel F2000X-PL
Hi This series enables vDPA/ifcvf support for virtio-net devices on Intel F2000X-PL Infrastructure Processing Unit. This series is rebased on MST's vhost tree, linux-next branch Please help review Thanks! Changes from V1: 1)Rebased on MST's tree 2)Remove IFCVF_MAX_QUEUES and other minor improvements Zhu Lingshan (3): vDPA/ifcvf: dynamic allocate vq data stores vDPA/ifcvf: detect and report max allowed vq size vDPA/ifcvf: implement new accessors for vq_state drivers/vdpa/ifcvf/ifcvf_base.c | 55 +++-- drivers/vdpa/ifcvf/ifcvf_base.h | 29 +++-- drivers/vdpa/ifcvf/ifcvf_main.c | 10 -- 3 files changed, 58 insertions(+), 36 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/3] vDPA/ifcvf: enable virtio-net on Intel F2000X-PL
On 6/10/2023 12:15 AM, Michael S. Tsirkin wrote: On Sat, Jun 10, 2023 at 12:21:57AM +0800, Zhu Lingshan wrote: Hi This series enables vDPA/ifcvf support for virtio-net devices on Intel F2000X-PL Infrastructure Processing Unit. Please help review Hard trouble applying - which tree is this against? Linus tree. I will rebase this series on your tree soon, linux-next branch, with some minor improvements. Thanks Thanks! Zhu Lingshan (3): vDPA/ifcvf: dynamic allocate vq data stores vDPA/ifcvf: detect and report max allowed vq size vDPA/ifcvf: implement new accessors for vq_state drivers/vdpa/ifcvf/ifcvf_base.c | 55 +++-- drivers/vdpa/ifcvf/ifcvf_base.h | 26 +++- drivers/vdpa/ifcvf/ifcvf_main.c | 10 -- 3 files changed, 58 insertions(+), 33 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 2/3] vDPA/ifcvf: detect and report max allowed vq size
Rather than a hardcode, this commit detects
and reports the max value of allowed size
of the virtqueues
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 31 +++
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +++-
3 files changed, 35 insertions(+), 2 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 3c40caf4aa0b..5f83524aa727 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -69,6 +69,37 @@ static int ifcvf_read_config_range(struct pci_dev *dev,
return 0;
}
+static u16 ifcvf_get_vq_size(struct ifcvf_hw *hw, u16 qid)
+{
+ u16 queue_size;
+
+ vp_iowrite16(qid, &hw->common_cfg->queue_select);
+ queue_size = vp_ioread16(&hw->common_cfg->queue_size);
+
+ return queue_size;
+}
+
+/* This function returns the max allowed safe size for
+ * all virtqueues. It is the minimal size that can be
+ * suppprted by all virtqueues.
+ */
+u16 ifcvf_get_max_vq_size(struct ifcvf_hw *hw)
+{
+ u16 queue_size, max_size, qid;
+
+ max_size = ifcvf_get_vq_size(hw, 0);
+ for (qid = 1; qid < hw->nr_vring; qid++) {
+ queue_size = ifcvf_get_vq_size(hw, qid);
+ /* 0 means the queue is unavailable*/
+ if (!queue_size)
+ continue;
+
+ max_size = min(queue_size, max_size);
+ }
+
+ return max_size;
+}
+
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
{
struct virtio_pci_cap cap;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 745282239d6d..e7803cc4ef39 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -28,7 +28,6 @@
#define IFCVF_MAX_QUEUES 17
#define IFCVF_QUEUE_ALIGNMENT PAGE_SIZE
-#define IFCVF_QUEUE_MAX32768
#define IFCVF_PCI_MAX_RESOURCE 6
#define IFCVF_LM_CFG_SIZE 0x40
@@ -137,4 +136,5 @@ int ifcvf_probed_virtio_net(struct ifcvf_hw *hw);
u32 ifcvf_get_config_size(struct ifcvf_hw *hw);
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector);
u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector);
+u16 ifcvf_get_max_vq_size(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index c97dde02bbb6..d7875b461a5f 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -509,7 +509,9 @@ static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_dev)
static u16 ifcvf_vdpa_get_vq_num_max(struct vdpa_device *vdpa_dev)
{
- return IFCVF_QUEUE_MAX;
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+
+ return ifcvf_get_max_vq_size(vf);
}
static int ifcvf_vdpa_get_vq_state(struct vdpa_device *vdpa_dev, u16 qid,
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 3/3] vDPA/ifcvf: implement new accessors for vq_state
This commit implements a better layout of the
live migration bar, therefore the accessors for virtqueue
state have been refactored.
This commit also add a comment to the probing-ids list,
indicating this driver drives F2000X-PL virtio-net
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 20
drivers/vdpa/ifcvf/ifcvf_base.h | 22 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +++-
3 files changed, 16 insertions(+), 30 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5f83524aa727..e04eeeccce05 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -335,31 +335,19 @@ static int ifcvf_config_features(struct ifcvf_hw *hw)
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
- struct ifcvf_lm_cfg __iomem *ifcvf_lm;
- void __iomem *avail_idx_addr;
+ struct ifcvf_lm_cfg __iomem *lm_cfg = hw->lm_cfg;
u16 last_avail_idx;
- u32 q_pair_id;
- ifcvf_lm = (struct ifcvf_lm_cfg __iomem *)hw->lm_cfg;
- q_pair_id = qid / 2;
- avail_idx_addr = &ifcvf_lm->vring_lm_cfg[q_pair_id].idx_addr[qid % 2];
- last_avail_idx = vp_ioread16(avail_idx_addr);
+ last_avail_idx = vp_ioread16(&lm_cfg->vq_state_region + qid * 2);
return last_avail_idx;
}
int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num)
{
+ struct ifcvf_lm_cfg __iomem *lm_cfg = hw->lm_cfg;
- struct ifcvf_lm_cfg __iomem *ifcvf_lm;
- void __iomem *avail_idx_addr;
- u32 q_pair_id;
-
- ifcvf_lm = (struct ifcvf_lm_cfg __iomem *)hw->lm_cfg;
- q_pair_id = qid / 2;
- avail_idx_addr = &ifcvf_lm->vring_lm_cfg[q_pair_id].idx_addr[qid % 2];
- hw->vring[qid].last_avail_idx = num;
- vp_iowrite16(num, avail_idx_addr);
+ vp_iowrite16(num, &lm_cfg->vq_state_region + qid * 2);
return 0;
}
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index e7803cc4ef39..2c11734a5270 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -30,8 +30,6 @@
#define IFCVF_QUEUE_ALIGNMENT PAGE_SIZE
#define IFCVF_PCI_MAX_RESOURCE 6
-#define IFCVF_LM_CFG_SIZE 0x40
-#define IFCVF_LM_RING_STATE_OFFSET 0x20
#define IFCVF_LM_BAR 4
#define IFCVF_ERR(pdev, fmt, ...) dev_err(&pdev->dev, fmt, ##__VA_ARGS__)
@@ -59,10 +57,18 @@ struct vring_info {
char msix_name[256];
};
+struct ifcvf_lm_cfg {
+ __le64 control;
+ __le64 status;
+ __le64 lm_mem_log_start_addr;
+ __le64 lm_mem_log_end_addr;
+ __le16 vq_state_region;
+};
+
struct ifcvf_hw {
u8 __iomem *isr;
/* Live migration */
- u8 __iomem *lm_cfg;
+ struct ifcvf_lm_cfg __iomem *lm_cfg;
/* Notification bar number */
u8 notify_bar;
u8 msix_vector_status;
@@ -97,16 +103,6 @@ struct ifcvf_adapter {
struct ifcvf_hw *vf;
};
-struct ifcvf_vring_lm_cfg {
- u32 idx_addr[2];
- u8 reserved[IFCVF_LM_CFG_SIZE - 8];
-};
-
-struct ifcvf_lm_cfg {
- u8 reserved[IFCVF_LM_RING_STATE_OFFSET];
- struct ifcvf_vring_lm_cfg vring_lm_cfg[IFCVF_MAX_QUEUES];
-};
-
struct ifcvf_vdpa_mgmt_dev {
struct vdpa_mgmt_dev mdev;
struct ifcvf_hw vf;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index d7875b461a5f..f25e413636dc 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -915,7 +915,9 @@ static struct pci_device_id ifcvf_pci_ids[] = {
N3000_DEVICE_ID,
PCI_VENDOR_ID_INTEL,
N3000_SUBSYS_DEVICE_ID) },
- /* C5000X-PL network device */
+ /* C5000X-PL network device
+* F2000X-PL network device
+*/
{ PCI_DEVICE_SUB(PCI_VENDOR_ID_REDHAT_QUMRANET,
VIRTIO_TRANS_ID_NET,
PCI_VENDOR_ID_INTEL,
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 0/3] vDPA/ifcvf: enable virtio-net on Intel F2000X-PL
Hi This series enables vDPA/ifcvf support for virtio-net devices on Intel F2000X-PL Infrastructure Processing Unit. Please help review Thanks! Zhu Lingshan (3): vDPA/ifcvf: dynamic allocate vq data stores vDPA/ifcvf: detect and report max allowed vq size vDPA/ifcvf: implement new accessors for vq_state drivers/vdpa/ifcvf/ifcvf_base.c | 55 +++-- drivers/vdpa/ifcvf/ifcvf_base.h | 26 +++- drivers/vdpa/ifcvf/ifcvf_main.c | 10 -- 3 files changed, 58 insertions(+), 33 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 1/3] vDPA/ifcvf: dynamic allocate vq data stores
This commit dynamically allocates the data
stores for the virtqueues based on
virtio_pci_common_cfg.num_queues.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 4
drivers/vdpa/ifcvf/ifcvf_base.h | 2 +-
drivers/vdpa/ifcvf/ifcvf_main.c | 2 ++
3 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5563b3a773c7..3c40caf4aa0b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -134,6 +134,9 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
}
hw->nr_vring = vp_ioread16(&hw->common_cfg->num_queues);
+ hw->vring = kzalloc(sizeof(struct vring_info) * hw->nr_vring,
GFP_KERNEL);
+ if (!hw->vring)
+ return -ENOMEM;
for (i = 0; i < hw->nr_vring; i++) {
vp_iowrite16(i, &hw->common_cfg->queue_select);
@@ -316,6 +319,7 @@ u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num)
{
+
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
void __iomem *avail_idx_addr;
u32 q_pair_id;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index c20d1c40214e..745282239d6d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -80,7 +80,7 @@ struct ifcvf_hw {
u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
- struct vring_info vring[IFCVF_MAX_QUEUES];
+ struct vring_info *vring;
void __iomem * const *base;
char config_msix_name[256];
struct vdpa_callback config_cb;
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7f78c47e40d6..c97dde02bbb6 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -892,6 +892,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
return 0;
err:
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
return ret;
}
@@ -902,6 +903,7 @@ static void ifcvf_remove(struct pci_dev *pdev)
ifcvf_mgmt_dev = pci_get_drvdata(pdev);
vdpa_mgmtdev_unregister(&ifcvf_mgmt_dev->mdev);
+ kfree(ifcvf_mgmt_dev->vf.vring);
kfree(ifcvf_mgmt_dev);
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [RFC] virtio-net: support modern-transtional devices
On 5/29/2023 8:04 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 06:41:54PM +0800, Zhu, Lingshan wrote:
On 5/29/2023 6:12 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 04:07:42PM +0800, Zhu, Lingshan wrote:
On 5/29/2023 2:38 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 02:19:36PM +0800, Zhu, Lingshan wrote:
On 5/28/2023 7:28 PM, Michael S. Tsirkin wrote:
On Sat, May 27, 2023 at 02:15:42AM +0800, Zhu Lingshan
wrote:
Current virtio-net only probes a device with
VIRITO_ID_NET == 1.
For a modern-transtional virtio-net device which has a
transtional
device id 0x1000 and acts as a modern device, current
virtio-pci
modern driver will assign the sub-device-id to its
mdev->id.device,
which may not be 0x1, this sub-device-id is up to the
vendor.
That means virtio-net driver doesn't probe a
modern-transitonal
virtio-net with a sub-device-id other than 0x1, which
is a bug.
No, the bug is in the device. Legacy linux drivers always
looked at
sub device id (other OSes might differ). So it makes no
sense
for a transitional device to have sub-device-id other than
0x1.
Don't have time to look at spec but I think you will find
it there.
That is true for a software emulated transitional device,
because there is only "generation" of instance in the
hypervisor,
that allowing it to ensure its sub-device-id always be 0x01,
and it fits VIRTIO_ID_NET.
However, a vendor may produce multiple generations of
transitional
hardware. The sub-device-id is up to the vendor, and it is the
only way to for a driver to identify a device, other IDs are
all
fixed as 0x1af4, 0x1000 and 0x8086 for Intel.
That is one of the issues with legacy virtio, yes.
So the sub-device-id has to be unique and differ from others,
can not always
be 0x01.
If you are trying to build a device and want to create a safe way
to
identify it without breaking legacy drivers, then
VIRTIO_PCI_CAP_VENDOR_CFG has been designed for things like this.
For example you can have:
struct virtio_pci_vndr_data {
u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */
u8 cap_next;/* Generic PCI field: next ptr. */
u8 cap_len; /* Generic PCI field: capability length */
u8 cfg_type;/* Identifies the structure. */
u16 vendor_id; /* Identifies the vendor-specific format.
*/
u16 device_generation; /* Device generation */
};
This can be a solution for sure.
I propose this fix, all changes are for modern-transitional
devices in
modern
code path, not for legacy nor legacy-transitional.
Thanks
But what good is this fix? If you just want the modern driver to
bind
and ignore legacy just create a modern device, you can play
with subsystem id and vendor to your heart's content then.
Not sure who but there are some use-cases require
transnational devices than modern devices,
I don't like this neither.
If you are using transitional then presumably you want
legacy drives to bind, they will not bind if subsystem device
id changes.
well actually it is a transitional device and act as a
modern device by default, so modern driver will probe.
I think this fix is common and easy, just let virtio-net
probe transitional device id 0x1000 just like it probes
modern device id 0x1. This is a once for all fix.
This fix only affects modern-transitional devices in modern code path,
legacy is untouched.
Thanks
The point of having transitional as opposed to modern is to allow
legacy drivers. If you don't need legacy just use a non transitional
device.
Your device is out of spec:
Transitional devices MUST have the PCI Subsystem Device ID
matching the Virtio Device ID, as indicated in section \ref{sec:Device
Types}.
OK, thanks for point this out. Since the spec says so, I assume transitional is
almost legacy.
However the spec also says:
Transitional Device a device supporting both drivers conforming to this
specification, and allowing legacy drivers.
The transitional devices have their own de
Re: [RFC] virtio-net: support modern-transtional devices
On 5/29/2023 6:12 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 04:07:42PM +0800, Zhu, Lingshan wrote:
On 5/29/2023 2:38 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 02:19:36PM +0800, Zhu, Lingshan wrote:
On 5/28/2023 7:28 PM, Michael S. Tsirkin wrote:
On Sat, May 27, 2023 at 02:15:42AM +0800, Zhu Lingshan wrote:
Current virtio-net only probes a device with VIRITO_ID_NET == 1.
For a modern-transtional virtio-net device which has a transtional
device id 0x1000 and acts as a modern device, current virtio-pci
modern driver will assign the sub-device-id to its mdev->id.device,
which may not be 0x1, this sub-device-id is up to the vendor.
That means virtio-net driver doesn't probe a modern-transitonal
virtio-net with a sub-device-id other than 0x1, which is a bug.
No, the bug is in the device. Legacy linux drivers always looked at
sub device id (other OSes might differ). So it makes no sense
for a transitional device to have sub-device-id other than 0x1.
Don't have time to look at spec but I think you will find it there.
That is true for a software emulated transitional device,
because there is only "generation" of instance in the hypervisor,
that allowing it to ensure its sub-device-id always be 0x01,
and it fits VIRTIO_ID_NET.
However, a vendor may produce multiple generations of transitional
hardware. The sub-device-id is up to the vendor, and it is the
only way to for a driver to identify a device, other IDs are all
fixed as 0x1af4, 0x1000 and 0x8086 for Intel.
That is one of the issues with legacy virtio, yes.
So the sub-device-id has to be unique and differ from others, can not always
be 0x01.
If you are trying to build a device and want to create a safe way to
identify it without breaking legacy drivers, then
VIRTIO_PCI_CAP_VENDOR_CFG has been designed for things like this.
For example you can have:
struct virtio_pci_vndr_data {
u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */
u8 cap_next;/* Generic PCI field: next ptr. */
u8 cap_len; /* Generic PCI field: capability length */
u8 cfg_type;/* Identifies the structure. */
u16 vendor_id; /* Identifies the vendor-specific format. */
u16 device_generation; /* Device generation */
};
This can be a solution for sure.
I propose this fix, all changes are for modern-transitional devices in
modern
code path, not for legacy nor legacy-transitional.
Thanks
But what good is this fix? If you just want the modern driver to bind
and ignore legacy just create a modern device, you can play
with subsystem id and vendor to your heart's content then.
Not sure who but there are some use-cases require
transnational devices than modern devices,
I don't like this neither.
If you are using transitional then presumably you want
legacy drives to bind, they will not bind if subsystem device
id changes.
well actually it is a transitional device and act as a
modern device by default, so modern driver will probe.
I think this fix is common and easy, just let virtio-net
probe transitional device id 0x1000 just like it probes
modern device id 0x1. This is a once for all fix.
This fix only affects modern-transitional devices in modern code path,
legacy is untouched.
Thanks
The point of having transitional as opposed to modern is to allow
legacy drivers. If you don't need legacy just use a non transitional
device.
Your device is out of spec:
Transitional devices MUST have the PCI Subsystem Device ID
matching the Virtio Device ID, as indicated in section \ref{sec:Device
Types}.
OK, thanks for point this out. Since the spec says so, I assume
transitional is almost legacy.
However the spec also says:
Transitional Device a device supporting both drivers conforming to this
specification, and allowing legacy drivers.
The transitional devices have their own device id, like 0x1000 indicates
it is a network device.
Then why the sub-device-id has to be 0x1 in the spec? Is it because we
have the driver first?
Thanks
So you will have to explain why the setup you are describing
makes any sense at all before we consider this a fix.
Other types of devices also have similar issues, like virito-blk.
I propose to fix this problem of modern-transitonal device
whith this solution, all in the modern code path:
1) assign the device id to mdev->id.device
2) add transitional device ids in the virtio-net(and others) probe table.
Comments are welcome!
Thanks!
Signed-off-by: Zhu Lingshan
---
drivers/net/virtio_net.c | 1 +
drivers/virtio/virtio_pci_modern_dev.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 56ca1d270304..6b45d8602a6b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -4250,6 +4250,7 @@ static __maybe_unused int virtnet_restore(struct
virtio_device *vdev)
Re: [RFC] virtio-net: support modern-transtional devices
On 5/29/2023 2:38 PM, Michael S. Tsirkin wrote:
On Mon, May 29, 2023 at 02:19:36PM +0800, Zhu, Lingshan wrote:
On 5/28/2023 7:28 PM, Michael S. Tsirkin wrote:
On Sat, May 27, 2023 at 02:15:42AM +0800, Zhu Lingshan wrote:
Current virtio-net only probes a device with VIRITO_ID_NET == 1.
For a modern-transtional virtio-net device which has a transtional
device id 0x1000 and acts as a modern device, current virtio-pci
modern driver will assign the sub-device-id to its mdev->id.device,
which may not be 0x1, this sub-device-id is up to the vendor.
That means virtio-net driver doesn't probe a modern-transitonal
virtio-net with a sub-device-id other than 0x1, which is a bug.
No, the bug is in the device. Legacy linux drivers always looked at
sub device id (other OSes might differ). So it makes no sense
for a transitional device to have sub-device-id other than 0x1.
Don't have time to look at spec but I think you will find it there.
That is true for a software emulated transitional device,
because there is only "generation" of instance in the hypervisor,
that allowing it to ensure its sub-device-id always be 0x01,
and it fits VIRTIO_ID_NET.
However, a vendor may produce multiple generations of transitional
hardware. The sub-device-id is up to the vendor, and it is the
only way to for a driver to identify a device, other IDs are all
fixed as 0x1af4, 0x1000 and 0x8086 for Intel.
That is one of the issues with legacy virtio, yes.
So the sub-device-id has to be unique and differ from others, can not always
be 0x01.
If you are trying to build a device and want to create a safe way to
identify it without breaking legacy drivers, then
VIRTIO_PCI_CAP_VENDOR_CFG has been designed for things like this.
For example you can have:
struct virtio_pci_vndr_data {
u8 cap_vndr;/* Generic PCI field: PCI_CAP_ID_VNDR */
u8 cap_next;/* Generic PCI field: next ptr. */
u8 cap_len; /* Generic PCI field: capability length */
u8 cfg_type;/* Identifies the structure. */
u16 vendor_id; /* Identifies the vendor-specific format. */
u16 device_generation; /* Device generation */
};
This can be a solution for sure.
I propose this fix, all changes are for modern-transitional devices in
modern
code path, not for legacy nor legacy-transitional.
Thanks
But what good is this fix? If you just want the modern driver to bind
and ignore legacy just create a modern device, you can play
with subsystem id and vendor to your heart's content then.
Not sure who but there are some use-cases require
transnational devices than modern devices,
I don't like this neither.
If you are using transitional then presumably you want
legacy drives to bind, they will not bind if subsystem device
id changes.
well actually it is a transitional device and act as a
modern device by default, so modern driver will probe.
I think this fix is common and easy, just let virtio-net
probe transitional device id 0x1000 just like it probes
modern device id 0x1. This is a once for all fix.
This fix only affects modern-transitional devices in modern code path,
legacy is untouched.
Thanks
Other types of devices also have similar issues, like virito-blk.
I propose to fix this problem of modern-transitonal device
whith this solution, all in the modern code path:
1) assign the device id to mdev->id.device
2) add transitional device ids in the virtio-net(and others) probe table.
Comments are welcome!
Thanks!
Signed-off-by: Zhu Lingshan
---
drivers/net/virtio_net.c | 1 +
drivers/virtio/virtio_pci_modern_dev.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 56ca1d270304..6b45d8602a6b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -4250,6 +4250,7 @@ static __maybe_unused int virtnet_restore(struct
virtio_device *vdev)
static struct virtio_device_id id_table[] = {
{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
+ { VIRTIO_TRANS_ID_NET, VIRTIO_DEV_ANY_ID },
{ 0 },
};
diff --git a/drivers/virtio/virtio_pci_modern_dev.c
b/drivers/virtio/virtio_pci_modern_dev.c
index 869cb46bef96..80846e1195ce 100644
--- a/drivers/virtio/virtio_pci_modern_dev.c
+++ b/drivers/virtio/virtio_pci_modern_dev.c
@@ -229,7 +229,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
/* Transitional devices: use the PCI subsystem device id as
* virtio device id, same as legacy driver always did.
*/
- mdev->id.device = pci_dev->subsystem_device;
+ mdev->id.device = pci_dev->device;
} else {
/* Modern devices: simply use PCI device id, but start from
0x1040. */
mdev->id.device = pci_dev->device - 0x1040;
--
2.39.1
___
Virtualization mailin
Re: [RFC] virtio-net: support modern-transtional devices
On 5/28/2023 7:28 PM, Michael S. Tsirkin wrote:
On Sat, May 27, 2023 at 02:15:42AM +0800, Zhu Lingshan wrote:
Current virtio-net only probes a device with VIRITO_ID_NET == 1.
For a modern-transtional virtio-net device which has a transtional
device id 0x1000 and acts as a modern device, current virtio-pci
modern driver will assign the sub-device-id to its mdev->id.device,
which may not be 0x1, this sub-device-id is up to the vendor.
That means virtio-net driver doesn't probe a modern-transitonal
virtio-net with a sub-device-id other than 0x1, which is a bug.
No, the bug is in the device. Legacy linux drivers always looked at
sub device id (other OSes might differ). So it makes no sense
for a transitional device to have sub-device-id other than 0x1.
Don't have time to look at spec but I think you will find it there.
That is true for a software emulated transitional device,
because there is only "generation" of instance in the hypervisor,
that allowing it to ensure its sub-device-id always be 0x01,
and it fits VIRTIO_ID_NET.
However, a vendor may produce multiple generations of transitional
hardware. The sub-device-id is up to the vendor, and it is the
only way to for a driver to identify a device, other IDs are all
fixed as 0x1af4, 0x1000 and 0x8086 for Intel.
So the sub-device-id has to be unique and differ from others, can not
always be 0x01.
I propose this fix, all changes are for modern-transitional devices in
modern
code path, not for legacy nor legacy-transitional.
Thanks
Other types of devices also have similar issues, like virito-blk.
I propose to fix this problem of modern-transitonal device
whith this solution, all in the modern code path:
1) assign the device id to mdev->id.device
2) add transitional device ids in the virtio-net(and others) probe table.
Comments are welcome!
Thanks!
Signed-off-by: Zhu Lingshan
---
drivers/net/virtio_net.c | 1 +
drivers/virtio/virtio_pci_modern_dev.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 56ca1d270304..6b45d8602a6b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -4250,6 +4250,7 @@ static __maybe_unused int virtnet_restore(struct
virtio_device *vdev)
static struct virtio_device_id id_table[] = {
{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
+ { VIRTIO_TRANS_ID_NET, VIRTIO_DEV_ANY_ID },
{ 0 },
};
diff --git a/drivers/virtio/virtio_pci_modern_dev.c b/drivers/virtio/virtio_pci_modern_dev.c
index 869cb46bef96..80846e1195ce 100644
--- a/drivers/virtio/virtio_pci_modern_dev.c
+++ b/drivers/virtio/virtio_pci_modern_dev.c
@@ -229,7 +229,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
/* Transitional devices: use the PCI subsystem device id as
* virtio device id, same as legacy driver always did.
*/
- mdev->id.device = pci_dev->subsystem_device;
+ mdev->id.device = pci_dev->device;
} else {
/* Modern devices: simply use PCI device id, but start from
0x1040. */
mdev->id.device = pci_dev->device - 0x1040;
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[RFC] virtio-net: support modern-transtional devices
Current virtio-net only probes a device with VIRITO_ID_NET == 1.
For a modern-transtional virtio-net device which has a transtional
device id 0x1000 and acts as a modern device, current virtio-pci
modern driver will assign the sub-device-id to its mdev->id.device,
which may not be 0x1, this sub-device-id is up to the vendor.
That means virtio-net driver doesn't probe a modern-transitonal
virtio-net with a sub-device-id other than 0x1, which is a bug.
Other types of devices also have similar issues, like virito-blk.
I propose to fix this problem of modern-transitonal device
whith this solution, all in the modern code path:
1) assign the device id to mdev->id.device
2) add transitional device ids in the virtio-net(and others) probe table.
Comments are welcome!
Thanks!
Signed-off-by: Zhu Lingshan
---
drivers/net/virtio_net.c | 1 +
drivers/virtio/virtio_pci_modern_dev.c | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 56ca1d270304..6b45d8602a6b 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -4250,6 +4250,7 @@ static __maybe_unused int virtnet_restore(struct
virtio_device *vdev)
static struct virtio_device_id id_table[] = {
{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
+ { VIRTIO_TRANS_ID_NET, VIRTIO_DEV_ANY_ID },
{ 0 },
};
diff --git a/drivers/virtio/virtio_pci_modern_dev.c
b/drivers/virtio/virtio_pci_modern_dev.c
index 869cb46bef96..80846e1195ce 100644
--- a/drivers/virtio/virtio_pci_modern_dev.c
+++ b/drivers/virtio/virtio_pci_modern_dev.c
@@ -229,7 +229,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
/* Transitional devices: use the PCI subsystem device id as
* virtio device id, same as legacy driver always did.
*/
- mdev->id.device = pci_dev->subsystem_device;
+ mdev->id.device = pci_dev->device;
} else {
/* Modern devices: simply use PCI device id, but start from
0x1040. */
mdev->id.device = pci_dev->device - 0x1040;
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V3 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop() and reset() are refactored as well.
This commit renames ifcvf_stop_hw() to ifcvf_stop()
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_base.c | 43 +-
drivers/vdpa/ifcvf/ifcvf_base.h | 3 ++-
drivers/vdpa/ifcvf/ifcvf_main.c | 46 +
3 files changed, 40 insertions(+), 52 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1b5da11f5403 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid,
bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
-void ifcvf_stop_hw(struct ifcvf_hw *hw)
+void ifcvf_stop(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..3110ffc50caf 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
@@ -110,7 +111,7 @@ struct ifcvf_vdpa_mgmt_dev {
};
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *dev);
-void ifcvf_stop_hw(struct ifcvf_hw *hw);
+void ifcvf_stop(struct ifcvf_hw *hw);
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);
void ifcvf_read_dev_config(struct ifcvf_hw *hw, u64 offset,
void *dst, int length);
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..5b7156209602 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
+ vf->num_msix_vectors = nvectors;
return 0;
}
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
- vf->vring[i].cb.callback = NULL;
- vf->vring[i].cb.private = NULL;
- }
-
- ifcvf_reset(vf);
-}
-
static struct ifcvf_adapter *vdpa_to_adapter(struct vdpa_device *vdpa_dev)
{
return container_of(vdpa_dev, struct ifcvf_adapter, vdpa);
@@ -462,23 +438,15 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 s
[PATCH V3 2/5] vDPA/ifcvf: get_driver_features from virtio registers
This commit implements a new function ifcvf_get_driver_feature()
which read driver_features from virtio registers.
To be less ambiguous, ifcvf_set_features() is renamed to
ifcvf_set_driver_features(), and ifcvf_get_features()
is renamed to ifcvf_get_dev_features() which returns
the provisioned vDPA device features.
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_base.c | 38 +
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 9 +---
3 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 6c5650f73007..546e923bcd16 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -204,11 +204,29 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
return features;
}
-u64 ifcvf_get_features(struct ifcvf_hw *hw)
+/* return provisioned vDPA dev features */
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw)
{
return hw->dev_features;
}
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u32 features_lo, features_hi;
+ u64 features;
+
+ vp_iowrite32(0, &cfg->device_feature_select);
+ features_lo = vp_ioread32(&cfg->guest_feature);
+
+ vp_iowrite32(1, &cfg->device_feature_select);
+ features_hi = vp_ioread32(&cfg->guest_feature);
+
+ features = ((u64)features_hi << 32) | features_lo;
+
+ return features;
+}
+
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
@@ -275,7 +293,7 @@ void ifcvf_write_dev_config(struct ifcvf_hw *hw, u64 offset,
vp_iowrite8(*p++, hw->dev_cfg + offset + i);
}
-static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -286,19 +304,6 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64
features)
vp_iowrite32(features >> 32, &cfg->guest_feature);
}
-static int ifcvf_config_features(struct ifcvf_hw *hw)
-{
- ifcvf_set_features(hw, hw->req_features);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
-
- if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
- return -EIO;
- }
-
- return 0;
-}
-
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
@@ -387,9 +392,6 @@ int ifcvf_start_hw(struct ifcvf_hw *hw)
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
- if (ifcvf_config_features(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d545a9411143..cb19196c3ece 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -69,7 +69,6 @@ struct ifcvf_hw {
phys_addr_t notify_base_pa;
u32 notify_off_multiplier;
u32 dev_type;
- u64 req_features;
u64 hw_features;
/* provisioned device features */
u64 dev_features;
@@ -122,7 +121,7 @@ u8 ifcvf_get_status(struct ifcvf_hw *hw);
void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
-u64 ifcvf_get_features(struct ifcvf_hw *hw);
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
@@ -137,4 +136,6 @@ int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64
desc_area,
u64 driver_area, u64 device_area);
bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features);
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 1357c67014ab..4588484bd53d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -410,7 +410,7 @@ static u64 ifcvf_vdpa_get_device_features(struct
vdpa_device *vdpa_dev)
u64 features;
if (type == VIRTIO_ID_NET || type == VIRTIO_ID_BLOCK)
- features = ifcvf_get_features(vf);
+ features = ifcvf_get_dev_features(vf);
else {
features = 0;
[PATCH V3 5/5] vDPA/ifcvf: a vendor driver should not set _CONFIG_S_FAILED
VIRTIO_CONFIG_S_FAILED indicates the guest driver has given up
the device due to fatal errors. So it is the guest decision,
the vendor driver should not set this status to the device.
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 5b7156209602..6e47ac2c669a 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -426,9 +426,7 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
!(status_old & VIRTIO_CONFIG_S_DRIVER_OK)) {
ret = ifcvf_request_irq(vf);
if (ret) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
+ IFCVF_ERR(vf->pdev, "failed to request irq with error
%d\n", ret);
return;
}
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V3 1/5] vDPA/ifcvf: virt queue ops take immediate actions
In this commit, virtqueue operations including:
set_vq_num(), set_vq_address(), set_vq_ready()
and get_vq_ready() access PCI registers directly
to take immediate actions.
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_base.c | 58 -
drivers/vdpa/ifcvf/ifcvf_base.h | 10 +++---
drivers/vdpa/ifcvf/ifcvf_main.c | 16 +++--
3 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5563b3a773c7..6c5650f73007 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -329,31 +329,49 @@ int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16
num)
return 0;
}
-static int ifcvf_hw_enable(struct ifcvf_hw *hw)
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num)
{
- struct virtio_pci_common_cfg __iomem *cfg;
- u32 i;
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
- cfg = hw->common_cfg;
- for (i = 0; i < hw->nr_vring; i++) {
- if (!hw->vring[i].ready)
- break;
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(num, &cfg->queue_size);
+}
- vp_iowrite16(i, &cfg->queue_select);
- vp_iowrite64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,
-&cfg->queue_desc_hi);
- vp_iowrite64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,
- &cfg->queue_avail_hi);
- vp_iowrite64_twopart(hw->vring[i].used, &cfg->queue_used_lo,
-&cfg->queue_used_hi);
- vp_iowrite16(hw->vring[i].size, &cfg->queue_size);
- ifcvf_set_vq_state(hw, i, hw->vring[i].last_avail_idx);
- vp_iowrite16(1, &cfg->queue_enable);
- }
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite64_twopart(desc_area, &cfg->queue_desc_lo,
+&cfg->queue_desc_hi);
+ vp_iowrite64_twopart(driver_area, &cfg->queue_avail_lo,
+&cfg->queue_avail_hi);
+ vp_iowrite64_twopart(device_area, &cfg->queue_used_lo,
+&cfg->queue_used_hi);
return 0;
}
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u16 queue_enable;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ queue_enable = vp_ioread16(&cfg->queue_enable);
+
+ return (bool)queue_enable;
+}
+
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(ready, &cfg->queue_enable);
+}
+
static void ifcvf_hw_disable(struct ifcvf_hw *hw)
{
u32 i;
@@ -366,16 +384,12 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
int ifcvf_start_hw(struct ifcvf_hw *hw)
{
- ifcvf_reset(hw);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
if (ifcvf_config_features(hw) < 0)
return -EINVAL;
- if (ifcvf_hw_enable(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index c20d1c40214e..d545a9411143 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -47,12 +47,7 @@
#define MSIX_VECTOR_DEV_SHARED 3
struct vring_info {
- u64 desc;
- u64 avail;
- u64 used;
- u16 size;
u16 last_avail_idx;
- bool ready;
void __iomem *notify_addr;
phys_addr_t notify_pa;
u32 irq;
@@ -137,4 +132,9 @@ int ifcvf_probed_virtio_net(struct ifcvf_hw *hw);
u32 ifcvf_get_config_size(struct ifcvf_hw *hw);
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector);
u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector);
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num);
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area);
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7f78c47e40
[PATCH V3 3/5] vDPA/ifcvf: retire ifcvf_start_datapath and ifcvf_add_status
Rather than former lazy-initialization mechanism,
now the virtqueue operations and driver_features related
ops access the virtio registers directly to take
immediate actions. So ifcvf_start_datapath() should
retire.
ifcvf_add_status() is retierd because we should not change
device status by a vendor driver's decision, this driver should
only set device status which is from virito drivers
upon vdpa_ops.set_status()
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_base.c | 19 ---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 -
drivers/vdpa/ifcvf/ifcvf_main.c | 23 ---
3 files changed, 43 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 546e923bcd16..79e313c5e10e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -178,15 +178,6 @@ void ifcvf_reset(struct ifcvf_hw *hw)
ifcvf_get_status(hw);
}
-static void ifcvf_add_status(struct ifcvf_hw *hw, u8 status)
-{
- if (status != 0)
- status |= ifcvf_get_status(hw);
-
- ifcvf_set_status(hw, status);
- ifcvf_get_status(hw);
-}
-
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -387,16 +378,6 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
}
}
-int ifcvf_start_hw(struct ifcvf_hw *hw)
-{
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
-
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
-
- return 0;
-}
-
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
ifcvf_hw_disable(hw);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index cb19196c3ece..d34d3bc0dbf4 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -110,7 +110,6 @@ struct ifcvf_vdpa_mgmt_dev {
};
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *dev);
-int ifcvf_start_hw(struct ifcvf_hw *hw);
void ifcvf_stop_hw(struct ifcvf_hw *hw);
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);
void ifcvf_read_dev_config(struct ifcvf_hw *hw, u64 offset,
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 4588484bd53d..968687159e44 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,22 +346,6 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_start_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- u8 status;
- int ret;
-
- ret = ifcvf_start_hw(vf);
- if (ret < 0) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
- }
-
- return ret;
-}
-
static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
{
struct ifcvf_hw *vf = adapter->vf;
@@ -452,13 +436,11 @@ static u8 ifcvf_vdpa_get_status(struct vdpa_device
*vdpa_dev)
static void ifcvf_vdpa_set_status(struct vdpa_device *vdpa_dev, u8 status)
{
- struct ifcvf_adapter *adapter;
struct ifcvf_hw *vf;
u8 status_old;
int ret;
vf = vdpa_to_vf(vdpa_dev);
- adapter = vdpa_to_adapter(vdpa_dev);
status_old = ifcvf_get_status(vf);
if (status_old == status)
@@ -473,11 +455,6 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
ifcvf_set_status(vf, status);
return;
}
-
- if (ifcvf_start_datapath(adapter) < 0)
- IFCVF_ERR(adapter->pdev,
- "Failed to set ifcvf vdpa status %u\n",
- status);
}
ifcvf_set_status(vf, status);
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V3 0/5] vDPA/ifcvf: implement immediate initialization mechanism
Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. This series also implement irq synchronization in the reset routine Changes from V2: rename function ifcvf_stop_hw to ifcvf_stop (Jason) Changes from V1: 1)pull device status in devce_reset (Jason) 2)simplify the procedure which sycn irqs (Jason) 3)fix typos(Michael) Zhu Lingshan (5): vDPA/ifcvf: virt queue ops take immediate actions vDPA/ifcvf: get_driver_features from virtio registers vDPA/ifcvf: retire ifcvf_start_datapath and ifcvf_add_status vDPA/ifcvf: synchronize irqs in the reset routine vDPA/ifcvf: a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 148 ++-- drivers/vdpa/ifcvf/ifcvf_base.h | 19 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 98 - 3 files changed, 110 insertions(+), 155 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
On 5/26/2023 2:09 PM, Jason Wang wrote:
On Fri, May 26, 2023 at 1:30 PM Zhu, Lingshan wrote:
On 5/26/2023 11:36 AM, Zhu, Lingshan wrote:
On 5/26/2023 9:34 AM, Jason Wang wrote:
On Thu, May 25, 2023 at 5:38 PM Zhu, Lingshan
wrote:
On 5/24/2023 4:03 PM, Jason Wang wrote:
On Mon, May 8, 2023 at 6:05 PM Zhu Lingshan
wrote:
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 41 +
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 46
+
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1f39290baa38 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8
status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw,
u16 qid, bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
Nit:
So the name of this function is kind of misleading since irq
synchronization and virtqueue/config handler are not belong to
hardware?
Maybe it would be better to call it ifcvf_stop().
Sure, I will send a V3 with this renaming,
do you ack patch 1/5?
Yes, I think I've acked to that patch.
I will send a V3 with this renaming and your ack for patch 1/5
By the way, do you ack this one after this function renaming?
If so, I will also include your ack in V3, so we don't need another
review process, I will ping Michael for a merge.
Have you seen the ack here?
https://lists.linuxfoundation.org/pipermail/virtualization/2023-May/066890.html
Sorry I missed this, a patch only renames a function may be trivial, so
I will
rebase this 4/5 with your ack. So all patches are ack-ed, thanks!
Thanks
Thanks
Thanks
Zhu Lingshan
Thanks
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..7430f80779be 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..3401b9901dd2 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw
*vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
-
Re: [PATCH V2 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
On 5/26/2023 11:36 AM, Zhu, Lingshan wrote:
On 5/26/2023 9:34 AM, Jason Wang wrote:
On Thu, May 25, 2023 at 5:38 PM Zhu, Lingshan
wrote:
On 5/24/2023 4:03 PM, Jason Wang wrote:
On Mon, May 8, 2023 at 6:05 PM Zhu Lingshan
wrote:
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 41 +
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 46
+
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1f39290baa38 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8
status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw,
u16 qid, bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
Nit:
So the name of this function is kind of misleading since irq
synchronization and virtqueue/config handler are not belong to
hardware?
Maybe it would be better to call it ifcvf_stop().
Sure, I will send a V3 with this renaming,
do you ack patch 1/5?
Yes, I think I've acked to that patch.
I will send a V3 with this renaming and your ack for patch 1/5
By the way, do you ack this one after this function renaming?
If so, I will also include your ack in V3, so we don't need another
review process, I will ping Michael for a merge.
Thanks
Thanks
Zhu Lingshan
Thanks
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..7430f80779be 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..3401b9901dd2 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw
*vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
+ vf->num_msix_vectors = nvectors;
return 0;
}
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
-
Re: [PATCH V2 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
On 5/26/2023 9:34 AM, Jason Wang wrote:
On Thu, May 25, 2023 at 5:38 PM Zhu, Lingshan wrote:
On 5/24/2023 4:03 PM, Jason Wang wrote:
On Mon, May 8, 2023 at 6:05 PM Zhu Lingshan wrote:
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 41 +
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 46 +
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1f39290baa38 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid,
bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
Nit:
So the name of this function is kind of misleading since irq
synchronization and virtqueue/config handler are not belong to
hardware?
Maybe it would be better to call it ifcvf_stop().
Sure, I will send a V3 with this renaming,
do you ack patch 1/5?
Yes, I think I've acked to that patch.
I will send a V3 with this renaming and your ack for patch 1/5
Thanks
Thanks
Zhu Lingshan
Thanks
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..7430f80779be 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..3401b9901dd2 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
+ vf->num_msix_vectors = nvectors;
return 0;
}
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
- vf->vring[i].cb.callback = NULL;
- vf->vring[i].cb.private = NULL;
- }
-
- ifcvf_reset(vf);
-}
-
static struct ifcvf_adapter *vdpa_to_adapter(struc
Re: [PATCH V2 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
On 5/24/2023 4:03 PM, Jason Wang wrote:
On Mon, May 8, 2023 at 6:05 PM Zhu Lingshan wrote:
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 41 +
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 46 +
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1f39290baa38 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid,
bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
Nit:
So the name of this function is kind of misleading since irq
synchronization and virtqueue/config handler are not belong to
hardware?
Maybe it would be better to call it ifcvf_stop().
Sure, I will send a V3 with this renaming,
do you ack patch 1/5?
Thanks
Zhu Lingshan
Thanks
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..7430f80779be 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..3401b9901dd2 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
+ vf->num_msix_vectors = nvectors;
return 0;
}
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
- vf->vring[i].cb.callback = NULL;
- vf->vring[i].cb.private = NULL;
- }
-
- ifcvf_reset(vf);
-}
-
static struct ifcvf_adapter *vdpa_to_adapter(struct vdpa_device *vdpa_dev)
{
return container_of(vdpa_dev, struct ifcvf_adapter, vdpa);
@@ -462,23 +438,15 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_de
Re: [PATCH V2 0/5] vDPA/ifcvf: implement immediate initialization mechanism
ping On 5/9/2023 2:05 AM, Zhu Lingshan wrote: Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. This series also implement irq synchronization in the reset routine Changes from V1: 1)pull device status in devce_reset (Jason) 2)simplify the procedure which sycn irqs (Jason) 3)fix typos(Michael) Zhu Lingshan (5): vDPA/ifcvf: virt queue ops take immediate actions vDPA/ifcvf: get_driver_features from virtio registers vDPA/ifcvf: retire ifcvf_start_datapath and ifcvf_add_status vDPA/ifcvf: synchronize irqs in the reset routine vDPA/ifcvf: a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 146 ++-- drivers/vdpa/ifcvf/ifcvf_base.h | 17 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 98 - 3 files changed, 108 insertions(+), 153 deletions(-) ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 3/5] vDPA/ifcvf: retire ifcvf_start_datapath and ifcvf_add_status
Rather than former lazy-initialization mechanism,
now the virtqueue operations and driver_features related
ops access the virtio registers directly to take
immediate actions. So ifcvf_start_datapath() should
retire.
ifcvf_add_status() is retierd because we should not change
device status by a vendor driver's decision, this driver should
only set device status which is from virito drivers
upon vdpa_ops.set_status()
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_base.c | 19 ---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 -
drivers/vdpa/ifcvf/ifcvf_main.c | 23 ---
3 files changed, 43 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 546e923bcd16..79e313c5e10e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -178,15 +178,6 @@ void ifcvf_reset(struct ifcvf_hw *hw)
ifcvf_get_status(hw);
}
-static void ifcvf_add_status(struct ifcvf_hw *hw, u8 status)
-{
- if (status != 0)
- status |= ifcvf_get_status(hw);
-
- ifcvf_set_status(hw, status);
- ifcvf_get_status(hw);
-}
-
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -387,16 +378,6 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
}
}
-int ifcvf_start_hw(struct ifcvf_hw *hw)
-{
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
-
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
-
- return 0;
-}
-
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
ifcvf_hw_disable(hw);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index cb19196c3ece..d34d3bc0dbf4 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -110,7 +110,6 @@ struct ifcvf_vdpa_mgmt_dev {
};
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *dev);
-int ifcvf_start_hw(struct ifcvf_hw *hw);
void ifcvf_stop_hw(struct ifcvf_hw *hw);
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);
void ifcvf_read_dev_config(struct ifcvf_hw *hw, u64 offset,
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 4588484bd53d..968687159e44 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,22 +346,6 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_start_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- u8 status;
- int ret;
-
- ret = ifcvf_start_hw(vf);
- if (ret < 0) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
- }
-
- return ret;
-}
-
static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
{
struct ifcvf_hw *vf = adapter->vf;
@@ -452,13 +436,11 @@ static u8 ifcvf_vdpa_get_status(struct vdpa_device
*vdpa_dev)
static void ifcvf_vdpa_set_status(struct vdpa_device *vdpa_dev, u8 status)
{
- struct ifcvf_adapter *adapter;
struct ifcvf_hw *vf;
u8 status_old;
int ret;
vf = vdpa_to_vf(vdpa_dev);
- adapter = vdpa_to_adapter(vdpa_dev);
status_old = ifcvf_get_status(vf);
if (status_old == status)
@@ -473,11 +455,6 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
ifcvf_set_status(vf, status);
return;
}
-
- if (ifcvf_start_datapath(adapter) < 0)
- IFCVF_ERR(adapter->pdev,
- "Failed to set ifcvf vdpa status %u\n",
- status);
}
ifcvf_set_status(vf, status);
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 4/5] vDPA/ifcvf: synchronize irqs in the reset routine
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 41 +
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 46 +
3 files changed, 38 insertions(+), 50 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..1f39290baa38 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,9 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
+ while (ifcvf_get_status(hw))
+ msleep(1);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +365,42 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid,
bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+}
+
+static void ifcvf_synchronize_irq(struct ifcvf_hw *hw)
+{
+ u32 nvectors = hw->num_msix_vectors;
+ struct pci_dev *pdev = hw->pdev;
+ int i, irq;
+
+ for (i = 0; i < nvectors; i++) {
+ irq = pci_irq_vector(pdev, i);
+ if (irq >= 0)
+ synchronize_irq(irq);
}
}
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_synchronize_irq(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d34d3bc0dbf4..7430f80779be 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -82,6 +82,7 @@ struct ifcvf_hw {
int vqs_reused_irq;
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
+ u32 num_msix_vectors;
u32 cap_dev_config_size;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..3401b9901dd2 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -125,6 +125,7 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
+ vf->num_msix_vectors = 0;
}
/* ifcvf MSIX vectors allocator, this helper tries to allocate
@@ -343,36 +344,11 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
if (ret)
return ret;
- return 0;
-}
-
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
+ vf->num_msix_vectors = nvectors;
return 0;
}
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
- vf->vring[i].cb.callback = NULL;
- vf->vring[i].cb.private = NULL;
- }
-
- ifcvf_reset(vf);
-}
-
static struct ifcvf_adapter *vdpa_to_adapter(struct vdpa_device *vdpa_dev)
{
return container_of(vdpa_dev, struct ifcvf_adapter, vdpa);
@@ -462,23 +438,15 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_dev)
{
- struct ifcvf_adapter *adapter;
- struct ifcvf_hw *vf;
- u8 status_old;
-
- vf = vdpa_to_vf(vdpa_dev);
- adapter = vdpa_to_adapter(vdpa_dev);
- status_old = ifcvf_get_status(vf);
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+ u8 status = ifcvf_get_status(vf);
- if (status_old == 0)
- return 0;
+ ifcvf_stop_hw(vf);
-
[PATCH V2 5/5] vDPA/ifcvf: a vendor driver should not set _CONFIG_S_FAILED
VIRTIO_CONFIG_S_FAILED indicates the guest driver has given up
the device due to fatal errors. So it is the guest decision,
the vendor driver should not set this status to the device.
Signed-off-by: Zhu Lingshan
Acked-by: Jason Wang
---
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 3401b9901dd2..b413688e13c4 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -426,9 +426,7 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
!(status_old & VIRTIO_CONFIG_S_DRIVER_OK)) {
ret = ifcvf_request_irq(vf);
if (ret) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
+ IFCVF_ERR(vf->pdev, "failed to request irq with error
%d\n", ret);
return;
}
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 1/5] vDPA/ifcvf: virt queue ops take immediate actions
In this commit, virtqueue operations including:
set_vq_num(), set_vq_address(), set_vq_ready()
and get_vq_ready() access PCI registers directly
to take immediate actions.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 58 -
drivers/vdpa/ifcvf/ifcvf_base.h | 10 +++---
drivers/vdpa/ifcvf/ifcvf_main.c | 16 +++--
3 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5563b3a773c7..6c5650f73007 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -329,31 +329,49 @@ int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16
num)
return 0;
}
-static int ifcvf_hw_enable(struct ifcvf_hw *hw)
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num)
{
- struct virtio_pci_common_cfg __iomem *cfg;
- u32 i;
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
- cfg = hw->common_cfg;
- for (i = 0; i < hw->nr_vring; i++) {
- if (!hw->vring[i].ready)
- break;
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(num, &cfg->queue_size);
+}
- vp_iowrite16(i, &cfg->queue_select);
- vp_iowrite64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,
-&cfg->queue_desc_hi);
- vp_iowrite64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,
- &cfg->queue_avail_hi);
- vp_iowrite64_twopart(hw->vring[i].used, &cfg->queue_used_lo,
-&cfg->queue_used_hi);
- vp_iowrite16(hw->vring[i].size, &cfg->queue_size);
- ifcvf_set_vq_state(hw, i, hw->vring[i].last_avail_idx);
- vp_iowrite16(1, &cfg->queue_enable);
- }
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite64_twopart(desc_area, &cfg->queue_desc_lo,
+&cfg->queue_desc_hi);
+ vp_iowrite64_twopart(driver_area, &cfg->queue_avail_lo,
+&cfg->queue_avail_hi);
+ vp_iowrite64_twopart(device_area, &cfg->queue_used_lo,
+&cfg->queue_used_hi);
return 0;
}
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u16 queue_enable;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ queue_enable = vp_ioread16(&cfg->queue_enable);
+
+ return (bool)queue_enable;
+}
+
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(ready, &cfg->queue_enable);
+}
+
static void ifcvf_hw_disable(struct ifcvf_hw *hw)
{
u32 i;
@@ -366,16 +384,12 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
int ifcvf_start_hw(struct ifcvf_hw *hw)
{
- ifcvf_reset(hw);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
if (ifcvf_config_features(hw) < 0)
return -EINVAL;
- if (ifcvf_hw_enable(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index c20d1c40214e..d545a9411143 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -47,12 +47,7 @@
#define MSIX_VECTOR_DEV_SHARED 3
struct vring_info {
- u64 desc;
- u64 avail;
- u64 used;
- u16 size;
u16 last_avail_idx;
- bool ready;
void __iomem *notify_addr;
phys_addr_t notify_pa;
u32 irq;
@@ -137,4 +132,9 @@ int ifcvf_probed_virtio_net(struct ifcvf_hw *hw);
u32 ifcvf_get_config_size(struct ifcvf_hw *hw);
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector);
u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector);
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num);
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area);
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7f78c47e40d6..1357c67014ab 100644
-
[PATCH V2 2/5] vDPA/ifcvf: get_driver_features from virtio registers
This commit implements a new function ifcvf_get_driver_feature()
which read driver_features from virtio registers.
To be less ambiguous, ifcvf_set_features() is renamed to
ifcvf_set_driver_features(), and ifcvf_get_features()
is renamed to ifcvf_get_dev_features() which returns
the provisioned vDPA device features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 38 +
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 9 +---
3 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 6c5650f73007..546e923bcd16 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -204,11 +204,29 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
return features;
}
-u64 ifcvf_get_features(struct ifcvf_hw *hw)
+/* return provisioned vDPA dev features */
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw)
{
return hw->dev_features;
}
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u32 features_lo, features_hi;
+ u64 features;
+
+ vp_iowrite32(0, &cfg->device_feature_select);
+ features_lo = vp_ioread32(&cfg->guest_feature);
+
+ vp_iowrite32(1, &cfg->device_feature_select);
+ features_hi = vp_ioread32(&cfg->guest_feature);
+
+ features = ((u64)features_hi << 32) | features_lo;
+
+ return features;
+}
+
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
@@ -275,7 +293,7 @@ void ifcvf_write_dev_config(struct ifcvf_hw *hw, u64 offset,
vp_iowrite8(*p++, hw->dev_cfg + offset + i);
}
-static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -286,19 +304,6 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64
features)
vp_iowrite32(features >> 32, &cfg->guest_feature);
}
-static int ifcvf_config_features(struct ifcvf_hw *hw)
-{
- ifcvf_set_features(hw, hw->req_features);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
-
- if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
- return -EIO;
- }
-
- return 0;
-}
-
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
@@ -387,9 +392,6 @@ int ifcvf_start_hw(struct ifcvf_hw *hw)
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
- if (ifcvf_config_features(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d545a9411143..cb19196c3ece 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -69,7 +69,6 @@ struct ifcvf_hw {
phys_addr_t notify_base_pa;
u32 notify_off_multiplier;
u32 dev_type;
- u64 req_features;
u64 hw_features;
/* provisioned device features */
u64 dev_features;
@@ -122,7 +121,7 @@ u8 ifcvf_get_status(struct ifcvf_hw *hw);
void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
-u64 ifcvf_get_features(struct ifcvf_hw *hw);
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
@@ -137,4 +136,6 @@ int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64
desc_area,
u64 driver_area, u64 device_area);
bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features);
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 1357c67014ab..4588484bd53d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -410,7 +410,7 @@ static u64 ifcvf_vdpa_get_device_features(struct
vdpa_device *vdpa_dev)
u64 features;
if (type == VIRTIO_ID_NET || type == VIRTIO_ID_BLOCK)
- features = ifcvf_get_features(vf);
+ features = ifcvf_get_dev_features(vf);
else {
features = 0;
[PATCH V2 0/5] vDPA/ifcvf: implement immediate initialization mechanism
Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. This series also implement irq synchronization in the reset routine Changes from V1: 1)pull device status in devce_reset (Jason) 2)simplify the procedure which sycn irqs (Jason) 3)fix typos(Michael) Zhu Lingshan (5): vDPA/ifcvf: virt queue ops take immediate actions vDPA/ifcvf: get_driver_features from virtio registers vDPA/ifcvf: retire ifcvf_start_datapath and ifcvf_add_status vDPA/ifcvf: synchronize irqs in the reset routine vDPA/ifcvf: a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 146 ++-- drivers/vdpa/ifcvf/ifcvf_base.h | 17 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 98 - 3 files changed, 108 insertions(+), 153 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 4/5] synchronize irqs in the reset routine
On 4/26/2023 1:06 PM, Jason Wang wrote:
在 2023/4/1 04:48, Zhu Lingshan 写道:
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 61 ++---
drivers/vdpa/ifcvf/ifcvf_main.c | 45 +++-
2 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..49949aec20ef 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,7 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8
status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
If we don't flush or poll how can we know the reset is done?
E.g modern virtio-pci did:
/* 0 status means a reset. */
vp_modern_set_status(mdev, 0);
/* After writing 0 to device_status, the driver MUST wait for
a read of
* device_status to return 0 before reinitializing the device.
* This will flush out the status write, and flush in device
writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);
/* Flush pending VQ/configuration callbacks. */
vp_synchronize_vectors(vdev);
Thanks, I can implement a similar get_status() here.
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +363,62 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw,
u16 qid, bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void synchronize_per_vq_irq(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
- ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ if (hw->vring[qid].irq != -EINVAL)
+ synchronize_irq(hw->vring[qid].irq);
}
}
+static void synchronize_vqs_reused_irq(struct ifcvf_hw *hw)
+{
+ if (hw->vqs_reused_irq != -EINVAL)
+ synchronize_irq(hw->vqs_reused_irq);
+}
+
+static void synchronize_vq_irq(struct ifcvf_hw *hw)
+{
+ u8 status = hw->msix_vector_status;
+
+ if (status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
+ synchronize_per_vq_irq(hw);
+ else
+ synchronize_vqs_reused_irq(hw);
+}
I wonder if we need to go with such complicated ways,can we
synchronize through the vectors like virtio-pci did?
for (i = 0; i < vp_dev->msix_vectors; ++i)
synchronize_irq(pci_irq_vector(vp_dev->pci_dev, i));
?
I can record the number of msix_vectors and sycn irq based on it in V2.
+
+static void synchronize_config_irq(struct ifcvf_hw *hw)
+{
+ if (hw->config_irq != -EINVAL)
+ synchronize_irq(hw->config_irq);
+}
+
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
+{
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ synchronize_vq_irq(hw);
Since IRQ could be shared, this will result extra complexity, like a
irq could be flushed multiple times?
No for this code path, E.g., if the all vqs share one irq, it will only
be flushed once in synchronize_vqs_reused_irq()
Thanks
Thanks
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ synchronize_config_irq(hw);
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
+ ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
+}
+
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..15c6157ee841 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,33 +346,6 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
-
- return 0;
-}
-
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
-
Re: [PATCH 2/5] get_driver_features from virito registers
On 4/26/2023 12:02 PM, Jason Wang wrote:
在 2023/4/1 04:48, Zhu Lingshan 写道:
This commit implements a new function ifcvf_get_driver_feature()
which read driver_features from virtio registers.
To be less ambiguous, ifcvf_set_features() is renamed to
ifcvf_set_driver_features(), and ifcvf_get_features()
is renamed to ifcvf_get_dev_features() which returns
the provisioned vDPA device features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 38 +
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 9 +---
3 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 6c5650f73007..546e923bcd16 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -204,11 +204,29 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
return features;
}
-u64 ifcvf_get_features(struct ifcvf_hw *hw)
+/* return provisioned vDPA dev features */
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw)
{
return hw->dev_features;
}
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u32 features_lo, features_hi;
+ u64 features;
+
+ vp_iowrite32(0, &cfg->device_feature_select);
+ features_lo = vp_ioread32(&cfg->guest_feature);
+
+ vp_iowrite32(1, &cfg->device_feature_select);
+ features_hi = vp_ioread32(&cfg->guest_feature);
+
+ features = ((u64)features_hi << 32) | features_lo;
+
+ return features;
+}
This duplicates with the logic ifcvf_get_hw_features(), it would be
simpler if we just do a rename.
Yes, they look very similar. ifcvf_get_hw_features() reads
virtio_pci_common_cfg.device_feature
and ifcvf_get_driver_features reads virtio_pci_common_cfg.driver_feature.
Do you suggest we merge these two functions? something like this may
look chaotic:
u64 ifcvf_get_features(struct ifcvf_hw *hw, bool device_feature)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
u32 features_lo, features_hi;
u64 features;
if (device_feature) {
vp_iowrite32(0, &cfg->device_feature_select);
features_lo = vp_ioread32(&cfg->guest_feature);
vp_iowrite32(1, &cfg->device_feature_select);
features_hi = vp_ioread32(&cfg->guest_feature);
} else {
vp_iowrite32(0, &cfg->device_feature_select);
features_lo = vp_ioread32(&cfg->guest_feature);
vp_iowrite32(1, &cfg->device_feature_select);
features_hi = vp_ioread32(&cfg->guest_feature);
}
features = ((u64)features_hi << 32) | features_lo;
return features;
}
Maybe separate functions looks better.
Thanks
+
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
@@ -275,7 +293,7 @@ void ifcvf_write_dev_config(struct ifcvf_hw *hw,
u64 offset,
vp_iowrite8(*p++, hw->dev_cfg + offset + i);
}
-static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -286,19 +304,6 @@ static void ifcvf_set_features(struct ifcvf_hw
*hw, u64 features)
vp_iowrite32(features >> 32, &cfg->guest_feature);
}
-static int ifcvf_config_features(struct ifcvf_hw *hw)
-{
- ifcvf_set_features(hw, hw->req_features);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
-
- if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
- return -EIO;
- }
-
- return 0;
-}
-
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
@@ -387,9 +392,6 @@ int ifcvf_start_hw(struct ifcvf_hw *hw)
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
- if (ifcvf_config_features(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index d545a9411143..cb19196c3ece 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -69,7 +69,6 @@ struct ifcvf_hw {
phys_addr_t notify_base_pa;
u32 notify_off_multiplier;
u32 dev_type;
- u64 req_features;
u64 hw_features;
/* provisioned device features */
u64 dev_features;
@@ -122,7 +121,7 @@ u8 ifcvf_get_status(struct ifcvf_hw *hw);
void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
-u64 ifcvf_get_features(struct ifcvf_h
Re: [PATCH 1/5] virt queue ops take immediate actions
On 4/26/2023 11:39 AM, Jason Wang wrote:
在 2023/4/1 04:48, Zhu Lingshan 写道:
In this commit, virtqueue operations including:
set_vq_num(), set_vq_address(), set_vq_ready()
and get_vq_ready() access PCI registers directly
to take immediate actions.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 58 -
drivers/vdpa/ifcvf/ifcvf_base.h | 10 +++---
drivers/vdpa/ifcvf/ifcvf_main.c | 16 +++--
3 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5563b3a773c7..6c5650f73007 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -329,31 +329,49 @@ int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16
qid, u16 num)
return 0;
}
-static int ifcvf_hw_enable(struct ifcvf_hw *hw)
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num)
{
- struct virtio_pci_common_cfg __iomem *cfg;
- u32 i;
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
- cfg = hw->common_cfg;
- for (i = 0; i < hw->nr_vring; i++) {
- if (!hw->vring[i].ready)
- break;
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(num, &cfg->queue_size);
+}
- vp_iowrite16(i, &cfg->queue_select);
- vp_iowrite64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,
- &cfg->queue_desc_hi);
- vp_iowrite64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,
- &cfg->queue_avail_hi);
- vp_iowrite64_twopart(hw->vring[i].used, &cfg->queue_used_lo,
- &cfg->queue_used_hi);
- vp_iowrite16(hw->vring[i].size, &cfg->queue_size);
- ifcvf_set_vq_state(hw, i, hw->vring[i].last_avail_idx);
- vp_iowrite16(1, &cfg->queue_enable);
- }
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+ u64 driver_area, u64 device_area)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite64_twopart(desc_area, &cfg->queue_desc_lo,
+ &cfg->queue_desc_hi);
+ vp_iowrite64_twopart(driver_area, &cfg->queue_avail_lo,
+ &cfg->queue_avail_hi);
+ vp_iowrite64_twopart(device_area, &cfg->queue_used_lo,
+ &cfg->queue_used_hi);
return 0;
}
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u16 queue_enable;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ queue_enable = vp_ioread16(&cfg->queue_enable);
+
+ return (bool)queue_enable;
+}
+
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(ready, &cfg->queue_enable);
+}
+
static void ifcvf_hw_disable(struct ifcvf_hw *hw)
{
u32 i;
@@ -366,16 +384,12 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
int ifcvf_start_hw(struct ifcvf_hw *hw)
{
- ifcvf_reset(hw);
This seems unrelated to the immediate actions?
This is because we must not reset the device after vq settings already
take affections, e.g., we must not reset the hw after set_vq_address or
set_vq_ready.
Thanks
The rest looks good.
Thanks
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
if (ifcvf_config_features(hw) < 0)
return -EINVAL;
- if (ifcvf_hw_enable(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index c20d1c40214e..d545a9411143 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -47,12 +47,7 @@
#define MSIX_VECTOR_DEV_SHARED 3
struct vring_info {
- u64 desc;
- u64 avail;
- u64 used;
- u16 size;
u16 last_avail_idx;
- bool ready;
void __iomem *notify_addr;
phys_addr_t notify_pa;
u32 irq;
@@ -137,4 +132,9 @@ int ifcvf_probed_virtio_net(struct ifcvf_hw *hw);
u32 ifcvf_get_config_size(struct ifcvf_hw *hw);
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector);
u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector);
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num);
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+ u64 driver_area, u64 device_area);
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers
Re: [PATCH 0/5] vDPA/ifcvf: implement immediate initialization mechanism
On 4/24/2023 12:51 PM, Michael S. Tsirkin wrote: On Sat, Apr 01, 2023 at 04:48:49AM +0800, Zhu Lingshan wrote: Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. This series also implement irq synchronization in the reset routine Please, prefix each patch subject with vDPA/ifcvf: I will fix this in V2. Thanks Zhu Lingshan (5): virt queue ops take immediate actions get_driver_features from virito registers retire ifcvf_start_datapath and ifcvf_add_status synchronize irqs in the reset routine a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 162 +++- drivers/vdpa/ifcvf/ifcvf_base.h | 16 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 97 --- 3 files changed, 122 insertions(+), 153 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/5] get_driver_features from virito registers
On 4/24/2023 12:50 PM, Michael S. Tsirkin wrote:
subj typo: virtio
will fix in V2, thanks!
On Sat, Apr 01, 2023 at 04:48:51AM +0800, Zhu Lingshan wrote:
This commit implements a new function ifcvf_get_driver_feature()
which read driver_features from virtio registers.
To be less ambiguous, ifcvf_set_features() is renamed to
ifcvf_set_driver_features(), and ifcvf_get_features()
is renamed to ifcvf_get_dev_features() which returns
the provisioned vDPA device features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 38 +
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 9 +---
3 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 6c5650f73007..546e923bcd16 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -204,11 +204,29 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
return features;
}
-u64 ifcvf_get_features(struct ifcvf_hw *hw)
+/* return provisioned vDPA dev features */
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw)
{
return hw->dev_features;
}
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u32 features_lo, features_hi;
+ u64 features;
+
+ vp_iowrite32(0, &cfg->device_feature_select);
+ features_lo = vp_ioread32(&cfg->guest_feature);
+
+ vp_iowrite32(1, &cfg->device_feature_select);
+ features_hi = vp_ioread32(&cfg->guest_feature);
+
+ features = ((u64)features_hi << 32) | features_lo;
+
+ return features;
+}
+
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
@@ -275,7 +293,7 @@ void ifcvf_write_dev_config(struct ifcvf_hw *hw, u64 offset,
vp_iowrite8(*p++, hw->dev_cfg + offset + i);
}
-static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -286,19 +304,6 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
vp_iowrite32(features >> 32, &cfg->guest_feature);
}
-static int ifcvf_config_features(struct ifcvf_hw *hw)
-{
- ifcvf_set_features(hw, hw->req_features);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
-
- if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
- return -EIO;
- }
-
- return 0;
-}
-
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
@@ -387,9 +392,6 @@ int ifcvf_start_hw(struct ifcvf_hw *hw)
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
- if (ifcvf_config_features(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d545a9411143..cb19196c3ece 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -69,7 +69,6 @@ struct ifcvf_hw {
phys_addr_t notify_base_pa;
u32 notify_off_multiplier;
u32 dev_type;
- u64 req_features;
u64 hw_features;
/* provisioned device features */
u64 dev_features;
@@ -122,7 +121,7 @@ u8 ifcvf_get_status(struct ifcvf_hw *hw);
void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
-u64 ifcvf_get_features(struct ifcvf_hw *hw);
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
@@ -137,4 +136,6 @@ int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64
desc_area,
u64 driver_area, u64 device_area);
bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features);
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 1357c67014ab..4588484bd53d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -410,7 +410,7 @@ static u64 ifcvf_vdpa_get_device_features(struct
vdpa_device *vdpa_dev)
u64 features;
if (type == VIRTIO_ID_NET || type ==
Re: [PATCH 0/5] vDPA/ifcvf: implement immediate initialization mechanism
On 4/3/2023 6:10 PM, Zhu, Lingshan wrote: On 4/3/2023 1:28 PM, Jason Wang wrote: On Fri, Mar 31, 2023 at 8:49 PM Zhu Lingshan wrote: Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. Is there any chance that ifcvf can use virtio_pci_modern_dev library? Then we don't need to duplicate the codes. Note that pds_vdpa will be the second user for virtio_pci_modern_dev library (and the first vDPA parent to use that library). Yes I agree this library can help a lot for a standard virtio pci device. But this change would be huge, its like require to change every line of the driver. For example current driver functions work on the adapter and ifcvf_hw, if we wants to reuse the lib, we need the driver work on struct virtio_pci_modern_device. Almost need to re-write the driver. Can we plan this huge change in following series? ping Thanks, Zhu Lingshan Thanks This series also implement irq synchronization in the reset routine Zhu Lingshan (5): virt queue ops take immediate actions get_driver_features from virito registers retire ifcvf_start_datapath and ifcvf_add_status synchronize irqs in the reset routine a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 162 +++- drivers/vdpa/ifcvf/ifcvf_base.h | 16 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 97 --- 3 files changed, 122 insertions(+), 153 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/5] vDPA/ifcvf: implement immediate initialization mechanism
On 4/3/2023 1:28 PM, Jason Wang wrote: On Fri, Mar 31, 2023 at 8:49 PM Zhu Lingshan wrote: Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. Is there any chance that ifcvf can use virtio_pci_modern_dev library? Then we don't need to duplicate the codes. Note that pds_vdpa will be the second user for virtio_pci_modern_dev library (and the first vDPA parent to use that library). Yes I agree this library can help a lot for a standard virtio pci device. But this change would be huge, its like require to change every line of the driver. For example current driver functions work on the adapter and ifcvf_hw, if we wants to reuse the lib, we need the driver work on struct virtio_pci_modern_device. Almost need to re-write the driver. Can we plan this huge change in following series? Thanks, Zhu Lingshan Thanks This series also implement irq synchronization in the reset routine Zhu Lingshan (5): virt queue ops take immediate actions get_driver_features from virito registers retire ifcvf_start_datapath and ifcvf_add_status synchronize irqs in the reset routine a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 162 +++- drivers/vdpa/ifcvf/ifcvf_base.h | 16 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 97 --- 3 files changed, 122 insertions(+), 153 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 0/5] vDPA/ifcvf: implement immediate initialization mechanism
Formerly, ifcvf driver has implemented a lazy-initialization mechanism for the virtqueues and other config space contents, it would store all configurations that passed down from the userspace, then load them to the device config space upon DRIVER_OK. This can not serve live migration, so this series implement an immediate initialization mechanism, which means rather than the former store-load process, the virtio operations like vq ops would take immediate actions by access the virtio registers. This series also implement irq synchronization in the reset routine Zhu Lingshan (5): virt queue ops take immediate actions get_driver_features from virito registers retire ifcvf_start_datapath and ifcvf_add_status synchronize irqs in the reset routine a vendor driver should not set _CONFIG_S_FAILED drivers/vdpa/ifcvf/ifcvf_base.c | 162 +++- drivers/vdpa/ifcvf/ifcvf_base.h | 16 ++-- drivers/vdpa/ifcvf/ifcvf_main.c | 97 --- 3 files changed, 122 insertions(+), 153 deletions(-) -- 2.39.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 5/5] a vendor driver should not set _CONFIG_S_FAILED
VIRTIO_CONFIG_S_FAILED indicates the guest driver has given up
the device due to fatal errors. So it is the guest decision,
the vendor driver should not set this status to the device.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_main.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 15c6157ee841..f228fba74c61 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -423,9 +423,7 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
!(status_old & VIRTIO_CONFIG_S_DRIVER_OK)) {
ret = ifcvf_request_irq(vf);
if (ret) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
+ IFCVF_ERR(vf->pdev, "failed to request irq with error
%d\n", ret);
return;
}
}
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 2/5] get_driver_features from virito registers
This commit implements a new function ifcvf_get_driver_feature()
which read driver_features from virtio registers.
To be less ambiguous, ifcvf_set_features() is renamed to
ifcvf_set_driver_features(), and ifcvf_get_features()
is renamed to ifcvf_get_dev_features() which returns
the provisioned vDPA device features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 38 +
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 9 +---
3 files changed, 29 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 6c5650f73007..546e923bcd16 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -204,11 +204,29 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
return features;
}
-u64 ifcvf_get_features(struct ifcvf_hw *hw)
+/* return provisioned vDPA dev features */
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw)
{
return hw->dev_features;
}
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u32 features_lo, features_hi;
+ u64 features;
+
+ vp_iowrite32(0, &cfg->device_feature_select);
+ features_lo = vp_ioread32(&cfg->guest_feature);
+
+ vp_iowrite32(1, &cfg->device_feature_select);
+ features_hi = vp_ioread32(&cfg->guest_feature);
+
+ features = ((u64)features_hi << 32) | features_lo;
+
+ return features;
+}
+
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
@@ -275,7 +293,7 @@ void ifcvf_write_dev_config(struct ifcvf_hw *hw, u64 offset,
vp_iowrite8(*p++, hw->dev_cfg + offset + i);
}
-static void ifcvf_set_features(struct ifcvf_hw *hw, u64 features)
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -286,19 +304,6 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64
features)
vp_iowrite32(features >> 32, &cfg->guest_feature);
}
-static int ifcvf_config_features(struct ifcvf_hw *hw)
-{
- ifcvf_set_features(hw, hw->req_features);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
-
- if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
- return -EIO;
- }
-
- return 0;
-}
-
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid)
{
struct ifcvf_lm_cfg __iomem *ifcvf_lm;
@@ -387,9 +392,6 @@ int ifcvf_start_hw(struct ifcvf_hw *hw)
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
- if (ifcvf_config_features(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d545a9411143..cb19196c3ece 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -69,7 +69,6 @@ struct ifcvf_hw {
phys_addr_t notify_base_pa;
u32 notify_off_multiplier;
u32 dev_type;
- u64 req_features;
u64 hw_features;
/* provisioned device features */
u64 dev_features;
@@ -122,7 +121,7 @@ u8 ifcvf_get_status(struct ifcvf_hw *hw);
void ifcvf_set_status(struct ifcvf_hw *hw, u8 status);
void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
-u64 ifcvf_get_features(struct ifcvf_hw *hw);
+u64 ifcvf_get_dev_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
@@ -137,4 +136,6 @@ int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64
desc_area,
u64 driver_area, u64 device_area);
bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
+void ifcvf_set_driver_features(struct ifcvf_hw *hw, u64 features);
+u64 ifcvf_get_driver_features(struct ifcvf_hw *hw);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 1357c67014ab..4588484bd53d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -410,7 +410,7 @@ static u64 ifcvf_vdpa_get_device_features(struct
vdpa_device *vdpa_dev)
u64 features;
if (type == VIRTIO_ID_NET || type == VIRTIO_ID_BLOCK)
- features = ifcvf_get_features(vf);
+ features = ifcvf_get_dev_features(vf);
else {
features = 0;
[PATCH 3/5] retire ifcvf_start_datapath and ifcvf_add_status
Rather than former lazy-initialization mechanism,
now the virtqueue operations and driver_features related
ops access the virtio registers directly to take
immediate actions. So ifcvf_start_datapath() should
retire.
ifcvf_add_status() is retierd because we should not change
device status by a vendor driver's decision, this driver should
only set device status which is from virito drivers
upon vdpa_ops.set_status()
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 19 ---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 -
drivers/vdpa/ifcvf/ifcvf_main.c | 23 ---
3 files changed, 43 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 546e923bcd16..79e313c5e10e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -178,15 +178,6 @@ void ifcvf_reset(struct ifcvf_hw *hw)
ifcvf_get_status(hw);
}
-static void ifcvf_add_status(struct ifcvf_hw *hw, u8 status)
-{
- if (status != 0)
- status |= ifcvf_get_status(hw);
-
- ifcvf_set_status(hw, status);
- ifcvf_get_status(hw);
-}
-
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -387,16 +378,6 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
}
}
-int ifcvf_start_hw(struct ifcvf_hw *hw)
-{
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
-
- ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
-
- return 0;
-}
-
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
ifcvf_hw_disable(hw);
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index cb19196c3ece..d34d3bc0dbf4 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -110,7 +110,6 @@ struct ifcvf_vdpa_mgmt_dev {
};
int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *dev);
-int ifcvf_start_hw(struct ifcvf_hw *hw);
void ifcvf_stop_hw(struct ifcvf_hw *hw);
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid);
void ifcvf_read_dev_config(struct ifcvf_hw *hw, u64 offset,
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 4588484bd53d..968687159e44 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,22 +346,6 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_start_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- u8 status;
- int ret;
-
- ret = ifcvf_start_hw(vf);
- if (ret < 0) {
- status = ifcvf_get_status(vf);
- status |= VIRTIO_CONFIG_S_FAILED;
- ifcvf_set_status(vf, status);
- }
-
- return ret;
-}
-
static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
{
struct ifcvf_hw *vf = adapter->vf;
@@ -452,13 +436,11 @@ static u8 ifcvf_vdpa_get_status(struct vdpa_device
*vdpa_dev)
static void ifcvf_vdpa_set_status(struct vdpa_device *vdpa_dev, u8 status)
{
- struct ifcvf_adapter *adapter;
struct ifcvf_hw *vf;
u8 status_old;
int ret;
vf = vdpa_to_vf(vdpa_dev);
- adapter = vdpa_to_adapter(vdpa_dev);
status_old = ifcvf_get_status(vf);
if (status_old == status)
@@ -473,11 +455,6 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
ifcvf_set_status(vf, status);
return;
}
-
- if (ifcvf_start_datapath(adapter) < 0)
- IFCVF_ERR(adapter->pdev,
- "Failed to set ifcvf vdpa status %u\n",
- status);
}
ifcvf_set_status(vf, status);
--
2.39.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 1/5] virt queue ops take immediate actions
In this commit, virtqueue operations including:
set_vq_num(), set_vq_address(), set_vq_ready()
and get_vq_ready() access PCI registers directly
to take immediate actions.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 58 -
drivers/vdpa/ifcvf/ifcvf_base.h | 10 +++---
drivers/vdpa/ifcvf/ifcvf_main.c | 16 +++--
3 files changed, 45 insertions(+), 39 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 5563b3a773c7..6c5650f73007 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -329,31 +329,49 @@ int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16
num)
return 0;
}
-static int ifcvf_hw_enable(struct ifcvf_hw *hw)
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num)
{
- struct virtio_pci_common_cfg __iomem *cfg;
- u32 i;
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
- cfg = hw->common_cfg;
- for (i = 0; i < hw->nr_vring; i++) {
- if (!hw->vring[i].ready)
- break;
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(num, &cfg->queue_size);
+}
- vp_iowrite16(i, &cfg->queue_select);
- vp_iowrite64_twopart(hw->vring[i].desc, &cfg->queue_desc_lo,
-&cfg->queue_desc_hi);
- vp_iowrite64_twopart(hw->vring[i].avail, &cfg->queue_avail_lo,
- &cfg->queue_avail_hi);
- vp_iowrite64_twopart(hw->vring[i].used, &cfg->queue_used_lo,
-&cfg->queue_used_hi);
- vp_iowrite16(hw->vring[i].size, &cfg->queue_size);
- ifcvf_set_vq_state(hw, i, hw->vring[i].last_avail_idx);
- vp_iowrite16(1, &cfg->queue_enable);
- }
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite64_twopart(desc_area, &cfg->queue_desc_lo,
+&cfg->queue_desc_hi);
+ vp_iowrite64_twopart(driver_area, &cfg->queue_avail_lo,
+&cfg->queue_avail_hi);
+ vp_iowrite64_twopart(device_area, &cfg->queue_used_lo,
+&cfg->queue_used_hi);
return 0;
}
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ u16 queue_enable;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ queue_enable = vp_ioread16(&cfg->queue_enable);
+
+ return (bool)queue_enable;
+}
+
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+
+ vp_iowrite16(qid, &cfg->queue_select);
+ vp_iowrite16(ready, &cfg->queue_enable);
+}
+
static void ifcvf_hw_disable(struct ifcvf_hw *hw)
{
u32 i;
@@ -366,16 +384,12 @@ static void ifcvf_hw_disable(struct ifcvf_hw *hw)
int ifcvf_start_hw(struct ifcvf_hw *hw)
{
- ifcvf_reset(hw);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_ACKNOWLEDGE);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER);
if (ifcvf_config_features(hw) < 0)
return -EINVAL;
- if (ifcvf_hw_enable(hw) < 0)
- return -EINVAL;
-
ifcvf_add_status(hw, VIRTIO_CONFIG_S_DRIVER_OK);
return 0;
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index c20d1c40214e..d545a9411143 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -47,12 +47,7 @@
#define MSIX_VECTOR_DEV_SHARED 3
struct vring_info {
- u64 desc;
- u64 avail;
- u64 used;
- u16 size;
u16 last_avail_idx;
- bool ready;
void __iomem *notify_addr;
phys_addr_t notify_pa;
u32 irq;
@@ -137,4 +132,9 @@ int ifcvf_probed_virtio_net(struct ifcvf_hw *hw);
u32 ifcvf_get_config_size(struct ifcvf_hw *hw);
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector);
u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector);
+void ifcvf_set_vq_num(struct ifcvf_hw *hw, u16 qid, u32 num);
+int ifcvf_set_vq_address(struct ifcvf_hw *hw, u16 qid, u64 desc_area,
+u64 driver_area, u64 device_area);
+bool ifcvf_get_vq_ready(struct ifcvf_hw *hw, u16 qid);
+void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid, bool ready);
#endif /* _IFCVF_H_ */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7f78c47e40d6..1357c67014ab 100644
-
[PATCH 4/5] synchronize irqs in the reset routine
This commit synchronize irqs of the virtqueues
and config space in the reset routine.
Thus ifcvf_stop_hw() and reset() are refactored as well.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 61 ++---
drivers/vdpa/ifcvf/ifcvf_main.c | 45 +++-
2 files changed, 54 insertions(+), 52 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 79e313c5e10e..49949aec20ef 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -170,12 +170,7 @@ void ifcvf_set_status(struct ifcvf_hw *hw, u8 status)
void ifcvf_reset(struct ifcvf_hw *hw)
{
- hw->config_cb.callback = NULL;
- hw->config_cb.private = NULL;
-
ifcvf_set_status(hw, 0);
- /* flush set_status, make sure VF is stopped, reset */
- ifcvf_get_status(hw);
}
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
@@ -368,20 +363,62 @@ void ifcvf_set_vq_ready(struct ifcvf_hw *hw, u16 qid,
bool ready)
vp_iowrite16(ready, &cfg->queue_enable);
}
-static void ifcvf_hw_disable(struct ifcvf_hw *hw)
+static void synchronize_per_vq_irq(struct ifcvf_hw *hw)
{
- u32 i;
+ u16 qid;
- ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
- for (i = 0; i < hw->nr_vring; i++) {
- ifcvf_set_vq_vector(hw, i, VIRTIO_MSI_NO_VECTOR);
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ if (hw->vring[qid].irq != -EINVAL)
+ synchronize_irq(hw->vring[qid].irq);
}
}
+static void synchronize_vqs_reused_irq(struct ifcvf_hw *hw)
+{
+ if (hw->vqs_reused_irq != -EINVAL)
+ synchronize_irq(hw->vqs_reused_irq);
+}
+
+static void synchronize_vq_irq(struct ifcvf_hw *hw)
+{
+ u8 status = hw->msix_vector_status;
+
+ if (status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
+ synchronize_per_vq_irq(hw);
+ else
+ synchronize_vqs_reused_irq(hw);
+}
+
+static void synchronize_config_irq(struct ifcvf_hw *hw)
+{
+ if (hw->config_irq != -EINVAL)
+ synchronize_irq(hw->config_irq);
+}
+
+static void ifcvf_reset_vring(struct ifcvf_hw *hw)
+{
+ u16 qid;
+
+ for (qid = 0; qid < hw->nr_vring; qid++) {
+ synchronize_vq_irq(hw);
+ hw->vring[qid].cb.callback = NULL;
+ hw->vring[qid].cb.private = NULL;
+ ifcvf_set_vq_vector(hw, qid, VIRTIO_MSI_NO_VECTOR);
+ }
+}
+
+static void ifcvf_reset_config_handler(struct ifcvf_hw *hw)
+{
+ synchronize_config_irq(hw);
+ hw->config_cb.callback = NULL;
+ hw->config_cb.private = NULL;
+ ifcvf_set_config_vector(hw, VIRTIO_MSI_NO_VECTOR);
+}
+
void ifcvf_stop_hw(struct ifcvf_hw *hw)
{
- ifcvf_hw_disable(hw);
- ifcvf_reset(hw);
+ ifcvf_reset_vring(hw);
+ ifcvf_reset_config_handler(hw);
}
void ifcvf_notify_queue(struct ifcvf_hw *hw, u16 qid)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 968687159e44..15c6157ee841 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,33 +346,6 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++)
- vf->vring[i].cb.callback = NULL;
-
- ifcvf_stop_hw(vf);
-
- return 0;
-}
-
-static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
-{
- struct ifcvf_hw *vf = adapter->vf;
- int i;
-
- for (i = 0; i < vf->nr_vring; i++) {
- vf->vring[i].last_avail_idx = 0;
- vf->vring[i].cb.callback = NULL;
- vf->vring[i].cb.private = NULL;
- }
-
- ifcvf_reset(vf);
-}
-
static struct ifcvf_adapter *vdpa_to_adapter(struct vdpa_device *vdpa_dev)
{
return container_of(vdpa_dev, struct ifcvf_adapter, vdpa);
@@ -462,23 +435,15 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_dev)
{
- struct ifcvf_adapter *adapter;
- struct ifcvf_hw *vf;
- u8 status_old;
-
- vf = vdpa_to_vf(vdpa_dev);
- adapter = vdpa_to_adapter(vdpa_dev);
- status_old = ifcvf_get_status(vf);
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+ u8 status = ifcvf_get_status(vf);
- if (status_old == 0)
- return 0;
+ ifcvf_stop_hw(vf);
- if (status_old & VIRTIO_CONFIG_S_DRIVER_OK) {
- ifcvf_stop_datapath(adapter);
+ if (status & VIRTIO_CONFIG_S_DRIVER_OK)
ifcvf_free_irq(vf);
- }
- ifcvf_reset_vring(adapter);
+ ifcvf_reset(vf);
return 0;
}
--
2.39.1
Re: [PATCH] vdpa: ifcvf: Do proper cleanup if IFCVF init fails
On 12/28/2022 2:58 PM, Jason Wang wrote:
On Wed, Dec 28, 2022 at 5:15 AM Tanmay Bhushan <0070472...@gmail.com> wrote:
From 7eae04667ddaac8baa4812d48ef2c942cedef946 Mon Sep 17 00:00:00 2001
From: Tanmay Bhushan <0070472...@gmail.com>
Date: Tue, 27 Dec 2022 22:02:16 +0100
Subject: [PATCH] vdpa: ifcvf: Do proper cleanup if IFCVF init fails
ifcvf_mgmt_dev leaks memory if it is not freed before
returning. Call is made to correct return statement
so memory does not leak. ifcvf_init_hw does not take
care of this so it is needed to do it here.
Signed-off-by: Tanmay Bhushan <0070472...@gmail.com>
Acked-by: Jason Wang
Thanks
Acked-by: Zhu Lingshan
Thanks
---
drivers/vdpa/ifcvf/ifcvf_main.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index f9c0044c6442..44b29289aa19 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -849,7 +849,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const
struct pci_device_id *id)
ret = ifcvf_init_hw(vf, pdev);
if (ret) {
IFCVF_ERR(pdev, "Failed to init IFCVF hw\n");
- return ret;
+ goto err;
}
for (i = 0; i < vf->nr_vring; i++)
--
2.34.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 00/12] ifcvf/vDPA implement features provisioning
On 12/13/2022 2:57 PM, Zhu, Lingshan wrote: On 12/6/2022 4:25 PM, Jason Wang wrote: On Fri, Nov 25, 2022 at 11:06 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. Examples: a)The management device supported features: $ vdpa mgmtdev show pci/:01:00.5 pci/:01:00.5: supported_classes net max_supported_vqs 9 dev_features MTU MAC MRG_RXBUF CTRL_VQ MQ ANY_LAYOUT VERSION_1 ACCESS_PLATFORM b)Provision a vDPA device with all supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 $ vdpa/vdpa dev config show vdpa0 vdpa0: mac 00:e8:ca:11:be:05 link up link_announce false max_vq_pairs 4 mtu 1500 negotiated_features MRG_RXBUF CTRL_VQ MQ VERSION_1 ACCESS_PLATFORM c)Provision a vDPA device with a subset of the supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 device_features 0x300020020 $ vdpa dev config show vdpa0 mac 00:e8:ca:11:be:05 link up link_announce false negotiated_features CTRL_VQ VERSION_1 ACCESS_PLATFORM Please help review Thanks Changes from V1: split original patch 1 ~ patch 3 to small patches that are less than 100 lines, True but. so they can be applied to stalbe kernel(Jason) It requires each patch fixes a real issue so I think those can not go to -stable. Btw, looking at git history what you want to decouple is basically functional equivalent to a partial revert of this commit: commit 378b2e956820ff5c082d05f42828badcfbabb614 Author: Zhu Lingshan Date: Fri Jul 22 19:53:05 2022 +0800 vDPA/ifcvf: support userspace to query features and MQ of a management device Adapting to current netlink interfaces, this commit allows userspace to query feature bits and MQ capability of a management device. Currently both the vDPA device and the management device are the VF itself, thus this ifcvf should initialize the virtio capabilities in probe() before setting up the struct vdpa_mgmt_dev. Signed-off-by: Zhu Lingshan Message-Id: <20220722115309.82746-3-lingshan@intel.com> Signed-off-by: Michael S. Tsirkin Before this commit. adapter was allocated/freed in device_add/del which should be fine. Can we go back to doing things that way? Hi Jason Thanks for your advice, my concern is, even revert this commit 378b2e95, we still need to re-implement the feature "support userspace to query features and MQ of a management device" in stable kernel(still not a patch fix something but implement something), or we remove a feature in the stable kernel. And there may still need to split patches to meet the <100 lines requirement The reason why I place the adapter allocation in probe is that currently the management device is the VF itself, move it from dev_add to probe can lighten the organization of data structures, it is not a good design anyway, so this series tries to fix them as well. Maybe not to to sable Thanks Hi Jason, Ping, shall I drop the "cc stable" tag and resend? Maybe squash the patches because they are not going to stable tree? Thanks Thanks Zhu Lingshan (12): vDPA/ifcvf: decouple hw features manipulators from the adapter vDPA/ifcvf: decouple config space ops from the adapter vDPA/ifcvf: alloc the mgmt_dev before the adapter vDPA/ifcvf: decouple vq IRQ releasers from the adapter vDPA/ifcvf: decouple config IRQ releaser from the adapter vDPA/ifcvf: decouple vq irq requester from the adapter vDPA/ifcvf: decouple config/dev IRQ requester and vectors allocator from the adapter vDPA/ifcvf: ifcvf_request_irq works on ifcvf_hw vDPA/ifcvf: manage ifcvf_hw in the mgmt_dev vDPA/ifcvf: allocate the adapter in dev_add() vDPA/ifcvf: retire ifcvf_private_to_vf vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 162 +++- 3 files changed, 91 insertions(+), 113 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 00/12] ifcvf/vDPA implement features provisioning
On 12/6/2022 4:25 PM, Jason Wang wrote: On Fri, Nov 25, 2022 at 11:06 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. Examples: a)The management device supported features: $ vdpa mgmtdev show pci/:01:00.5 pci/:01:00.5: supported_classes net max_supported_vqs 9 dev_features MTU MAC MRG_RXBUF CTRL_VQ MQ ANY_LAYOUT VERSION_1 ACCESS_PLATFORM b)Provision a vDPA device with all supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 $ vdpa/vdpa dev config show vdpa0 vdpa0: mac 00:e8:ca:11:be:05 link up link_announce false max_vq_pairs 4 mtu 1500 negotiated_features MRG_RXBUF CTRL_VQ MQ VERSION_1 ACCESS_PLATFORM c)Provision a vDPA device with a subset of the supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 device_features 0x300020020 $ vdpa dev config show vdpa0 mac 00:e8:ca:11:be:05 link up link_announce false negotiated_features CTRL_VQ VERSION_1 ACCESS_PLATFORM Please help review Thanks Changes from V1: split original patch 1 ~ patch 3 to small patches that are less than 100 lines, True but. so they can be applied to stalbe kernel(Jason) It requires each patch fixes a real issue so I think those can not go to -stable. Btw, looking at git history what you want to decouple is basically functional equivalent to a partial revert of this commit: commit 378b2e956820ff5c082d05f42828badcfbabb614 Author: Zhu Lingshan Date: Fri Jul 22 19:53:05 2022 +0800 vDPA/ifcvf: support userspace to query features and MQ of a management device Adapting to current netlink interfaces, this commit allows userspace to query feature bits and MQ capability of a management device. Currently both the vDPA device and the management device are the VF itself, thus this ifcvf should initialize the virtio capabilities in probe() before setting up the struct vdpa_mgmt_dev. Signed-off-by: Zhu Lingshan Message-Id: <20220722115309.82746-3-lingshan@intel.com> Signed-off-by: Michael S. Tsirkin Before this commit. adapter was allocated/freed in device_add/del which should be fine. Can we go back to doing things that way? Hi Jason Thanks for your advice, my concern is, even revert this commit 378b2e95, we still need to re-implement the feature "support userspace to query features and MQ of a management device" in stable kernel(still not a patch fix something but implement something), or we remove a feature in the stable kernel. And there may still need to split patches to meet the <100 lines requirement The reason why I place the adapter allocation in probe is that currently the management device is the VF itself, move it from dev_add to probe can lighten the organization of data structures, it is not a good design anyway, so this series tries to fix them as well. Maybe not to to sable Thanks Thanks Zhu Lingshan (12): vDPA/ifcvf: decouple hw features manipulators from the adapter vDPA/ifcvf: decouple config space ops from the adapter vDPA/ifcvf: alloc the mgmt_dev before the adapter vDPA/ifcvf: decouple vq IRQ releasers from the adapter vDPA/ifcvf: decouple config IRQ releaser from the adapter vDPA/ifcvf: decouple vq irq requester from the adapter vDPA/ifcvf: decouple config/dev IRQ requester and vectors allocator from the adapter vDPA/ifcvf: ifcvf_request_irq works on ifcvf_hw vDPA/ifcvf: manage ifcvf_hw in the mgmt_dev vDPA/ifcvf: allocate the adapter in dev_add() vDPA/ifcvf: retire ifcvf_private_to_vf vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 162 +++- 3 files changed, 91 insertions(+), 113 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 12/12] vDPA/ifcvf: implement features provisioning
This commit implements features provisioning for ifcvf, that means:
1)checkk whether the provisioned features are supported by
the management device
2)vDPA device only presents selected feature bits
Examples:
a)The management device supported features:
$ vdpa mgmtdev show pci/:01:00.5
pci/:01:00.5:
supported_classes net
max_supported_vqs 9
dev_features MTU MAC MRG_RXBUF CTRL_VQ MQ ANY_LAYOUT VERSION_1 ACCESS_PLATFORM
b)Provision a vDPA device with all supported features:
$ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5
$ vdpa/vdpa dev config show vdpa0
vdpa0: mac 00:e8:ca:11:be:05 link up link_announce false max_vq_pairs 4 mtu 1500
negotiated_features MRG_RXBUF CTRL_VQ MQ VERSION_1 ACCESS_PLATFORM
c)Provision a vDPA device with a subset of the supported features:
$ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 device_features 0x300020020
$ vdpa dev config show vdpa0
mac 00:e8:ca:11:be:05 link up link_announce false
negotiated_features CTRL_VQ VERSION_1 ACCESS_PLATFORM
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 2 +-
drivers/vdpa/ifcvf/ifcvf_base.h | 3 +++
drivers/vdpa/ifcvf/ifcvf_main.c | 13 +
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 3ec5ca3aefe1..5563b3a773c7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -206,7 +206,7 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
u64 ifcvf_get_features(struct ifcvf_hw *hw)
{
- return hw->hw_features;
+ return hw->dev_features;
}
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index d41e255c581b..c20d1c40214e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -19,6 +19,7 @@
#include
#include
#include
+#include
#define N3000_DEVICE_ID0x1041
#define N3000_SUBSYS_DEVICE_ID 0x001A
@@ -75,6 +76,8 @@ struct ifcvf_hw {
u32 dev_type;
u64 req_features;
u64 hw_features;
+ /* provisioned device features */
+ u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
struct vring_info vring[IFCVF_MAX_QUEUES];
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 5fb3580594d5..cc826bfd3866 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -743,6 +743,7 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
struct vdpa_device *vdpa_dev;
struct pci_dev *pdev;
struct ifcvf_hw *vf;
+ u64 device_features;
int ret;
ifcvf_mgmt_dev = container_of(mdev, struct ifcvf_vdpa_mgmt_dev, mdev);
@@ -762,6 +763,17 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
adapter->vf = vf;
vdpa_dev = &adapter->vdpa;
+ device_features = vf->hw_features;
+ if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) {
+ if (config->device_features & ~device_features) {
+ IFCVF_ERR(pdev, "The provisioned features 0x%llx are
not supported by this device with features 0x%llx\n",
+ config->device_features, device_features);
+ return -EINVAL;
+ }
+ device_features &= config->device_features;
+ }
+ vf->dev_features = device_features;
+
if (name)
ret = dev_set_name(&vdpa_dev->dev, "%s", name);
else
@@ -866,6 +878,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
ifcvf_mgmt_dev->mdev.device = dev;
ifcvf_mgmt_dev->mdev.max_supported_vqs = vf->nr_vring;
ifcvf_mgmt_dev->mdev.supported_features = vf->hw_features;
+ ifcvf_mgmt_dev->mdev.config_attr_mask = (1 << VDPA_ATTR_DEV_FEATURES);
ret = vdpa_mgmtdev_register(&ifcvf_mgmt_dev->mdev);
if (ret) {
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 11/12] vDPA/ifcvf: retire ifcvf_private_to_vf
This commit retires ifcvf_private_to_vf, because
the vf is already a member of the adapter,
so it could be easily addressed by adapter->vf.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 3 ---
drivers/vdpa/ifcvf/ifcvf_main.c | 10 +-
2 files changed, 5 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 25bd4e927b27..d41e255c581b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -38,9 +38,6 @@
#define IFCVF_DBG(pdev, fmt, ...) dev_dbg(&pdev->dev, fmt, ##__VA_ARGS__)
#define IFCVF_INFO(pdev, fmt, ...) dev_info(&pdev->dev, fmt, ##__VA_ARGS__)
-#define ifcvf_private_to_vf(adapter) \
- (((struct ifcvf_adapter *)adapter)->vf)
-
/* all vqs and config interrupt has its own vector */
#define MSIX_VECTOR_PER_VQ_AND_CONFIG 1
/* all vqs share a vector, and config interrupt has a separate vector */
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 4450ddb53806..5fb3580594d5 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -346,9 +346,9 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_start_datapath(void *private)
+static int ifcvf_start_datapath(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(private);
+ struct ifcvf_hw *vf = adapter->vf;
u8 status;
int ret;
@@ -362,9 +362,9 @@ static int ifcvf_start_datapath(void *private)
return ret;
}
-static int ifcvf_stop_datapath(void *private)
+static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(private);
+ struct ifcvf_hw *vf = adapter->vf;
int i;
for (i = 0; i < vf->nr_vring; i++)
@@ -377,7 +377,7 @@ static int ifcvf_stop_datapath(void *private)
static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(adapter);
+ struct ifcvf_hw *vf = adapter->vf;
int i;
for (i = 0; i < vf->nr_vring; i++) {
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 08/12] vDPA/ifcvf: ifcvf_request_irq works on ifcvf_hw
All ifcvf_request_irq's callees are refactored
to work on ifcvf_hw, so it should be decoupled
from the adapter as well
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 8320bdacace8..cb3df395d3fb 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -314,9 +314,8 @@ static int ifcvf_request_config_irq(struct ifcvf_hw *vf)
return -EFAULT;
}
-static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_irq(struct ifcvf_hw *vf)
{
- struct ifcvf_hw *vf = &adapter->vf;
int nvectors, ret, max_intr;
nvectors = ifcvf_alloc_vectors(vf);
@@ -468,7 +467,7 @@ static void ifcvf_vdpa_set_status(struct vdpa_device
*vdpa_dev, u8 status)
if ((status & VIRTIO_CONFIG_S_DRIVER_OK) &&
!(status_old & VIRTIO_CONFIG_S_DRIVER_OK)) {
- ret = ifcvf_request_irq(adapter);
+ ret = ifcvf_request_irq(vf);
if (ret) {
status = ifcvf_get_status(vf);
status |= VIRTIO_CONFIG_S_FAILED;
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 10/12] vDPA/ifcvf: allocate the adapter in dev_add()
The adapter is the container of the vdpa_device,
this commits allocate the adapter in dev_add()
rather than in probe(). So that the vdpa_device()
could be re-created when the userspace creates
the vdpa device, and free-ed in dev_del()
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 34 +
1 file changed, 13 insertions(+), 21 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index b6f5f7a3a767..4450ddb53806 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -746,12 +746,20 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
int ret;
ifcvf_mgmt_dev = container_of(mdev, struct ifcvf_vdpa_mgmt_dev, mdev);
- if (!ifcvf_mgmt_dev->adapter)
- return -EOPNOTSUPP;
+ vf = &ifcvf_mgmt_dev->vf;
+ pdev = vf->pdev;
+ adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
+ &pdev->dev, &ifc_vdpa_ops, 1, 1, NULL,
false);
+ if (IS_ERR(adapter)) {
+ IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
+ return PTR_ERR(adapter);
+ }
- adapter = ifcvf_mgmt_dev->adapter;
- vf = adapter->vf;
- pdev = adapter->pdev;
+ ifcvf_mgmt_dev->adapter = adapter;
+ adapter->pdev = pdev;
+ adapter->vdpa.dma_dev = &pdev->dev;
+ adapter->vdpa.mdev = mdev;
+ adapter->vf = vf;
vdpa_dev = &adapter->vdpa;
if (name)
@@ -769,7 +777,6 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
return 0;
}
-
static void ifcvf_vdpa_dev_del(struct vdpa_mgmt_dev *mdev, struct vdpa_device
*dev)
{
struct ifcvf_vdpa_mgmt_dev *ifcvf_mgmt_dev;
@@ -788,7 +795,6 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
{
struct ifcvf_vdpa_mgmt_dev *ifcvf_mgmt_dev;
struct device *dev = &pdev->dev;
- struct ifcvf_adapter *adapter;
struct ifcvf_hw *vf;
u32 dev_type;
int ret, i;
@@ -825,24 +831,10 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
return -ENOMEM;
}
- adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
- dev, &ifc_vdpa_ops, 1, 1, NULL, false);
- if (IS_ERR(adapter)) {
- IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
- ret = PTR_ERR(adapter);
- goto err;
- }
-
- adapter->pdev = pdev;
- adapter->vdpa.dma_dev = &pdev->dev;
- adapter->vdpa.mdev = &ifcvf_mgmt_dev->mdev;
- ifcvf_mgmt_dev->adapter = adapter;
-
vf = &ifcvf_mgmt_dev->vf;
vf->dev_type = get_dev_type(pdev);
vf->base = pcim_iomap_table(pdev);
vf->pdev = pdev;
- adapter->vf = vf;
ret = ifcvf_init_hw(vf, pdev);
if (ret) {
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 09/12] vDPA/ifcvf: manage ifcvf_hw in the mgmt_dev
This commit allocates the hw structure in the
management device structure. So the hardware
can be initialized once the management device
is allocated in probe.
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +++--
drivers/vdpa/ifcvf/ifcvf_main.c | 7 ---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index e1fe947d61b7..25bd4e927b27 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -39,7 +39,7 @@
#define IFCVF_INFO(pdev, fmt, ...) dev_info(&pdev->dev, fmt, ##__VA_ARGS__)
#define ifcvf_private_to_vf(adapter) \
- (&((struct ifcvf_adapter *)adapter)->vf)
+ (((struct ifcvf_adapter *)adapter)->vf)
/* all vqs and config interrupt has its own vector */
#define MSIX_VECTOR_PER_VQ_AND_CONFIG 1
@@ -95,7 +95,7 @@ struct ifcvf_hw {
struct ifcvf_adapter {
struct vdpa_device vdpa;
struct pci_dev *pdev;
- struct ifcvf_hw vf;
+ struct ifcvf_hw *vf;
};
struct ifcvf_vring_lm_cfg {
@@ -110,6 +110,7 @@ struct ifcvf_lm_cfg {
struct ifcvf_vdpa_mgmt_dev {
struct vdpa_mgmt_dev mdev;
+ struct ifcvf_hw vf;
struct ifcvf_adapter *adapter;
struct pci_dev *pdev;
};
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index cb3df395d3fb..b6f5f7a3a767 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -402,7 +402,7 @@ static struct ifcvf_hw *vdpa_to_vf(struct vdpa_device
*vdpa_dev)
{
struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
- return &adapter->vf;
+ return adapter->vf;
}
static u64 ifcvf_vdpa_get_device_features(struct vdpa_device *vdpa_dev)
@@ -750,7 +750,7 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
return -EOPNOTSUPP;
adapter = ifcvf_mgmt_dev->adapter;
- vf = &adapter->vf;
+ vf = adapter->vf;
pdev = adapter->pdev;
vdpa_dev = &adapter->vdpa;
@@ -838,10 +838,11 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
adapter->vdpa.mdev = &ifcvf_mgmt_dev->mdev;
ifcvf_mgmt_dev->adapter = adapter;
- vf = &adapter->vf;
+ vf = &ifcvf_mgmt_dev->vf;
vf->dev_type = get_dev_type(pdev);
vf->base = pcim_iomap_table(pdev);
vf->pdev = pdev;
+ adapter->vf = vf;
ret = ifcvf_init_hw(vf, pdev);
if (ret) {
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 06/12] vDPA/ifcvf: decouple vq irq requester from the adapter
This commit decouples the vq irq requester from the adapter,
so that these functions can be invoked since probe.
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 19 ---
1 file changed, 8 insertions(+), 11 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index c635f78f5c4c..ee9c22975119 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -155,10 +155,9 @@ static int ifcvf_alloc_vectors(struct ifcvf_adapter
*adapter)
return ret;
}
-static int ifcvf_request_per_vq_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_per_vq_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vf->vqs_reused_irq = -EINVAL;
@@ -190,10 +189,9 @@ static int ifcvf_request_per_vq_irq(struct ifcvf_adapter
*adapter)
return -EFAULT;
}
-static int ifcvf_request_vqs_reused_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_vqs_reused_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vector = 0;
@@ -266,15 +264,14 @@ static int ifcvf_request_dev_irq(struct ifcvf_adapter
*adapter)
}
-static int ifcvf_request_vq_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_vq_irq(struct ifcvf_hw *vf)
{
- struct ifcvf_hw *vf = &adapter->vf;
int ret;
if (vf->msix_vector_status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
- ret = ifcvf_request_per_vq_irq(adapter);
+ ret = ifcvf_request_per_vq_irq(vf);
else
- ret = ifcvf_request_vqs_reused_irq(adapter);
+ ret = ifcvf_request_vqs_reused_irq(vf);
return ret;
}
@@ -341,7 +338,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
return ret;
}
- ret = ifcvf_request_vq_irq(adapter);
+ ret = ifcvf_request_vq_irq(vf);
if (ret)
return ret;
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 07/12] vDPA/ifcvf: decouple config/dev IRQ requester and vectors allocator from the adapter
This commit decouples the config irq requester, the device
shared irq requester and the MSI vectors allocator from
the adapter. So they can be safely invoked since probe
before the adapter is allocated.
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 21 +
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index ee9c22975119..8320bdacace8 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -132,10 +132,9 @@ static void ifcvf_free_irq(struct ifcvf_hw *vf)
* It returns the number of allocated vectors, negative
* return value when fails.
*/
-static int ifcvf_alloc_vectors(struct ifcvf_adapter *adapter)
+static int ifcvf_alloc_vectors(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int max_intr, ret;
/* all queues and config interrupt */
@@ -222,10 +221,9 @@ static int ifcvf_request_vqs_reused_irq(struct ifcvf_hw
*vf)
return -EFAULT;
}
-static int ifcvf_request_dev_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_dev_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vector = 0;
@@ -276,10 +274,9 @@ static int ifcvf_request_vq_irq(struct ifcvf_hw *vf)
return ret;
}
-static int ifcvf_request_config_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_config_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int config_vector, ret;
if (vf->msix_vector_status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
@@ -322,7 +319,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
struct ifcvf_hw *vf = &adapter->vf;
int nvectors, ret, max_intr;
- nvectors = ifcvf_alloc_vectors(adapter);
+ nvectors = ifcvf_alloc_vectors(vf);
if (nvectors <= 0)
return -EFAULT;
@@ -333,7 +330,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
if (nvectors == 1) {
vf->msix_vector_status = MSIX_VECTOR_DEV_SHARED;
- ret = ifcvf_request_dev_irq(adapter);
+ ret = ifcvf_request_dev_irq(vf);
return ret;
}
@@ -342,7 +339,7 @@ static int ifcvf_request_irq(struct ifcvf_adapter *adapter)
if (ret)
return ret;
- ret = ifcvf_request_config_irq(adapter);
+ ret = ifcvf_request_config_irq(vf);
if (ret)
return ret;
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 05/12] vDPA/ifcvf: decouple config IRQ releaser from the adapter
This commit decouples config IRQ releaser from the adapter,
so that it could be invoked once probe or in err handlers.
ifcvf_free_irq() works on ifcvf_hw in this commit
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 22 ++
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7dac0285b71d..c635f78f5c4c 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -101,10 +101,9 @@ static void ifcvf_free_vq_irq(struct ifcvf_hw *vf)
ifcvf_free_vqs_reused_irq(vf);
}
-static void ifcvf_free_config_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_config_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
if (vf->config_irq == -EINVAL)
return;
@@ -119,13 +118,12 @@ static void ifcvf_free_config_irq(struct ifcvf_adapter
*adapter)
}
}
-static void ifcvf_free_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
ifcvf_free_vq_irq(vf);
- ifcvf_free_config_irq(adapter);
+ ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
}
@@ -187,7 +185,7 @@ static int ifcvf_request_per_vq_irq(struct ifcvf_adapter
*adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
}
@@ -221,7 +219,7 @@ static int ifcvf_request_vqs_reused_irq(struct
ifcvf_adapter *adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
}
@@ -262,7 +260,7 @@ static int ifcvf_request_dev_irq(struct ifcvf_adapter
*adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
@@ -317,7 +315,7 @@ static int ifcvf_request_config_irq(struct ifcvf_adapter
*adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
}
@@ -508,7 +506,7 @@ static int ifcvf_vdpa_reset(struct vdpa_device *vdpa_dev)
if (status_old & VIRTIO_CONFIG_S_DRIVER_OK) {
ifcvf_stop_datapath(adapter);
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
}
ifcvf_reset_vring(adapter);
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 04/12] vDPA/ifcvf: decouple vq IRQ releasers from the adapter
This commit decouples the IRQ releasers from the
adapter, so that these functions could be
safely invoked once probe
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 21 +
1 file changed, 9 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 306a57c05509..7dac0285b71d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -69,10 +69,9 @@ static void ifcvf_free_irq_vectors(void *data)
pci_free_irq_vectors(data);
}
-static void ifcvf_free_per_vq_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_per_vq_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i;
for (i = 0; i < vf->nr_vring; i++) {
@@ -83,10 +82,9 @@ static void ifcvf_free_per_vq_irq(struct ifcvf_adapter
*adapter)
}
}
-static void ifcvf_free_vqs_reused_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_vqs_reused_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
if (vf->vqs_reused_irq != -EINVAL) {
devm_free_irq(&pdev->dev, vf->vqs_reused_irq, vf);
@@ -95,14 +93,12 @@ static void ifcvf_free_vqs_reused_irq(struct ifcvf_adapter
*adapter)
}
-static void ifcvf_free_vq_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_vq_irq(struct ifcvf_hw *vf)
{
- struct ifcvf_hw *vf = &adapter->vf;
-
if (vf->msix_vector_status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
- ifcvf_free_per_vq_irq(adapter);
+ ifcvf_free_per_vq_irq(vf);
else
- ifcvf_free_vqs_reused_irq(adapter);
+ ifcvf_free_vqs_reused_irq(vf);
}
static void ifcvf_free_config_irq(struct ifcvf_adapter *adapter)
@@ -126,8 +122,9 @@ static void ifcvf_free_config_irq(struct ifcvf_adapter
*adapter)
static void ifcvf_free_irq(struct ifcvf_adapter *adapter)
{
struct pci_dev *pdev = adapter->pdev;
+ struct ifcvf_hw *vf = &adapter->vf;
- ifcvf_free_vq_irq(adapter);
+ ifcvf_free_vq_irq(vf);
ifcvf_free_config_irq(adapter);
ifcvf_free_irq_vectors(pdev);
}
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 03/12] vDPA/ifcvf: alloc the mgmt_dev before the adapter
This commit reverses the order of allocating the
management device and the adapter. So that it would
be possible to move the allocation of the adapter
to dev_add().
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_main.c | 31 ++-
1 file changed, 14 insertions(+), 17 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 28c82d796c90..306a57c05509 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -831,22 +831,30 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
}
pci_set_master(pdev);
+ ifcvf_mgmt_dev = kzalloc(sizeof(struct ifcvf_vdpa_mgmt_dev),
GFP_KERNEL);
+ if (!ifcvf_mgmt_dev) {
+ IFCVF_ERR(pdev, "Failed to alloc memory for the vDPA management
device\n");
+ return -ENOMEM;
+ }
adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
dev, &ifc_vdpa_ops, 1, 1, NULL, false);
if (IS_ERR(adapter)) {
IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
- return PTR_ERR(adapter);
+ ret = PTR_ERR(adapter);
+ goto err;
}
+ adapter->pdev = pdev;
+ adapter->vdpa.dma_dev = &pdev->dev;
+ adapter->vdpa.mdev = &ifcvf_mgmt_dev->mdev;
+ ifcvf_mgmt_dev->adapter = adapter;
+
vf = &adapter->vf;
vf->dev_type = get_dev_type(pdev);
vf->base = pcim_iomap_table(pdev);
vf->pdev = pdev;
- adapter->pdev = pdev;
- adapter->vdpa.dma_dev = &pdev->dev;
-
ret = ifcvf_init_hw(vf, pdev);
if (ret) {
IFCVF_ERR(pdev, "Failed to init IFCVF hw\n");
@@ -859,16 +867,6 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
vf->hw_features = ifcvf_get_hw_features(vf);
vf->config_size = ifcvf_get_config_size(vf);
- ifcvf_mgmt_dev = kzalloc(sizeof(struct ifcvf_vdpa_mgmt_dev),
GFP_KERNEL);
- if (!ifcvf_mgmt_dev) {
- IFCVF_ERR(pdev, "Failed to alloc memory for the vDPA management
device\n");
- return -ENOMEM;
- }
-
- ifcvf_mgmt_dev->mdev.ops = &ifcvf_vdpa_mgmt_dev_ops;
- ifcvf_mgmt_dev->mdev.device = dev;
- ifcvf_mgmt_dev->adapter = adapter;
-
dev_type = get_dev_type(pdev);
switch (dev_type) {
case VIRTIO_ID_NET:
@@ -883,12 +881,11 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
goto err;
}
+ ifcvf_mgmt_dev->mdev.ops = &ifcvf_vdpa_mgmt_dev_ops;
+ ifcvf_mgmt_dev->mdev.device = dev;
ifcvf_mgmt_dev->mdev.max_supported_vqs = vf->nr_vring;
ifcvf_mgmt_dev->mdev.supported_features = vf->hw_features;
- adapter->vdpa.mdev = &ifcvf_mgmt_dev->mdev;
-
-
ret = vdpa_mgmtdev_register(&ifcvf_mgmt_dev->mdev);
if (ret) {
IFCVF_ERR(pdev,
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 00/12] ifcvf/vDPA implement features provisioning
This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. Examples: a)The management device supported features: $ vdpa mgmtdev show pci/:01:00.5 pci/:01:00.5: supported_classes net max_supported_vqs 9 dev_features MTU MAC MRG_RXBUF CTRL_VQ MQ ANY_LAYOUT VERSION_1 ACCESS_PLATFORM b)Provision a vDPA device with all supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 $ vdpa/vdpa dev config show vdpa0 vdpa0: mac 00:e8:ca:11:be:05 link up link_announce false max_vq_pairs 4 mtu 1500 negotiated_features MRG_RXBUF CTRL_VQ MQ VERSION_1 ACCESS_PLATFORM c)Provision a vDPA device with a subset of the supported features: $ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 device_features 0x300020020 $ vdpa dev config show vdpa0 mac 00:e8:ca:11:be:05 link up link_announce false negotiated_features CTRL_VQ VERSION_1 ACCESS_PLATFORM Please help review Thanks Changes from V1: split original patch 1 ~ patch 3 to small patches that are less than 100 lines, so they can be applied to stalbe kernel(Jason) Zhu Lingshan (12): vDPA/ifcvf: decouple hw features manipulators from the adapter vDPA/ifcvf: decouple config space ops from the adapter vDPA/ifcvf: alloc the mgmt_dev before the adapter vDPA/ifcvf: decouple vq IRQ releasers from the adapter vDPA/ifcvf: decouple config IRQ releaser from the adapter vDPA/ifcvf: decouple vq irq requester from the adapter vDPA/ifcvf: decouple config/dev IRQ requester and vectors allocator from the adapter vDPA/ifcvf: ifcvf_request_irq works on ifcvf_hw vDPA/ifcvf: manage ifcvf_hw in the mgmt_dev vDPA/ifcvf: allocate the adapter in dev_add() vDPA/ifcvf: retire ifcvf_private_to_vf vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 162 +++- 3 files changed, 91 insertions(+), 113 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 02/12] vDPA/ifcvf: decouple config space ops from the adapter
This commit decopules the config space ops from the
adapter layer, so these functions can be invoked
once the device is probed.
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_base.c | 21 +
1 file changed, 5 insertions(+), 16 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 7a7e6ba66f88..3ec5ca3aefe1 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -10,11 +10,6 @@
#include "ifcvf_base.h"
-struct ifcvf_adapter *vf_to_adapter(struct ifcvf_hw *hw)
-{
- return container_of(hw, struct ifcvf_adapter, vf);
-}
-
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -37,8 +32,6 @@ u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector)
static void __iomem *get_cap_addr(struct ifcvf_hw *hw,
struct virtio_pci_cap *cap)
{
- struct ifcvf_adapter *ifcvf;
- struct pci_dev *pdev;
u32 length, offset;
u8 bar;
@@ -46,17 +39,14 @@ static void __iomem *get_cap_addr(struct ifcvf_hw *hw,
offset = le32_to_cpu(cap->offset);
bar = cap->bar;
- ifcvf= vf_to_adapter(hw);
- pdev = ifcvf->pdev;
-
if (bar >= IFCVF_PCI_MAX_RESOURCE) {
- IFCVF_DBG(pdev,
+ IFCVF_DBG(hw->pdev,
"Invalid bar number %u to get capabilities\n", bar);
return NULL;
}
- if (offset + length > pci_resource_len(pdev, bar)) {
- IFCVF_DBG(pdev,
+ if (offset + length > pci_resource_len(hw->pdev, bar)) {
+ IFCVF_DBG(hw->pdev,
"offset(%u) + len(%u) overflows bar%u's capability\n",
offset, length, bar);
return NULL;
@@ -92,6 +82,7 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
IFCVF_ERR(pdev, "Failed to read PCI capability list\n");
return -EIO;
}
+ hw->pdev = pdev;
while (pos) {
ret = ifcvf_read_config_range(pdev, (u32 *)&cap,
@@ -230,13 +221,11 @@ int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64
features)
u32 ifcvf_get_config_size(struct ifcvf_hw *hw)
{
- struct ifcvf_adapter *adapter;
u32 net_config_size = sizeof(struct virtio_net_config);
u32 blk_config_size = sizeof(struct virtio_blk_config);
u32 cap_size = hw->cap_dev_config_size;
u32 config_size;
- adapter = vf_to_adapter(hw);
/* If the onboard device config space size is greater than
* the size of struct virtio_net/blk_config, only the spec
* implementing contents size is returned, this is very
@@ -251,7 +240,7 @@ u32 ifcvf_get_config_size(struct ifcvf_hw *hw)
break;
default:
config_size = 0;
- IFCVF_ERR(adapter->pdev, "VIRTIO ID %u not supported\n",
hw->dev_type);
+ IFCVF_ERR(hw->pdev, "VIRTIO ID %u not supported\n",
hw->dev_type);
}
return config_size;
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH V2 01/12] vDPA/ifcvf: decouple hw features manipulators from the adapter
This commit gets rid of ifcvf_adapter in hw features related
functions in ifcvf_base. Then these functions are more rubust
and de-coupling from the ifcvf_adapter layer. So these
functions could be invoded once the device is probed, even
before the adapter is allocaed.
Signed-off-by: Zhu Lingshan
Cc: sta...@vger.kernel.org
---
drivers/vdpa/ifcvf/ifcvf_base.c | 9 ++---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 1 +
3 files changed, 4 insertions(+), 7 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 3e4486bfa0b7..7a7e6ba66f88 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -220,10 +220,8 @@ u64 ifcvf_get_features(struct ifcvf_hw *hw)
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
- struct ifcvf_adapter *ifcvf = vf_to_adapter(hw);
-
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
- IFCVF_ERR(ifcvf->pdev, "VIRTIO_F_ACCESS_PLATFORM is not
negotiated\n");
+ IFCVF_ERR(hw->pdev, "VIRTIO_F_ACCESS_PLATFORM is not
negotiated\n");
return -EINVAL;
}
@@ -301,14 +299,11 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64
features)
static int ifcvf_config_features(struct ifcvf_hw *hw)
{
- struct ifcvf_adapter *ifcvf;
-
- ifcvf = vf_to_adapter(hw);
ifcvf_set_features(hw, hw->req_features);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(ifcvf->pdev, "Failed to set FEATURES_OK status\n");
+ IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
return -EIO;
}
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index f5563f665cc6..e1fe947d61b7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -89,6 +89,7 @@ struct ifcvf_hw {
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
u32 cap_dev_config_size;
+ struct pci_dev *pdev;
};
struct ifcvf_adapter {
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index f9c0044c6442..28c82d796c90 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -842,6 +842,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
vf = &adapter->vf;
vf->dev_type = get_dev_type(pdev);
vf->base = pcim_iomap_table(pdev);
+ vf->pdev = pdev;
adapter->pdev = pdev;
adapter->vdpa.dma_dev = &pdev->dev;
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/4] ifcvf/vDPA implement features provisioning
On 11/10/2022 5:13 PM, Jason Wang wrote: On Thu, Nov 10, 2022 at 4:59 PM Zhu, Lingshan wrote: On 11/10/2022 2:29 PM, Jason Wang wrote: 在 2022/11/10 14:20, Zhu, Lingshan 写道: On 11/10/2022 11:49 AM, Jason Wang wrote: On Wed, Nov 9, 2022 at 5:06 PM Zhu, Lingshan wrote: On 11/9/2022 4:59 PM, Jason Wang wrote: On Wed, Nov 9, 2022 at 4:14 PM Zhu, Lingshan wrote: On 11/9/2022 2:51 PM, Jason Wang wrote: On Mon, Nov 7, 2022 at 5:42 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. I don't see a direct relationship between the first 3 and the last. Maybe you can state the reason why the restructure is a must for the feature provisioning. Otherwise, we'd better split the series. When introducing features provisioning ability to ifcvf, there is a need to re-create vDPA devices on a VF with different feature bits. This seems a requirement even without feature provisioning? Device could be deleted from the management device anyhow. Yes, we need this to delete and re-create a vDPA device. I wonder if we need something that works for -stable. I can add a fix tag, so these three patches could apply to stable It's too huge for -stable. AFAIK, we can move the vdpa_alloc_device() from probe() to dev_add() and it seems to work? Yes and this is done in this series and that's why we need these refactoring code. I meant there's probably no need to change the association of existing structure but just do the allocation in dev_add(), then we will have a patch with much more small changeset that fit for -stable. Patch 1(ifcvf_base only work on ifcvf_hw) and patch 2(irq functions only work on ifcvf_hw) are not needed for stable. I have already done this allocation of ifcvf_adapter which is the container of struct vdpa_device in dev_add() in Patch 3, this should be merged to stable. Patch 3 is huge but necessary, not only allocate ifcvf_adapter in dev_add(), it also refactors the structures of ifcvf_mgmt_dev and ifcvf_adapter, because we need to initialize the VF's hw structure ifcvf_hw(which was a member of ifcvf_adapter but now should be a member of ifcvf_mgmt_dev) in probe. Is it still huge? Then please reorder the patches, stable-kernel-rules.rst said: - It cannot be bigger than 100 lines, with context. Let's see. It is over 180 lines, so maybe re-ordering can not help here, I will try to split patch 3. Thanks, Zhu Lingshan Thanks Thanks Thanks By the way, do you have any comments to the patches? Thanks, Zhu Lingshan Thanks We create vDPA device from a VF, so without features provisioning requirements, we don't need to re-create the vDPA device. But with features provisioning, it is a must now. Thanks Thakns When remove a vDPA device, the container of struct vdpa_device (here is ifcvf_adapter) is free-ed in dev_del() interface, so we need to allocate ifcvf_adapter in dev_add() than in probe(). That's why I have re-factored the adapter/mgmt_dev code. For re-factoring the irq related code and ifcvf_base, let them work on struct ifcvf_hw, the reason is that the adapter is allocated in dev_add(), if we want theses functions to work before dev_add(), like in probe, we need them work on ifcvf_hw than the adapter. Thanks Zhu Lingshan Thanks Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/4] ifcvf/vDPA implement features provisioning
On 11/10/2022 2:29 PM, Jason Wang wrote: 在 2022/11/10 14:20, Zhu, Lingshan 写道: On 11/10/2022 11:49 AM, Jason Wang wrote: On Wed, Nov 9, 2022 at 5:06 PM Zhu, Lingshan wrote: On 11/9/2022 4:59 PM, Jason Wang wrote: On Wed, Nov 9, 2022 at 4:14 PM Zhu, Lingshan wrote: On 11/9/2022 2:51 PM, Jason Wang wrote: On Mon, Nov 7, 2022 at 5:42 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. I don't see a direct relationship between the first 3 and the last. Maybe you can state the reason why the restructure is a must for the feature provisioning. Otherwise, we'd better split the series. When introducing features provisioning ability to ifcvf, there is a need to re-create vDPA devices on a VF with different feature bits. This seems a requirement even without feature provisioning? Device could be deleted from the management device anyhow. Yes, we need this to delete and re-create a vDPA device. I wonder if we need something that works for -stable. I can add a fix tag, so these three patches could apply to stable It's too huge for -stable. AFAIK, we can move the vdpa_alloc_device() from probe() to dev_add() and it seems to work? Yes and this is done in this series and that's why we need these refactoring code. I meant there's probably no need to change the association of existing structure but just do the allocation in dev_add(), then we will have a patch with much more small changeset that fit for -stable. Patch 1(ifcvf_base only work on ifcvf_hw) and patch 2(irq functions only work on ifcvf_hw) are not needed for stable. I have already done this allocation of ifcvf_adapter which is the container of struct vdpa_device in dev_add() in Patch 3, this should be merged to stable. Patch 3 is huge but necessary, not only allocate ifcvf_adapter in dev_add(), it also refactors the structures of ifcvf_mgmt_dev and ifcvf_adapter, because we need to initialize the VF's hw structure ifcvf_hw(which was a member of ifcvf_adapter but now should be a member of ifcvf_mgmt_dev) in probe. Is it still huge? Thanks Thanks By the way, do you have any comments to the patches? Thanks, Zhu Lingshan Thanks We create vDPA device from a VF, so without features provisioning requirements, we don't need to re-create the vDPA device. But with features provisioning, it is a must now. Thanks Thakns When remove a vDPA device, the container of struct vdpa_device (here is ifcvf_adapter) is free-ed in dev_del() interface, so we need to allocate ifcvf_adapter in dev_add() than in probe(). That's why I have re-factored the adapter/mgmt_dev code. For re-factoring the irq related code and ifcvf_base, let them work on struct ifcvf_hw, the reason is that the adapter is allocated in dev_add(), if we want theses functions to work before dev_add(), like in probe, we need them work on ifcvf_hw than the adapter. Thanks Zhu Lingshan Thanks Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/4] ifcvf/vDPA implement features provisioning
On 11/10/2022 11:49 AM, Jason Wang wrote: On Wed, Nov 9, 2022 at 5:06 PM Zhu, Lingshan wrote: On 11/9/2022 4:59 PM, Jason Wang wrote: On Wed, Nov 9, 2022 at 4:14 PM Zhu, Lingshan wrote: On 11/9/2022 2:51 PM, Jason Wang wrote: On Mon, Nov 7, 2022 at 5:42 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. I don't see a direct relationship between the first 3 and the last. Maybe you can state the reason why the restructure is a must for the feature provisioning. Otherwise, we'd better split the series. When introducing features provisioning ability to ifcvf, there is a need to re-create vDPA devices on a VF with different feature bits. This seems a requirement even without feature provisioning? Device could be deleted from the management device anyhow. Yes, we need this to delete and re-create a vDPA device. I wonder if we need something that works for -stable. I can add a fix tag, so these three patches could apply to stable AFAIK, we can move the vdpa_alloc_device() from probe() to dev_add() and it seems to work? Yes and this is done in this series and that's why we need these refactoring code. By the way, do you have any comments to the patches? Thanks, Zhu Lingshan Thanks We create vDPA device from a VF, so without features provisioning requirements, we don't need to re-create the vDPA device. But with features provisioning, it is a must now. Thanks Thakns When remove a vDPA device, the container of struct vdpa_device (here is ifcvf_adapter) is free-ed in dev_del() interface, so we need to allocate ifcvf_adapter in dev_add() than in probe(). That's why I have re-factored the adapter/mgmt_dev code. For re-factoring the irq related code and ifcvf_base, let them work on struct ifcvf_hw, the reason is that the adapter is allocated in dev_add(), if we want theses functions to work before dev_add(), like in probe, we need them work on ifcvf_hw than the adapter. Thanks Zhu Lingshan Thanks Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/4] ifcvf/vDPA implement features provisioning
On 11/9/2022 4:59 PM, Jason Wang wrote: On Wed, Nov 9, 2022 at 4:14 PM Zhu, Lingshan wrote: On 11/9/2022 2:51 PM, Jason Wang wrote: On Mon, Nov 7, 2022 at 5:42 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. I don't see a direct relationship between the first 3 and the last. Maybe you can state the reason why the restructure is a must for the feature provisioning. Otherwise, we'd better split the series. When introducing features provisioning ability to ifcvf, there is a need to re-create vDPA devices on a VF with different feature bits. This seems a requirement even without feature provisioning? Device could be deleted from the management device anyhow. Yes, we need this to delete and re-create a vDPA device. We create vDPA device from a VF, so without features provisioning requirements, we don't need to re-create the vDPA device. But with features provisioning, it is a must now. Thanks Thakns When remove a vDPA device, the container of struct vdpa_device (here is ifcvf_adapter) is free-ed in dev_del() interface, so we need to allocate ifcvf_adapter in dev_add() than in probe(). That's why I have re-factored the adapter/mgmt_dev code. For re-factoring the irq related code and ifcvf_base, let them work on struct ifcvf_hw, the reason is that the adapter is allocated in dev_add(), if we want theses functions to work before dev_add(), like in probe, we need them work on ifcvf_hw than the adapter. Thanks Zhu Lingshan Thanks Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 0/4] ifcvf/vDPA implement features provisioning
On 11/9/2022 2:51 PM, Jason Wang wrote: On Mon, Nov 7, 2022 at 5:42 PM Zhu Lingshan wrote: This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. I don't see a direct relationship between the first 3 and the last. Maybe you can state the reason why the restructure is a must for the feature provisioning. Otherwise, we'd better split the series. When introducing features provisioning ability to ifcvf, there is a need to re-create vDPA devices on a VF with different feature bits. When remove a vDPA device, the container of struct vdpa_device (here is ifcvf_adapter) is free-ed in dev_del() interface, so we need to allocate ifcvf_adapter in dev_add() than in probe(). That's why I have re-factored the adapter/mgmt_dev code. For re-factoring the irq related code and ifcvf_base, let them work on struct ifcvf_hw, the reason is that the adapter is allocated in dev_add(), if we want theses functions to work before dev_add(), like in probe, we need them work on ifcvf_hw than the adapter. Thanks Zhu Lingshan Thanks Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 4/4] vDPA/ifcvf: implement features provisioning
This commit implements features provisioning for ifcvf, that means:
1)checkk whether the provisioned features are supported by
the management device
2)vDPA device only presents selected feature bits
Examples:
a)The management device supported features:
$ vdpa mgmtdev show pci/:01:00.5
pci/:01:00.5:
supported_classes net
max_supported_vqs 9
dev_features MTU MAC MRG_RXBUF CTRL_VQ MQ ANY_LAYOUT VERSION_1 ACCESS_PLATFORM
b)Provision a vDPA device with all supported features:
$ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5
$ vdpa/vdpa dev config show vdpa0
vdpa0: mac 00:e8:ca:11:be:05 link up link_announce false max_vq_pairs 4 mtu 1500
negotiated_features MRG_RXBUF CTRL_VQ MQ VERSION_1 ACCESS_PLATFORM
c)Provision a vDPA device with a subset of the supported features:
$ vdpa dev add name vdpa0 mgmtdev pci/:01:00.5 device_features 0x300020020
$ vdpa dev config show vdpa0
mac 00:e8:ca:11:be:05 link up link_announce false
negotiated_features CTRL_VQ VERSION_1 ACCESS_PLATFORM
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 2 +-
drivers/vdpa/ifcvf/ifcvf_base.h | 3 +++
drivers/vdpa/ifcvf/ifcvf_main.c | 13 +
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 3ec5ca3aefe1..5563b3a773c7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -206,7 +206,7 @@ u64 ifcvf_get_hw_features(struct ifcvf_hw *hw)
u64 ifcvf_get_features(struct ifcvf_hw *hw)
{
- return hw->hw_features;
+ return hw->dev_features;
}
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 15d5badc7dbd..e2dff9a46388 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -19,6 +19,7 @@
#include
#include
#include
+#include
#define N3000_DEVICE_ID0x1041
#define N3000_SUBSYS_DEVICE_ID 0x001A
@@ -75,6 +76,8 @@ struct ifcvf_hw {
u32 dev_type;
u64 req_features;
u64 hw_features;
+ /* provisioned device features */
+ u64 dev_features;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *dev_cfg;
struct vring_info vring[IFCVF_MAX_QUEUES];
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 76ac324c271b..22bf7029399e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -745,6 +745,7 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
struct vdpa_device *vdpa_dev;
struct pci_dev *pdev;
struct ifcvf_hw *vf;
+ u64 device_features;
int ret;
ifcvf_mgmt_dev = container_of(mdev, struct ifcvf_vdpa_mgmt_dev, mdev);
@@ -764,6 +765,17 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
adapter->vf = vf;
vdpa_dev = &adapter->vdpa;
+ device_features = vf->hw_features;
+ if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) {
+ if (config->device_features & ~device_features) {
+ IFCVF_ERR(pdev, "The provisioned features 0x%llx are
not supported by this device with features 0x%llx\n",
+ config->device_features, device_features);
+ return -EINVAL;
+ }
+ device_features &= config->device_features;
+ }
+ vf->dev_features = device_features;
+
if (name)
ret = dev_set_name(&vdpa_dev->dev, "%s", name);
else
@@ -868,6 +880,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
ifcvf_mgmt_dev->mdev.max_supported_vqs = vf->nr_vring;
ifcvf_mgmt_dev->mdev.supported_features = vf->hw_features;
+ ifcvf_mgmt_dev->mdev.config_attr_mask = (1 << VDPA_ATTR_DEV_FEATURES);
ret = vdpa_mgmtdev_register(&ifcvf_mgmt_dev->mdev);
if (ret) {
--
2.31.1
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
[PATCH 3/4] vDPA/ifcvf: allocate ifcvf_adapter in dev_add()
This commits allocates ifcvf_adapter which is
a container of struct vdpa_device in dev_add()
interface than in probe()
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 5 +--
drivers/vdpa/ifcvf/ifcvf_main.c | 58 ++---
2 files changed, 26 insertions(+), 37 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index 63ce1f7f6841..15d5badc7dbd 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -38,9 +38,6 @@
#define IFCVF_DBG(pdev, fmt, ...) dev_dbg(&pdev->dev, fmt, ##__VA_ARGS__)
#define IFCVF_INFO(pdev, fmt, ...) dev_info(&pdev->dev, fmt, ##__VA_ARGS__)
-#define ifcvf_private_to_vf(adapter) \
- (&((struct ifcvf_adapter *)adapter)->vf)
-
/* all vqs and config interrupt has its own vector */
#define MSIX_VECTOR_PER_VQ_AND_CONFIG 1
/* all vqs share a vector, and config interrupt has a separate vector */
@@ -95,7 +92,7 @@ struct ifcvf_hw {
struct ifcvf_adapter {
struct vdpa_device vdpa;
struct pci_dev *pdev;
- struct ifcvf_hw vf;
+ struct ifcvf_hw *vf;
};
struct ifcvf_vring_lm_cfg {
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index bae518ff6234..76ac324c271b 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -347,9 +347,9 @@ static int ifcvf_request_irq(struct ifcvf_hw *vf)
return 0;
}
-static int ifcvf_start_datapath(void *private)
+static int ifcvf_start_datapath(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(private);
+ struct ifcvf_hw *vf = adapter->vf;
u8 status;
int ret;
@@ -363,9 +363,10 @@ static int ifcvf_start_datapath(void *private)
return ret;
}
-static int ifcvf_stop_datapath(void *private)
+static int ifcvf_stop_datapath(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(private);
+
+ struct ifcvf_hw *vf = adapter->vf;
int i;
for (i = 0; i < vf->nr_vring; i++)
@@ -378,7 +379,7 @@ static int ifcvf_stop_datapath(void *private)
static void ifcvf_reset_vring(struct ifcvf_adapter *adapter)
{
- struct ifcvf_hw *vf = ifcvf_private_to_vf(adapter);
+ struct ifcvf_hw *vf = adapter->vf;
int i;
for (i = 0; i < vf->nr_vring; i++) {
@@ -403,7 +404,7 @@ static struct ifcvf_hw *vdpa_to_vf(struct vdpa_device
*vdpa_dev)
{
struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
- return &adapter->vf;
+ return adapter->vf;
}
static u64 ifcvf_vdpa_get_device_features(struct vdpa_device *vdpa_dev)
@@ -747,12 +748,20 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
int ret;
ifcvf_mgmt_dev = container_of(mdev, struct ifcvf_vdpa_mgmt_dev, mdev);
- if (!ifcvf_mgmt_dev->adapter)
- return -EOPNOTSUPP;
+ vf = &ifcvf_mgmt_dev->vf;
+ pdev = vf->pdev;
+ adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
+ &pdev->dev, &ifc_vdpa_ops, 1, 1, NULL,
false);
+ if (IS_ERR(adapter)) {
+ IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
+ return PTR_ERR(adapter);
+ }
- adapter = ifcvf_mgmt_dev->adapter;
- vf = &adapter->vf;
- pdev = adapter->pdev;
+ ifcvf_mgmt_dev->adapter = adapter;
+ adapter->pdev = pdev;
+ adapter->vdpa.dma_dev = &pdev->dev;
+ adapter->vdpa.mdev = mdev;
+ adapter->vf = vf;
vdpa_dev = &adapter->vdpa;
if (name)
@@ -770,7 +779,6 @@ static int ifcvf_vdpa_dev_add(struct vdpa_mgmt_dev *mdev,
const char *name,
return 0;
}
-
static void ifcvf_vdpa_dev_del(struct vdpa_mgmt_dev *mdev, struct vdpa_device
*dev)
{
struct ifcvf_vdpa_mgmt_dev *ifcvf_mgmt_dev;
@@ -789,7 +797,6 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
{
struct ifcvf_vdpa_mgmt_dev *ifcvf_mgmt_dev;
struct device *dev = &pdev->dev;
- struct ifcvf_adapter *adapter;
struct ifcvf_hw *vf;
u32 dev_type;
int ret, i;
@@ -820,21 +827,16 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct
pci_device_id *id)
}
pci_set_master(pdev);
-
- adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
- dev, &ifc_vdpa_ops, 1, 1, NULL, false);
- if (IS_ERR(adapter)) {
- IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
- return PTR_ERR(adapter);
+ ifcvf_mgmt_dev = kzalloc(sizeof(struct ifcvf_vdpa_mgmt_dev),
GFP_KERNEL);
+ if (!ifcvf_mgmt_dev) {
+ IFCVF_ERR(pdev, "Failed to alloc memory for the vDPA management
dev
[PATCH 2/4] vDPA/ifcvf: IRQ interfaces work on ifcvf_hw
In this commit, ifcvf IRQ interfaces work on ifcvf_hw,
so these functions can be safely invoked before
the adapter struct is allocated since probe.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_main.c | 85 ++---
1 file changed, 37 insertions(+), 48 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index f9c0044c6442..bae518ff6234 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -69,10 +69,9 @@ static void ifcvf_free_irq_vectors(void *data)
pci_free_irq_vectors(data);
}
-static void ifcvf_free_per_vq_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_per_vq_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i;
for (i = 0; i < vf->nr_vring; i++) {
@@ -83,10 +82,9 @@ static void ifcvf_free_per_vq_irq(struct ifcvf_adapter
*adapter)
}
}
-static void ifcvf_free_vqs_reused_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_vqs_reused_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
if (vf->vqs_reused_irq != -EINVAL) {
devm_free_irq(&pdev->dev, vf->vqs_reused_irq, vf);
@@ -95,20 +93,18 @@ static void ifcvf_free_vqs_reused_irq(struct ifcvf_adapter
*adapter)
}
-static void ifcvf_free_vq_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_vq_irq(struct ifcvf_hw *vf)
{
- struct ifcvf_hw *vf = &adapter->vf;
if (vf->msix_vector_status == MSIX_VECTOR_PER_VQ_AND_CONFIG)
- ifcvf_free_per_vq_irq(adapter);
+ ifcvf_free_per_vq_irq(vf);
else
- ifcvf_free_vqs_reused_irq(adapter);
+ ifcvf_free_vqs_reused_irq(vf);
}
-static void ifcvf_free_config_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_config_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
if (vf->config_irq == -EINVAL)
return;
@@ -123,12 +119,12 @@ static void ifcvf_free_config_irq(struct ifcvf_adapter
*adapter)
}
}
-static void ifcvf_free_irq(struct ifcvf_adapter *adapter)
+static void ifcvf_free_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
+ struct pci_dev *pdev = vf->pdev;
- ifcvf_free_vq_irq(adapter);
- ifcvf_free_config_irq(adapter);
+ ifcvf_free_vq_irq(vf);
+ ifcvf_free_config_irq(vf);
ifcvf_free_irq_vectors(pdev);
}
@@ -137,10 +133,9 @@ static void ifcvf_free_irq(struct ifcvf_adapter *adapter)
* It returns the number of allocated vectors, negative
* return value when fails.
*/
-static int ifcvf_alloc_vectors(struct ifcvf_adapter *adapter)
+static int ifcvf_alloc_vectors(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int max_intr, ret;
/* all queues and config interrupt */
@@ -160,10 +155,9 @@ static int ifcvf_alloc_vectors(struct ifcvf_adapter
*adapter)
return ret;
}
-static int ifcvf_request_per_vq_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_per_vq_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vf->vqs_reused_irq = -EINVAL;
@@ -190,15 +184,14 @@ static int ifcvf_request_per_vq_irq(struct ifcvf_adapter
*adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
}
-static int ifcvf_request_vqs_reused_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_vqs_reused_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vector = 0;
@@ -224,15 +217,14 @@ static int ifcvf_request_vqs_reused_irq(struct
ifcvf_adapter *adapter)
return 0;
err:
- ifcvf_free_irq(adapter);
+ ifcvf_free_irq(vf);
return -EFAULT;
}
-static int ifcvf_request_dev_irq(struct ifcvf_adapter *adapter)
+static int ifcvf_request_dev_irq(struct ifcvf_hw *vf)
{
- struct pci_dev *pdev = adapter->pdev;
- struct ifcvf_hw *vf = &adapter->vf;
+ struct pci_dev *pdev = vf->pdev;
int i, vector, ret, irq;
vector = 0;
@@ -265,29 +257,27 @@ static int ifcvf_request_dev_irq(struct ifcvf_adapter
*adapter)
return 0;
err:
-
[PATCH 1/4] vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw
To be more rubust and low coupling in ifcvf base layer,
this commit gets rid of struct ifcvf_adapter in ifcvf_base
which is introduced in ifcvf_main.
Now ifcvf base layer interfaces work on ifcvf_hw only, so that
the base interfaces can be safely invoked since probe.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 30 +++---
drivers/vdpa/ifcvf/ifcvf_base.h | 2 ++
2 files changed, 9 insertions(+), 23 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c b/drivers/vdpa/ifcvf/ifcvf_base.c
index 3e4486bfa0b7..3ec5ca3aefe1 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -10,11 +10,6 @@
#include "ifcvf_base.h"
-struct ifcvf_adapter *vf_to_adapter(struct ifcvf_hw *hw)
-{
- return container_of(hw, struct ifcvf_adapter, vf);
-}
-
u16 ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector)
{
struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
@@ -37,8 +32,6 @@ u16 ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector)
static void __iomem *get_cap_addr(struct ifcvf_hw *hw,
struct virtio_pci_cap *cap)
{
- struct ifcvf_adapter *ifcvf;
- struct pci_dev *pdev;
u32 length, offset;
u8 bar;
@@ -46,17 +39,14 @@ static void __iomem *get_cap_addr(struct ifcvf_hw *hw,
offset = le32_to_cpu(cap->offset);
bar = cap->bar;
- ifcvf= vf_to_adapter(hw);
- pdev = ifcvf->pdev;
-
if (bar >= IFCVF_PCI_MAX_RESOURCE) {
- IFCVF_DBG(pdev,
+ IFCVF_DBG(hw->pdev,
"Invalid bar number %u to get capabilities\n", bar);
return NULL;
}
- if (offset + length > pci_resource_len(pdev, bar)) {
- IFCVF_DBG(pdev,
+ if (offset + length > pci_resource_len(hw->pdev, bar)) {
+ IFCVF_DBG(hw->pdev,
"offset(%u) + len(%u) overflows bar%u's capability\n",
offset, length, bar);
return NULL;
@@ -92,6 +82,7 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct pci_dev *pdev)
IFCVF_ERR(pdev, "Failed to read PCI capability list\n");
return -EIO;
}
+ hw->pdev = pdev;
while (pos) {
ret = ifcvf_read_config_range(pdev, (u32 *)&cap,
@@ -220,10 +211,8 @@ u64 ifcvf_get_features(struct ifcvf_hw *hw)
int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64 features)
{
- struct ifcvf_adapter *ifcvf = vf_to_adapter(hw);
-
if (!(features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)) && features) {
- IFCVF_ERR(ifcvf->pdev, "VIRTIO_F_ACCESS_PLATFORM is not
negotiated\n");
+ IFCVF_ERR(hw->pdev, "VIRTIO_F_ACCESS_PLATFORM is not
negotiated\n");
return -EINVAL;
}
@@ -232,13 +221,11 @@ int ifcvf_verify_min_features(struct ifcvf_hw *hw, u64
features)
u32 ifcvf_get_config_size(struct ifcvf_hw *hw)
{
- struct ifcvf_adapter *adapter;
u32 net_config_size = sizeof(struct virtio_net_config);
u32 blk_config_size = sizeof(struct virtio_blk_config);
u32 cap_size = hw->cap_dev_config_size;
u32 config_size;
- adapter = vf_to_adapter(hw);
/* If the onboard device config space size is greater than
* the size of struct virtio_net/blk_config, only the spec
* implementing contents size is returned, this is very
@@ -253,7 +240,7 @@ u32 ifcvf_get_config_size(struct ifcvf_hw *hw)
break;
default:
config_size = 0;
- IFCVF_ERR(adapter->pdev, "VIRTIO ID %u not supported\n",
hw->dev_type);
+ IFCVF_ERR(hw->pdev, "VIRTIO ID %u not supported\n",
hw->dev_type);
}
return config_size;
@@ -301,14 +288,11 @@ static void ifcvf_set_features(struct ifcvf_hw *hw, u64
features)
static int ifcvf_config_features(struct ifcvf_hw *hw)
{
- struct ifcvf_adapter *ifcvf;
-
- ifcvf = vf_to_adapter(hw);
ifcvf_set_features(hw, hw->req_features);
ifcvf_add_status(hw, VIRTIO_CONFIG_S_FEATURES_OK);
if (!(ifcvf_get_status(hw) & VIRTIO_CONFIG_S_FEATURES_OK)) {
- IFCVF_ERR(ifcvf->pdev, "Failed to set FEATURES_OK status\n");
+ IFCVF_ERR(hw->pdev, "Failed to set FEATURES_OK status\n");
return -EIO;
}
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h b/drivers/vdpa/ifcvf/ifcvf_base.h
index f5563f665cc6..63ce1f7f6841 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -89,6 +89,7 @@ struct ifcvf_hw {
u16 nr_vring;
/* VIRTIO_PCI_CAP_DEVICE_CFG size */
u32 cap_dev_config_size;
+ struct pci_dev *pdev;
}
[PATCH 0/4] ifcvf/vDPA implement features provisioning
This series implements features provisioning for ifcvf. By applying this series, we allow userspace to create a vDPA device with selected (management device supported) feature bits and mask out others. Please help review Thanks Zhu Lingshan (4): vDPA/ifcvf: ifcvf base layer interfaces work on struct ifcvf_hw vDPA/ifcvf: IRQ interfaces work on ifcvf_hw vDPA/ifcvf: allocate ifcvf_adapter in dev_add() vDPA/ifcvf: implement features provisioning drivers/vdpa/ifcvf/ifcvf_base.c | 32 ++- drivers/vdpa/ifcvf/ifcvf_base.h | 10 +- drivers/vdpa/ifcvf/ifcvf_main.c | 156 +++- 3 files changed, 89 insertions(+), 109 deletions(-) -- 2.31.1 ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V4 3/6] vDPA: allow userspace to query features of a vDPA device
On 7/26/2022 7:06 PM, Parav Pandit via Virtualization wrote:
From: Zhu, Lingshan
Sent: Tuesday, July 26, 2022 7:03 AM
On 7/24/2022 11:21 PM, Parav Pandit wrote:
From: Zhu, Lingshan
Sent: Saturday, July 23, 2022 7:24 AM
On 7/22/2022 9:12 PM, Parav Pandit wrote:
From: Zhu Lingshan
Sent: Friday, July 22, 2022 7:53 AM
This commit adds a new vDPA netlink attribution
VDPA_ATTR_VDPA_DEV_SUPPORTED_FEATURES. Userspace can
query
features
of vDPA devices through this new attr.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/vdpa.c | 13 +
include/uapi/linux/vdpa.h | 1 +
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c index
ebf2f363fbe7..9b0e39b2f022 100644
--- a/drivers/vdpa/vdpa.c
+++ b/drivers/vdpa/vdpa.c
@@ -815,7 +815,7 @@ static int vdpa_dev_net_mq_config_fill(struct
vdpa_device *vdev, static int vdpa_dev_net_config_fill(struct
vdpa_device *vdev, struct sk_buff *msg) {
struct virtio_net_config config = {};
- u64 features;
+ u64 features_device, features_driver;
u16 val_u16;
vdpa_get_config_unlocked(vdev, 0, &config, sizeof(config)); @@
-
832,12 +832,17 @@ static int vdpa_dev_net_config_fill(struct
vdpa_device *vdev, struct sk_buff *ms
if (nla_put_u16(msg, VDPA_ATTR_DEV_NET_CFG_MTU, val_u16))
return -EMSGSIZE;
- features = vdev->config->get_driver_features(vdev);
- if (nla_put_u64_64bit(msg,
VDPA_ATTR_DEV_NEGOTIATED_FEATURES, features,
+ features_driver = vdev->config->get_driver_features(vdev);
+ if (nla_put_u64_64bit(msg,
VDPA_ATTR_DEV_NEGOTIATED_FEATURES, features_driver,
+ VDPA_ATTR_PAD))
+ return -EMSGSIZE;
+
+ features_device = vdev->config->get_device_features(vdev);
+ if (nla_put_u64_64bit(msg,
VDPA_ATTR_VDPA_DEV_SUPPORTED_FEATURES,
+features_device,
VDPA_ATTR_PAD))
return -EMSGSIZE;
- return vdpa_dev_net_mq_config_fill(vdev, msg, features, &config);
+ return vdpa_dev_net_mq_config_fill(vdev, msg, features_driver,
+&config);
}
static int
diff --git a/include/uapi/linux/vdpa.h b/include/uapi/linux/vdpa.h
index
25c55cab3d7c..39f1c3d7c112 100644
--- a/include/uapi/linux/vdpa.h
+++ b/include/uapi/linux/vdpa.h
@@ -47,6 +47,7 @@ enum vdpa_attr {
VDPA_ATTR_DEV_NEGOTIATED_FEATURES, /* u64 */
VDPA_ATTR_DEV_MGMTDEV_MAX_VQS, /* u32 */
VDPA_ATTR_DEV_SUPPORTED_FEATURES, /* u64 */
+ VDPA_ATTR_VDPA_DEV_SUPPORTED_FEATURES, /* u64 */
I have answered in previous emails.
I disagree with the change.
Please reuse VDPA_ATTR_DEV_SUPPORTED_FEATURES.
I believe we have already discussed this before in the V3 thread.
I have told you that reusing this attr will lead to a new race condition.
Returning attribute cannot lead to any race condition.
Please refer to our discussion in the V3 series, I have explained if re-use this
attr, it will be a multiple consumers and multiple produces model, it is a
typical racing condition.
I read the emails with subject = " Re: [PATCH V3 3/6] vDPA: allow userspace to query
features of a vDPA device"
I couldn’t find multiple consumers multiple producers working on same nla
message.
what if two or more iproute2 instance or other userspace tools querying
the features of the management device and the vDPA device
simultaneously? In such a case, there are multiple consumers in the
userspace, and the kernel functions(to fill management device
features and vDPA device features) are the multiple producers. And there
are no locks.
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V4 4/4] vDPA/ifcvf: implement shared IRQ feature
On 2/14/2022 3:19 PM, Jason Wang wrote:
在 2022/2/3 下午3:27, Zhu Lingshan 写道:
On some platforms/devices, there may not be enough MSI vector
slots allocated for virtqueues and config changes. In such a case,
the interrupt sources(virtqueues, config changes) must share
an IRQ/vector, to avoid initialization failures, keep
the device functional.
This commit handles three cases:
(1) number of the allocated vectors == the number of virtqueues + 1
(config changes), every virtqueue and the config interrupt has
a separated vector/IRQ, the best and the most likely case.
(2) number of the allocated vectors is less than the best case, but
greater than 1. In this case, all virtqueues share a vector/IRQ,
the config interrupt has a separated vector/IRQ
(3) only one vector is allocated, in this case, the virtqueues and
the config interrupt share a vector/IRQ. The worst and most
unlikely case.
Otherwise, it needs to fail.
This commit introduces some helper functions:
ifcvf_set_vq_vector() and ifcvf_set_config_vector() sets virtqueue
vector and config vector in the device config space, so that
the device can send interrupt DMA.
This commit adds some fields in struct ifcvf_hw and re-placed
the existed fields to be aligned with the cacheline.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 47 --
drivers/vdpa/ifcvf/ifcvf_base.h | 23 ++-
drivers/vdpa/ifcvf/ifcvf_main.c | 243 +++-
3 files changed, 256 insertions(+), 57 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index 397692ae671c..18dcb63ab1e3 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -15,6 +15,36 @@ struct ifcvf_adapter *vf_to_adapter(struct
ifcvf_hw *hw)
return container_of(hw, struct ifcvf_adapter, vf);
}
+int ifcvf_set_vq_vector(struct ifcvf_hw *hw, u16 qid, int vector)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ struct ifcvf_adapter *ifcvf = vf_to_adapter(hw);
+
+ ifc_iowrite16(qid, &cfg->queue_select);
+ ifc_iowrite16(vector, &cfg->queue_msix_vector);
+ if (ifc_ioread16(&cfg->queue_msix_vector) ==
VIRTIO_MSI_NO_VECTOR) {
+ IFCVF_ERR(ifcvf->pdev, "No msix vector for queue %u\n", qid);
+ return -EINVAL;
+ }
Let's leave this check for the caller, E.g can caller try to assign
NO_VECTOR during uni-nit?
ifcvf driver sets NO_VECTOR when call hw_disable(). I am not sure
whether I get it,
Yes we can let the caller check a vq vector, however this may cause more
than three levels brackets, may looks ugly.
+
+ return 0;
+}
+
+int ifcvf_set_config_vector(struct ifcvf_hw *hw, int vector)
+{
+ struct virtio_pci_common_cfg __iomem *cfg = hw->common_cfg;
+ struct ifcvf_adapter *ifcvf = vf_to_adapter(hw);
+
+ cfg = hw->common_cfg;
+ ifc_iowrite16(vector, &cfg->msix_config);
+ if (ifc_ioread16(&cfg->msix_config) == VIRTIO_MSI_NO_VECTOR) {
+ IFCVF_ERR(ifcvf->pdev, "No msix vector for device config\n");
+ return -EINVAL;
+ }
Similar question as above.
+
+ return 0;
+}
+
static void __iomem *get_cap_addr(struct ifcvf_hw *hw,
struct virtio_pci_cap *cap)
{
@@ -140,6 +170,8 @@ int ifcvf_init_hw(struct ifcvf_hw *hw, struct
pci_dev *pdev)
hw->common_cfg, hw->notify_base, hw->isr,
hw->dev_cfg, hw->notify_off_multiplier);
+ hw->vqs_shared_irq = -EINVAL;
+
return 0;
}
@@ -321,12 +353,6 @@ static int ifcvf_hw_enable(struct ifcvf_hw *hw)
ifcvf = vf_to_adapter(hw);
cfg = hw->common_cfg;
- ifc_iowrite16(IFCVF_MSI_CONFIG_OFF, &cfg->msix_config);
-
- if (ifc_ioread16(&cfg->msix_config) == VIRTIO_MSI_NO_VECTOR) {
- IFCVF_ERR(ifcvf->pdev, "No msix vector for device config\n");
- return -EINVAL;
- }
for (i = 0; i < hw->nr_vring; i++) {
if (!hw->vring[i].ready)
@@ -340,15 +366,6 @@ static int ifcvf_hw_enable(struct ifcvf_hw *hw)
ifc_iowrite64_twopart(hw->vring[i].used, &cfg->queue_used_lo,
&cfg->queue_used_hi);
ifc_iowrite16(hw->vring[i].size, &cfg->queue_size);
- ifc_iowrite16(i + IFCVF_MSI_QUEUE_OFF,
&cfg->queue_msix_vector);
-
- if (ifc_ioread16(&cfg->queue_msix_vector) ==
- VIRTIO_MSI_NO_VECTOR) {
- IFCVF_ERR(ifcvf->pdev,
- "No msix vector for queue %u\n", i);
- return -EINVAL;
- }
-
ifcvf_set_vq_state(hw, i, hw->vring[i].last_avail_idx);
ifc_iowrite16(1, &cfg->queue_enable);
}
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index 949b4fb9d554..9cfe088c82e9 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf
Re: [PATCH V2 2/3] vDPA/ifcvf: enable Intel C5000X-PL virtio-block for vDPA
On 4/15/2021 9:41 PM, Stefano Garzarella wrote:
On Thu, Apr 15, 2021 at 05:53:35PM +0800, Zhu Lingshan wrote:
This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block
for vDPA.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 8 +++-
drivers/vdpa/ifcvf/ifcvf_main.c | 10 +-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index 1c04cd256fa7..0111bfdeb342 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -15,6 +15,7 @@
#include
#include
#include
+#include
#include
#include
@@ -28,7 +29,12 @@
#define C5000X_PL_SUBSYS_VENDOR_ID 0x8086
#define C5000X_PL_SUBSYS_DEVICE_ID 0x0001
-#define IFCVF_SUPPORTED_FEATURES \
+#define C5000X_PL_BLK_VENDOR_ID 0x1AF4
+#define C5000X_PL_BLK_DEVICE_ID 0x1001
+#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086
+#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002
+
+#define IFCVF_NET_SUPPORTED_FEATURES \
((1ULL << VIRTIO_NET_F_MAC) | \
(1ULL << VIRTIO_F_ANY_LAYOUT) | \
(1ULL << VIRTIO_F_VERSION_1) | \
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 469a9b5737b7..cea1313b1a3f 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -171,7 +171,11 @@ static u64 ifcvf_vdpa_get_features(struct
vdpa_device *vdpa_dev)
struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
u64 features;
- features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES;
+ if (vf->dev_type == VIRTIO_ID_NET)
+ features = ifcvf_get_features(vf) &
IFCVF_NET_SUPPORTED_FEATURES;
+
+ if (vf->dev_type == VIRTIO_ID_BLOCK)
+ features = ifcvf_get_features(vf);
Should we put a warning here too otherwise feature could be seen
unassigned?
Thanks, it will be a switch code block too.
Thanks,
Stefano
return features;
}
@@ -517,6 +521,10 @@ static struct pci_device_id ifcvf_pci_ids[] = {
C5000X_PL_DEVICE_ID,
C5000X_PL_SUBSYS_VENDOR_ID,
C5000X_PL_SUBSYS_DEVICE_ID) },
+ { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID,
+ C5000X_PL_BLK_DEVICE_ID,
+ C5000X_PL_BLK_SUBSYS_VENDOR_ID,
+ C5000X_PL_BLK_SUBSYS_DEVICE_ID) },
{ 0 },
};
--
2.27.0
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V2 3/3] vDPA/ifcvf: get_config_size should return dev specific config size
On 4/15/2021 9:48 PM, Stefano Garzarella wrote:
On Thu, Apr 15, 2021 at 05:53:36PM +0800, Zhu Lingshan wrote:
get_config_size() should return the size based on the decected
device type.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_main.c | 18 +-
1 file changed, 17 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index cea1313b1a3f..6844c49fe1de 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -347,7 +347,23 @@ static u32 ifcvf_vdpa_get_vq_align(struct
vdpa_device *vdpa_dev)
static size_t ifcvf_vdpa_get_config_size(struct vdpa_device *vdpa_dev)
{
- return sizeof(struct virtio_net_config);
+ struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+ struct pci_dev *pdev = adapter->pdev;
+ size_t size;
+
+ if (vf->dev_type == VIRTIO_ID_NET)
+ size = sizeof(struct virtio_net_config);
+
+ else if (vf->dev_type == VIRTIO_ID_BLOCK)
+ size = sizeof(struct virtio_blk_config);
+
+ else {
+ size = 0;
+ IFCVF_ERR(pdev, "VIRTIO ID %u not supported\n", vf->dev_type);
+ }
I slightly prefer the switch, but I don't have a strong opinion.
However, if we want to use if/else, we should follow
`Documentation/process/coding-style.rst` line 166:
Note that the closing brace is empty on a line of its own,
**except** in
the cases where it is followed by a continuation of the same
statement,
ie a ``while`` in a do-statement or an ``else`` in an if-statement,
like
also `scripts/checkpatch.pl --strict` complains:
CHECK: braces {} should be used on all arms of this statement
#209: FILE: drivers/vdpa/ifcvf/ifcvf_main.c:355:
+ if (vf->dev_type == VIRTIO_ID_NET)
[...]
+ else if (vf->dev_type == VIRTIO_ID_BLOCK)
[...]
+ else {
[...]
CHECK: Unbalanced braces around else statement
#215: FILE: drivers/vdpa/ifcvf/ifcvf_main.c:361:
+ else {
Thanks Stefano, the reason is we only have one line code after if, so
looks like {} is unnecessary, I agree switch can clear up
code style confusions. I will add this in v3.
Thanks!
Thanks,
Stefano
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 3/3] vDPA/ifcvf: get_config_size should return dev specific config size
On 4/15/2021 4:12 PM, Stefano Garzarella wrote:
On Wed, Apr 14, 2021 at 05:18:32PM +0800, Zhu Lingshan wrote:
get_config_size() should return the size based on the decected
device type.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_main.c | 11 ++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 9b6a38b798fa..b48b9789b69e 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -347,7 +347,16 @@ static u32 ifcvf_vdpa_get_vq_align(struct
vdpa_device *vdpa_dev)
static size_t ifcvf_vdpa_get_config_size(struct vdpa_device *vdpa_dev)
{
- return sizeof(struct virtio_net_config);
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+ size_t size;
+
+ if (vf->dev_type == VIRTIO_ID_NET)
+ size = sizeof(struct virtio_net_config);
+
+ if (vf->dev_type == VIRTIO_ID_BLOCK)
+ size = sizeof(struct virtio_blk_config);
+
+ return size;
I'm not familiar with the ifcvf details, but can it happen that the
device is not block or net?
Should we set `size` to 0 by default to handle this case or are we
sure it's one of the two?
Maybe we should add a comment or a warning message in this case, to
prevent some analysis tool or compiler from worrying that `size` might
be uninitialized.
I was thinking something like this:
switch(vf->dev_type) {
case VIRTIO_ID_NET:
size = sizeof(struct virtio_net_config);
break;
case VIRTIO_ID_BLOCK:
size = sizeof(struct virtio_blk_config);
break;
default:
/* or WARN(1, "") if dev_warn() not apply */
dev_warn(... , "virtio ID [0x%x] not supported\n")
size = 0;
}
Thanks,
Stefano
agree, will add this in V2
Thanks
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/3] vDPA/ifcvf: enable Intel C5000X-PL virtio-block for vDPA
On 4/15/2021 3:17 PM, Jason Wang wrote: 在 2021/4/15 下午2:41, Zhu Lingshan 写道: I think we've discussed this sometime in the past but what's the reason for such whitelist consider there's already a get_features() implemention? E.g Any reason to block VIRTIO_BLK_F_WRITE_ZEROS or VIRTIO_F_RING_PACKED? Thanks The reason is some feature bits are supported in the device but not supported by the driver, e.g, for virtio-net, mq & cq implementation is not ready in the driver. I understand the case of virtio-net but I wonder why we need this for block where we don't vq cvq. Thanks This is still a subset of the feature bits read from hardware, I leave it here to code consistently, and indicate what we support clearly. Are you suggesting remove this feature bits list and just use what we read from hardware? Thansk Yes, please do that. The whiltelist doesn't help in this case I think. OK, will remove this in V2 Thanks Thanks ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/3] vDPA/ifcvf: deduce VIRTIO device ID when probe
On 4/15/2021 3:16 PM, Jason Wang wrote:
在 2021/4/15 下午2:36, Zhu Lingshan 写道:
On 4/15/2021 2:30 PM, Jason Wang wrote:
在 2021/4/15 下午1:52, Zhu Lingshan 写道:
On 4/15/2021 11:30 AM, Jason Wang wrote:
在 2021/4/14 下午5:18, Zhu Lingshan 写道:
This commit deduces VIRTIO device ID as device type when probe,
then ifcvf_vdpa_get_device_id() can simply return the ID.
ifcvf_vdpa_get_features() and ifcvf_vdpa_get_config_size()
can work properly based on the device ID.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 22 ++
2 files changed, 11 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index b2eeb16b9c2c..1c04cd256fa7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -84,6 +84,7 @@ struct ifcvf_hw {
u32 notify_off_multiplier;
u64 req_features;
u64 hw_features;
+ u32 dev_type;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *net_cfg;
struct vring_info vring[IFCVF_MAX_QUEUE_PAIRS * 2];
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 44d7586019da..99b0a6b4c227 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -323,19 +323,9 @@ static u32 ifcvf_vdpa_get_generation(struct
vdpa_device *vdpa_dev)
static u32 ifcvf_vdpa_get_device_id(struct vdpa_device
*vdpa_dev)
{
- struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
- struct pci_dev *pdev = adapter->pdev;
- u32 ret = -ENODEV;
-
- if (pdev->device < 0x1000 || pdev->device > 0x107f)
- return ret;
-
- if (pdev->device < 0x1040)
- ret = pdev->subsystem_device;
- else
- ret = pdev->device-0x1040;
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
- return ret;
+ return vf->dev_type;
}
static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device
*vdpa_dev)
@@ -466,6 +456,14 @@ static int ifcvf_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
pci_set_drvdata(pdev, adapter);
vf = &adapter->vf;
+ if (pdev->device < 0x1000 || pdev->device > 0x107f)
+ return -EOPNOTSUPP;
+
+ if (pdev->device < 0x1040)
+ vf->dev_type = pdev->subsystem_device;
+ else
+ vf->dev_type = pdev->device - 0x1040;
So a question here, is the device a transtional device or modern one?
If it's a transitonal one, can it swtich endianess automatically
or not?
Thanks
Hi Jason,
This driver should drive both modern and transitional devices as we
discussed before.
If it's a transitional one, it will act as a modern device by
default, legacy mode is a fail-over path.
Note that legacy driver use native endian, support legacy driver
requires the device to know native endian which I'm not sure your
device can do that.
Thanks
Yes, legacy requires guest native endianess, I think we don't need to
worry about this because our transitional device should work in
modern mode by
default(legacy mode is the failover path we will never reach,
get_features will fail if no ACCESS_PLATFORM), we don't support
legacy device in vDPA.
Thanks
Ok, so I think it's better to add a comment here.
sure, will add a comment in V2
Thanks
Thanks
For vDPA, it has to support VIRTIO_1 and ACCESS_PLATFORM, so it
must in modern mode.
I think we don't need to worry about endianess for legacy mode.
Thanks
Zhu Lingshan
+
vf->base = pcim_iomap_table(pdev);
adapter->pdev = pdev;
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/3] vDPA/ifcvf: enable Intel C5000X-PL virtio-block for vDPA
On 4/15/2021 2:31 PM, Jason Wang wrote:
在 2021/4/15 下午1:55, Zhu Lingshan 写道:
On 4/15/2021 11:34 AM, Jason Wang wrote:
在 2021/4/14 下午5:18, Zhu Lingshan 写道:
This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block
for vDPA.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 17 -
drivers/vdpa/ifcvf/ifcvf_main.c | 10 +-
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index 1c04cd256fa7..8b403522bf06 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -15,6 +15,7 @@
#include
#include
#include
+#include
#include
#include
@@ -28,7 +29,12 @@
#define C5000X_PL_SUBSYS_VENDOR_ID 0x8086
#define C5000X_PL_SUBSYS_DEVICE_ID 0x0001
-#define IFCVF_SUPPORTED_FEATURES \
+#define C5000X_PL_BLK_VENDOR_ID 0x1AF4
+#define C5000X_PL_BLK_DEVICE_ID 0x1001
+#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086
+#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002
+
+#define IFCVF_NET_SUPPORTED_FEATURES \
((1ULL << VIRTIO_NET_F_MAC) | \
(1ULL << VIRTIO_F_ANY_LAYOUT) | \
(1ULL << VIRTIO_F_VERSION_1) | \
@@ -37,6 +43,15 @@
(1ULL << VIRTIO_F_ACCESS_PLATFORM) | \
(1ULL << VIRTIO_NET_F_MRG_RXBUF))
+#define IFCVF_BLK_SUPPORTED_FEATURES \
+ ((1ULL << VIRTIO_BLK_F_SIZE_MAX) | \
+ (1ULL << VIRTIO_BLK_F_SEG_MAX) | \
+ (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \
+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | \
+ (1ULL << VIRTIO_BLK_F_MQ) | \
+ (1ULL << VIRTIO_F_VERSION_1) | \
+ (1ULL << VIRTIO_F_ACCESS_PLATFORM))
I think we've discussed this sometime in the past but what's the
reason for such whitelist consider there's already a get_features()
implemention?
E.g Any reason to block VIRTIO_BLK_F_WRITE_ZEROS or
VIRTIO_F_RING_PACKED?
Thanks
The reason is some feature bits are supported in the device but not
supported by the driver, e.g, for virtio-net, mq & cq implementation
is not ready in the driver.
I understand the case of virtio-net but I wonder why we need this for
block where we don't vq cvq.
Thanks
This is still a subset of the feature bits read from hardware, I leave
it here to code consistently, and indicate what we support clearly.
Are you suggesting remove this feature bits list and just use what we
read from hardware?
Thansk
Thanks!
+
/* Only one queue pair for now. */
#define IFCVF_MAX_QUEUE_PAIRS 1
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 99b0a6b4c227..9b6a38b798fa 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -171,7 +171,11 @@ static u64 ifcvf_vdpa_get_features(struct
vdpa_device *vdpa_dev)
struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
u64 features;
- features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES;
+ if (vf->dev_type == VIRTIO_ID_NET)
+ features = ifcvf_get_features(vf) &
IFCVF_NET_SUPPORTED_FEATURES;
+
+ if (vf->dev_type == VIRTIO_ID_BLOCK)
+ features = ifcvf_get_features(vf) &
IFCVF_BLK_SUPPORTED_FEATURES;
return features;
}
@@ -509,6 +513,10 @@ static struct pci_device_id ifcvf_pci_ids[] = {
C5000X_PL_DEVICE_ID,
C5000X_PL_SUBSYS_VENDOR_ID,
C5000X_PL_SUBSYS_DEVICE_ID) },
+ { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID,
+ C5000X_PL_BLK_DEVICE_ID,
+ C5000X_PL_BLK_SUBSYS_VENDOR_ID,
+ C5000X_PL_BLK_SUBSYS_DEVICE_ID) },
{ 0 },
};
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/3] vDPA/ifcvf: deduce VIRTIO device ID when probe
On 4/15/2021 2:30 PM, Jason Wang wrote:
在 2021/4/15 下午1:52, Zhu Lingshan 写道:
On 4/15/2021 11:30 AM, Jason Wang wrote:
在 2021/4/14 下午5:18, Zhu Lingshan 写道:
This commit deduces VIRTIO device ID as device type when probe,
then ifcvf_vdpa_get_device_id() can simply return the ID.
ifcvf_vdpa_get_features() and ifcvf_vdpa_get_config_size()
can work properly based on the device ID.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 22 ++
2 files changed, 11 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index b2eeb16b9c2c..1c04cd256fa7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -84,6 +84,7 @@ struct ifcvf_hw {
u32 notify_off_multiplier;
u64 req_features;
u64 hw_features;
+ u32 dev_type;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *net_cfg;
struct vring_info vring[IFCVF_MAX_QUEUE_PAIRS * 2];
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 44d7586019da..99b0a6b4c227 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -323,19 +323,9 @@ static u32 ifcvf_vdpa_get_generation(struct
vdpa_device *vdpa_dev)
static u32 ifcvf_vdpa_get_device_id(struct vdpa_device *vdpa_dev)
{
- struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
- struct pci_dev *pdev = adapter->pdev;
- u32 ret = -ENODEV;
-
- if (pdev->device < 0x1000 || pdev->device > 0x107f)
- return ret;
-
- if (pdev->device < 0x1040)
- ret = pdev->subsystem_device;
- else
- ret = pdev->device -0x1040;
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
- return ret;
+ return vf->dev_type;
}
static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device *vdpa_dev)
@@ -466,6 +456,14 @@ static int ifcvf_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
pci_set_drvdata(pdev, adapter);
vf = &adapter->vf;
+ if (pdev->device < 0x1000 || pdev->device > 0x107f)
+ return -EOPNOTSUPP;
+
+ if (pdev->device < 0x1040)
+ vf->dev_type = pdev->subsystem_device;
+ else
+ vf->dev_type = pdev->device - 0x1040;
So a question here, is the device a transtional device or modern one?
If it's a transitonal one, can it swtich endianess automatically or
not?
Thanks
Hi Jason,
This driver should drive both modern and transitional devices as we
discussed before.
If it's a transitional one, it will act as a modern device by
default, legacy mode is a fail-over path.
Note that legacy driver use native endian, support legacy driver
requires the device to know native endian which I'm not sure your
device can do that.
Thanks
Yes, legacy requires guest native endianess, I think we don't need to
worry about this because our transitional device should work in modern
mode by
default(legacy mode is the failover path we will never reach,
get_features will fail if no ACCESS_PLATFORM), we don't support legacy
device in vDPA.
Thanks
For vDPA, it has to support VIRTIO_1 and ACCESS_PLATFORM, so it must
in modern mode.
I think we don't need to worry about endianess for legacy mode.
Thanks
Zhu Lingshan
+
vf->base = pcim_iomap_table(pdev);
adapter->pdev = pdev;
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/3] vDPA/ifcvf: enable Intel C5000X-PL virtio-block for vDPA
On 4/15/2021 11:34 AM, Jason Wang wrote:
在 2021/4/14 下午5:18, Zhu Lingshan 写道:
This commit enabled Intel FPGA SmartNIC C5000X-PL virtio-block
for vDPA.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 17 -
drivers/vdpa/ifcvf/ifcvf_main.c | 10 +-
2 files changed, 25 insertions(+), 2 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index 1c04cd256fa7..8b403522bf06 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -15,6 +15,7 @@
#include
#include
#include
+#include
#include
#include
@@ -28,7 +29,12 @@
#define C5000X_PL_SUBSYS_VENDOR_ID 0x8086
#define C5000X_PL_SUBSYS_DEVICE_ID 0x0001
-#define IFCVF_SUPPORTED_FEATURES \
+#define C5000X_PL_BLK_VENDOR_ID 0x1AF4
+#define C5000X_PL_BLK_DEVICE_ID 0x1001
+#define C5000X_PL_BLK_SUBSYS_VENDOR_ID 0x8086
+#define C5000X_PL_BLK_SUBSYS_DEVICE_ID 0x0002
+
+#define IFCVF_NET_SUPPORTED_FEATURES \
((1ULL << VIRTIO_NET_F_MAC) | \
(1ULL << VIRTIO_F_ANY_LAYOUT) | \
(1ULL << VIRTIO_F_VERSION_1) | \
@@ -37,6 +43,15 @@
(1ULL << VIRTIO_F_ACCESS_PLATFORM) | \
(1ULL << VIRTIO_NET_F_MRG_RXBUF))
+#define IFCVF_BLK_SUPPORTED_FEATURES \
+ ((1ULL << VIRTIO_BLK_F_SIZE_MAX) | \
+ (1ULL << VIRTIO_BLK_F_SEG_MAX) | \
+ (1ULL << VIRTIO_BLK_F_BLK_SIZE) | \
+ (1ULL << VIRTIO_BLK_F_TOPOLOGY) | \
+ (1ULL << VIRTIO_BLK_F_MQ) | \
+ (1ULL << VIRTIO_F_VERSION_1) | \
+ (1ULL << VIRTIO_F_ACCESS_PLATFORM))
I think we've discussed this sometime in the past but what's the
reason for such whitelist consider there's already a get_features()
implemention?
E.g Any reason to block VIRTIO_BLK_F_WRITE_ZEROS or VIRTIO_F_RING_PACKED?
Thanks
The reason is some feature bits are supported in the device but not
supported by the driver, e.g, for virtio-net, mq & cq implementation is
not ready in the driver.
Thanks!
+
/* Only one queue pair for now. */
#define IFCVF_MAX_QUEUE_PAIRS 1
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 99b0a6b4c227..9b6a38b798fa 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -171,7 +171,11 @@ static u64 ifcvf_vdpa_get_features(struct
vdpa_device *vdpa_dev)
struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
u64 features;
- features = ifcvf_get_features(vf) & IFCVF_SUPPORTED_FEATURES;
+ if (vf->dev_type == VIRTIO_ID_NET)
+ features = ifcvf_get_features(vf) &
IFCVF_NET_SUPPORTED_FEATURES;
+
+ if (vf->dev_type == VIRTIO_ID_BLOCK)
+ features = ifcvf_get_features(vf) &
IFCVF_BLK_SUPPORTED_FEATURES;
return features;
}
@@ -509,6 +513,10 @@ static struct pci_device_id ifcvf_pci_ids[] = {
C5000X_PL_DEVICE_ID,
C5000X_PL_SUBSYS_VENDOR_ID,
C5000X_PL_SUBSYS_DEVICE_ID) },
+ { PCI_DEVICE_SUB(C5000X_PL_BLK_VENDOR_ID,
+ C5000X_PL_BLK_DEVICE_ID,
+ C5000X_PL_BLK_SUBSYS_VENDOR_ID,
+ C5000X_PL_BLK_SUBSYS_DEVICE_ID) },
{ 0 },
};
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/3] vDPA/ifcvf: deduce VIRTIO device ID when probe
On 4/15/2021 11:30 AM, Jason Wang wrote:
在 2021/4/14 下午5:18, Zhu Lingshan 写道:
This commit deduces VIRTIO device ID as device type when probe,
then ifcvf_vdpa_get_device_id() can simply return the ID.
ifcvf_vdpa_get_features() and ifcvf_vdpa_get_config_size()
can work properly based on the device ID.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 22 ++
2 files changed, 11 insertions(+), 12 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index b2eeb16b9c2c..1c04cd256fa7 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -84,6 +84,7 @@ struct ifcvf_hw {
u32 notify_off_multiplier;
u64 req_features;
u64 hw_features;
+ u32 dev_type;
struct virtio_pci_common_cfg __iomem *common_cfg;
void __iomem *net_cfg;
struct vring_info vring[IFCVF_MAX_QUEUE_PAIRS * 2];
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 44d7586019da..99b0a6b4c227 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -323,19 +323,9 @@ static u32 ifcvf_vdpa_get_generation(struct
vdpa_device *vdpa_dev)
static u32 ifcvf_vdpa_get_device_id(struct vdpa_device *vdpa_dev)
{
- struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
- struct pci_dev *pdev = adapter->pdev;
- u32 ret = -ENODEV;
-
- if (pdev->device < 0x1000 || pdev->device > 0x107f)
- return ret;
-
- if (pdev->device < 0x1040)
- ret = pdev->subsystem_device;
- else
- ret = pdev->device - 0x1040;
+ struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
- return ret;
+ return vf->dev_type;
}
static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device *vdpa_dev)
@@ -466,6 +456,14 @@ static int ifcvf_probe(struct pci_dev *pdev,
const struct pci_device_id *id)
pci_set_drvdata(pdev, adapter);
vf = &adapter->vf;
+ if (pdev->device < 0x1000 || pdev->device > 0x107f)
+ return -EOPNOTSUPP;
+
+ if (pdev->device < 0x1040)
+ vf->dev_type = pdev->subsystem_device;
+ else
+ vf->dev_type = pdev->device - 0x1040;
So a question here, is the device a transtional device or modern one?
If it's a transitonal one, can it swtich endianess automatically or not?
Thanks
Hi Jason,
This driver should drive both modern and transitional devices as we
discussed before.
If it's a transitional one, it will act as a modern device by default,
legacy mode is a fail-over path.
For vDPA, it has to support VIRTIO_1 and ACCESS_PLATFORM, so it must in
modern mode.
I think we don't need to worry about endianess for legacy mode.
Thanks
Zhu Lingshan
+
vf->base = pcim_iomap_table(pdev);
adapter->pdev = pdev;
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 3/6] vDPA/ifcvf: rename original IFCVF dev ids to N3000 ids
On 3/11/2021 11:25 AM, Jason Wang wrote:
On 2021/3/10 5:00 下午, Zhu Lingshan wrote:
IFCVF driver probes multiple types of devices now,
to distinguish the original device driven by IFCVF
from others, it is renamed as "N3000".
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.h | 8
drivers/vdpa/ifcvf/ifcvf_main.c | 8
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index 75d9a8052039..794d1505d857 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -18,10 +18,10 @@
#include
#include
-#define IFCVF_VENDOR_ID 0x1AF4
-#define IFCVF_DEVICE_ID 0x1041
-#define IFCVF_SUBSYS_VENDOR_ID 0x8086
-#define IFCVF_SUBSYS_DEVICE_ID 0x001A
+#define N3000_VENDOR_ID 0x1AF4
+#define N3000_DEVICE_ID 0x1041
+#define N3000_SUBSYS_VENDOR_ID 0x8086
+#define N3000_SUBSYS_DEVICE_ID 0x001A
#define C5000X_PL_VENDOR_ID 0x1AF4
#define C5000X_PL_DEVICE_ID 0x1000
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 26a2dab7ca66..fd5befc5cbcc 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -480,10 +480,10 @@ static void ifcvf_remove(struct pci_dev *pdev)
}
static struct pci_device_id ifcvf_pci_ids[] = {
- { PCI_DEVICE_SUB(IFCVF_VENDOR_ID,
- IFCVF_DEVICE_ID,
- IFCVF_SUBSYS_VENDOR_ID,
- IFCVF_SUBSYS_DEVICE_ID) },
+ { PCI_DEVICE_SUB(N3000_VENDOR_ID,
+ N3000_DEVICE_ID,
I am not sure the plan for Intel but I wonder if we can simply use
PCI_ANY_ID for device id here. Otherewise you need to maintain a very
long list of ids here.
Thanks
Hi Jason,
Thanks! but maybe if we present a very simple and clear list like what
e1000 does can help the users understand what we support easily.
Thanks!
+ N3000_SUBSYS_VENDOR_ID,
+ N3000_SUBSYS_DEVICE_ID) },
{ PCI_DEVICE_SUB(C5000X_PL_VENDOR_ID,
C5000X_PL_DEVICE_ID,
C5000X_PL_SUBSYS_VENDOR_ID,
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 1/6] vDPA/ifcvf: get_vendor_id returns a device specific vendor id
On 3/11/2021 11:23 AM, Jason Wang wrote:
On 2021/3/10 5:00 下午, Zhu Lingshan wrote:
In this commit, ifcvf_get_vendor_id() will return
a device specific vendor id of the probed pci device
than a hard code.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_main.c | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index fa1af301cf55..e501ee07de17 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -324,7 +324,10 @@ static u32 ifcvf_vdpa_get_device_id(struct
vdpa_device *vdpa_dev)
static u32 ifcvf_vdpa_get_vendor_id(struct vdpa_device *vdpa_dev)
{
- return IFCVF_SUBSYS_VENDOR_ID;
+ struct ifcvf_adapter *adapter = vdpa_to_adapter(vdpa_dev);
+ struct pci_dev *pdev = adapter->pdev;
+
+ return pdev->subsystem_vendor;
}
While at this, I wonder if we can do something similar in
get_device_id() if it could be simple deduced from some simple math
from the pci device id?
Thanks
Hi Jason,
IMHO, this implementation is just some memory read ops, I think other
implementations may not save many cpu cycles, an if cost at least three
cpu cycles.
Thanks!
static u32 ifcvf_vdpa_get_vq_align(struct vdpa_device *vdpa_dev)
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 6/6] vDPA/ifcvf: verify mandatory feature bits for vDPA
On 3/11/2021 11:20 AM, Jason Wang wrote:
On 2021/3/10 5:00 下午, Zhu Lingshan wrote:
vDPA requres VIRTIO_F_ACCESS_PLATFORM as a must, this commit
examines this when set features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 8
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 5 +
3 files changed, 14 insertions(+)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index ea6a78791c9b..58f47fdce385 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -224,6 +224,14 @@ u64 ifcvf_get_features(struct ifcvf_hw *hw)
return hw->hw_features;
}
+int ifcvf_verify_min_features(struct ifcvf_hw *hw)
+{
+ if (!(hw->hw_features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)))
+ return -EINVAL;
+
+ return 0;
+}
+
void ifcvf_read_net_config(struct ifcvf_hw *hw, u64 offset,
void *dst, int length)
{
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index dbb8c10aa3b1..91c5735d4dc9 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -123,6 +123,7 @@ void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
u64 ifcvf_get_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
+int ifcvf_verify_min_features(struct ifcvf_hw *hw);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num);
struct ifcvf_adapter *vf_to_adapter(struct ifcvf_hw *hw);
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 25fb9dfe23f0..f624f202447d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -179,6 +179,11 @@ static u64 ifcvf_vdpa_get_features(struct
vdpa_device *vdpa_dev)
static int ifcvf_vdpa_set_features(struct vdpa_device *vdpa_dev,
u64 features)
{
struct ifcvf_hw *vf = vdpa_to_vf(vdpa_dev);
+ int ret;
+
+ ret = ifcvf_verify_min_features(vf);
So this validate device features instead of driver which is the one we
really want to check?
Thanks
Hi Jason,
Here we check device feature bits to make sure the device support
ACCESS_PLATFORM. In get_features(),
it will return a intersection of device features bit and driver
supported features bits(which includes ACCESS_PLATFORM).
Other components like QEMU should not set features bits more than this
intersection of bits. so we can make sure if this
ifcvf_verify_min_features() passed, both device and driver support
ACCESS_PLATFORM.
Are you suggesting check driver feature bits in
ifcvf_verify_min_features() in the meantime as well?
Thanks！
+ if (ret)
+ return ret;
vf->req_features = features;
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 6/6] vDPA/ifcvf: verify mandatory feature bits for vDPA
On 3/11/2021 11:20 AM, Jason Wang wrote:
On 2021/3/10 5:00 下午, Zhu Lingshan wrote:
vDPA requres VIRTIO_F_ACCESS_PLATFORM as a must, this commit
examines this when set features.
Signed-off-by: Zhu Lingshan
---
drivers/vdpa/ifcvf/ifcvf_base.c | 8
drivers/vdpa/ifcvf/ifcvf_base.h | 1 +
drivers/vdpa/ifcvf/ifcvf_main.c | 5 +
3 files changed, 14 insertions(+)
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.c
b/drivers/vdpa/ifcvf/ifcvf_base.c
index ea6a78791c9b..58f47fdce385 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.c
+++ b/drivers/vdpa/ifcvf/ifcvf_base.c
@@ -224,6 +224,14 @@ u64 ifcvf_get_features(struct ifcvf_hw *hw)
return hw->hw_features;
}
+int ifcvf_verify_min_features(struct ifcvf_hw *hw)
+{
+ if (!(hw->hw_features & BIT_ULL(VIRTIO_F_ACCESS_PLATFORM)))
+ return -EINVAL;
+
+ return 0;
+}
+
void ifcvf_read_net_config(struct ifcvf_hw *hw, u64 offset,
void *dst, int length)
{
diff --git a/drivers/vdpa/ifcvf/ifcvf_base.h
b/drivers/vdpa/ifcvf/ifcvf_base.h
index dbb8c10aa3b1..91c5735d4dc9 100644
--- a/drivers/vdpa/ifcvf/ifcvf_base.h
+++ b/drivers/vdpa/ifcvf/ifcvf_base.h
@@ -123,6 +123,7 @@ void io_write64_twopart(u64 val, u32 *lo, u32 *hi);
void ifcvf_reset(struct ifcvf_hw *hw);
u64 ifcvf_get_features(struct ifcvf_hw *hw);
u64 ifcvf_get_hw_features(struct ifcvf_hw *hw);
+int ifcvf_verify_min_features(struct ifcvf_hw *hw);
u16 ifcvf_get_vq_state(struct ifcvf_hw *hw, u16 qid);
int ifcvf_set_vq_state(struct ifcvf_hw *hw, u16 qid, u16 num);
struct ifcvf_adapter *vf_to_adapter(struct ifcvf_hw *hw);
diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c
b/drivers/vdpa/ifcvf/ifcvf_main.c
index 25fb9dfe23f0..f624f202447d 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -179,6 +179,11 @@ static u64 ifcvf_vdpa_get_features(struct
vdpa_device *vdpa_dev)
static int ifcvf_vdpa_set_features(struct vdpa_device *vdpa_dev,
u64 features)
{
struct ifcvf_hw *vf = vdpa_to_intersectionvf(vdpa_dev);
+ int ret;
+
+ ret = ifcvf_verify_min_features(vf);
So this validate device features instead of driver which is the one we
really want to check?
Thanks
Hi Jason,
Here we check device feature bits to make sure the device support
ACCESS_PLATFORM. In get_features(),
it will return a intersection of device features bit and driver
supported features bits(which includes ACCESS_PLATFORM).
Other components like QEMU should not set features bits more than this
intersection of bits. so we can make sure if this
ifcvf_verify_min_features() passed, both device and driver support
ACCESS_PLATFORM.
Are you suggesting check driver feature bits in
ifcvf_verify_min_features() in the meantime as well?
Thanks！
+ if (ret)
+ return ret;
vf->req_features = features;
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
___
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 4/6] vDPA/ifcvf: remove the version number string
Thanks Leon, I will include your ROB if there is a V4. On 3/10/2021 5:16 PM, Leon Romanovsky wrote: On Wed, Mar 10, 2021 at 05:00:50PM +0800, Zhu Lingshan wrote: This commit removes the version number string, using kernel version is enough. Signed-off-by: Zhu Lingshan --- drivers/vdpa/ifcvf/ifcvf_main.c | 2 -- 1 file changed, 2 deletions(-) I already added my ROB, but will add again. Thanks, Reviewed-by: Leon Romanovsky ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH V3 1/6] vhost: introduce vhost_vring_call
Please ignore this patchset incorrect metadata, will resend soon. Thanks!
On 7/22/2020 5:49 PM, Zhu Lingshan wrote:
From: Zhu Lingshan
This commit introduces struct vhost_vring_call which replaced
raw struct eventfd_ctx *call_ctx in struct vhost_virtqueue.
Besides eventfd_ctx, it contains a spin lock and an
irq_bypass_producer in its structure.
Signed-off-by: Zhu Lingshan
Signed-off-by: lszhu
Signed-off-by: Zhu Lingshan
---
drivers/vhost/vdpa.c | 4 ++--
drivers/vhost/vhost.c | 22 --
drivers/vhost/vhost.h | 9 -
3 files changed, 26 insertions(+), 9 deletions(-)
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index a54b60d6623f..df3cf386b0cd 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -96,7 +96,7 @@ static void handle_vq_kick(struct vhost_work *work)
static irqreturn_t vhost_vdpa_virtqueue_cb(void *private)
{
struct vhost_virtqueue *vq = private;
- struct eventfd_ctx *call_ctx = vq->call_ctx;
+ struct eventfd_ctx *call_ctx = vq->call_ctx.ctx;
if (call_ctx)
eventfd_signal(call_ctx, 1);
@@ -382,7 +382,7 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v,
unsigned int cmd,
break;
case VHOST_SET_VRING_CALL:
- if (vq->call_ctx) {
+ if (vq->call_ctx.ctx) {
cb.callback = vhost_vdpa_virtqueue_cb;
cb.private = vq;
} else {
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index d7b8df3edffc..9f1a845a9302 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -298,6 +298,13 @@ static void vhost_vq_meta_reset(struct vhost_dev *d)
__vhost_vq_meta_reset(d->vqs[i]);
}
+static void vhost_vring_call_reset(struct vhost_vring_call *call_ctx)
+{
+ call_ctx->ctx = NULL;
+ memset(&call_ctx->producer, 0x0, sizeof(struct irq_bypass_producer));
+ spin_lock_init(&call_ctx->ctx_lock);
+}
+
static void vhost_vq_reset(struct vhost_dev *dev,
struct vhost_virtqueue *vq)
{
@@ -319,13 +326,13 @@ static void vhost_vq_reset(struct vhost_dev *dev,
vq->log_base = NULL;
vq->error_ctx = NULL;
vq->kick = NULL;
- vq->call_ctx = NULL;
vq->log_ctx = NULL;
vhost_reset_is_le(vq);
vhost_disable_cross_endian(vq);
vq->busyloop_timeout = 0;
vq->umem = NULL;
vq->iotlb = NULL;
+ vhost_vring_call_reset(&vq->call_ctx);
__vhost_vq_meta_reset(vq);
}
@@ -685,8 +692,8 @@ void vhost_dev_cleanup(struct vhost_dev *dev)
eventfd_ctx_put(dev->vqs[i]->error_ctx);
if (dev->vqs[i]->kick)
fput(dev->vqs[i]->kick);
- if (dev->vqs[i]->call_ctx)
- eventfd_ctx_put(dev->vqs[i]->call_ctx);
+ if (dev->vqs[i]->call_ctx.ctx)
+ eventfd_ctx_put(dev->vqs[i]->call_ctx.ctx);
vhost_vq_reset(dev, dev->vqs[i]);
}
vhost_dev_free_iovecs(dev);
@@ -1629,7 +1636,10 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int
ioctl, void __user *arg
r = PTR_ERR(ctx);
break;
}
- swap(ctx, vq->call_ctx);
+
+ spin_lock(&vq->call_ctx.ctx_lock);
+ swap(ctx, vq->call_ctx.ctx);
+ spin_unlock(&vq->call_ctx.ctx_lock);
break;
case VHOST_SET_VRING_ERR:
if (copy_from_user(&f, argp, sizeof f)) {
@@ -2440,8 +2450,8 @@ static bool vhost_notify(struct vhost_dev *dev, struct
vhost_virtqueue *vq)
void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq)
{
/* Signal the Guest tell them we used something up. */
- if (vq->call_ctx && vhost_notify(dev, vq))
- eventfd_signal(vq->call_ctx, 1);
+ if (vq->call_ctx.ctx && vhost_notify(dev, vq))
+ eventfd_signal(vq->call_ctx.ctx, 1);
}
EXPORT_SYMBOL_GPL(vhost_signal);
diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h
index c8e96a095d3b..38eb1aa3b68d 100644
--- a/drivers/vhost/vhost.h
+++ b/drivers/vhost/vhost.h
@@ -13,6 +13,7 @@
#include
#include
#include
+#include
struct vhost_work;
typedef void (*vhost_work_fn_t)(struct vhost_work *work);
@@ -60,6 +61,12 @@ enum vhost_uaddr_type {
VHOST_NUM_ADDRS = 3,
};
+struct vhost_vring_call {
+ struct eventfd_ctx *ctx;
+ struct irq_bypass_producer producer;
+ spinlock_t ctx_lock;
+};
+
/* The virtqueue structure describes a queue attached to a device. */
struct vhost_virtqueue {
struct vhost_dev *dev;
@@ -72,7 +79,7 @@ struct vhost_virtqueue {
vring_used_t __user *used;
