Re: virtio-iommu hotplug issue
On Thu, Apr 13, 2023 at 08:01:54PM +0900, Akihiko Odaki wrote: > Yes, that's right. The guest can dynamically create and delete VFs. The > device is emulated by QEMU: igb, an Intel NIC recently added to QEMU and > projected to be released as part of QEMU 8.0. Ah great, that's really useful, I'll add it to my tests > > Yes, I think this is an issue in the virtio-iommu driver, which should be > > sending a DETACH request when the VF is disabled, likely from > > viommu_release_device(). I'll work on a fix unless you would like to do it > > It will be nice if you prepare a fix. I will test your patch with my > workload if you share it with me. I sent a fix: https://lore.kernel.org/linux-iommu/20230414150744.562456-1-jean-phili...@linaro.org/ Thank you for reporting this, it must have been annoying to debug Thanks, Jean ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: virtio-iommu hotplug issue
Hi, On 4/13/23 13:01, Akihiko Odaki wrote: > On 2023/04/13 19:40, Jean-Philippe Brucker wrote: >> Hello, >> >> On Thu, Apr 13, 2023 at 01:49:43PM +0900, Akihiko Odaki wrote: >>> Hi, >>> >>> Recently I encountered a problem with the combination of Linux's >>> virtio-iommu driver and QEMU when a SR-IOV virtual function gets >>> disabled. >>> I'd like to ask you what kind of solution is appropriate here and >>> implement >>> the solution if possible. >>> >>> A PCIe device implementing the SR-IOV specification exports a virtual >>> function, and the guest can enable or disable it at runtime by >>> writing to a >>> configuration register. This effectively looks like a PCI device is >>> hotplugged for the guest. >> >> Just so I understand this better: the guest gets a whole PCIe device PF >> that implements SR-IOV, and so the guest can dynamically create VFs? >> Out >> of curiosity, is that a hardware device assigned to the guest with VFIO, >> or a device emulated by QEMU? > > Yes, that's right. The guest can dynamically create and delete VFs. > The device is emulated by QEMU: igb, an Intel NIC recently added to > QEMU and projected to be released as part of QEMU 8.0. From below description In understand you then bind this emulated device to VFIO on guest, correct? > >> >>> In such a case, the kernel assumes the endpoint is >>> detached from the virtio-iommu domain, but QEMU actually does not >>> detach it. The QEMU virtio-iommu device executes commands from the virtio-iommu driver and my understanding is the VFIO infra is not in trouble here. As suggested by Jean, a detach command probably is missed. >>> >>> This inconsistent view of the removed device sometimes prevents the >>> VM from >>> correctly performing the following procedure, for example: >>> 1. Enable a VF. >>> 2. Disable the VF. >>> 3. Open a vfio container. >>> 4. Open the group which the PF belongs to. >>> 5. Add the group to the vfio container. >>> 6. Map some memory region. >>> 7. Close the group. >>> 8. Close the vfio container. >>> 9. Repeat 3-8 >>> >>> When the VF gets disabled, the kernel assumes the endpoint is >>> detached from >>> the IOMMU domain, but QEMU actually doesn't detach it. Later, the >>> domain >>> will be reused in step 3-8. >>> >>> In step 7, the PF will be detached, and the kernel thinks there is no >>> endpoint attached and the mapping the domain holds is cleared, but >>> the VF >>> endpoint is still attached and the mapping is kept intact. >>> >>> In step 9, the same domain will be reused again, and the kernel >>> requests to >>> create a new mapping, but it will conflict with the existing mapping >>> and >>> result in -EINVAL. >>> >>> This problem can be fixed by either of: >>> - requesting the detachment of the endpoint from the guest when the PCI >>> device is unplugged (the VF is disabled) >> >> Yes, I think this is an issue in the virtio-iommu driver, which >> should be >> sending a DETACH request when the VF is disabled, likely from >> viommu_release_device(). I'll work on a fix unless you would like to >> do it > > It will be nice if you prepare a fix. I will test your patch with my > workload if you share it with me. I can help testing too Thanks Eric > > Regards, > Akihiko Odaki > >> >>> - detecting that the PCI device is gone and automatically detach it on >>> QEMU-side. >>> >>> It is not completely clear for me which solution is more appropriate >>> as the >>> virtio-iommu specification is written in a way independent of the >>> endpoint >>> mechanism and does not say what should be done when a PCI device is >>> unplugged. >> >> Yes, I'm not sure it's in scope for the specification, it's more about >> software guidance >> >> Thanks, >> Jean > ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: virtio-iommu hotplug issue
Hello, On Thu, Apr 13, 2023 at 01:49:43PM +0900, Akihiko Odaki wrote: > Hi, > > Recently I encountered a problem with the combination of Linux's > virtio-iommu driver and QEMU when a SR-IOV virtual function gets disabled. > I'd like to ask you what kind of solution is appropriate here and implement > the solution if possible. > > A PCIe device implementing the SR-IOV specification exports a virtual > function, and the guest can enable or disable it at runtime by writing to a > configuration register. This effectively looks like a PCI device is > hotplugged for the guest. Just so I understand this better: the guest gets a whole PCIe device PF that implements SR-IOV, and so the guest can dynamically create VFs? Out of curiosity, is that a hardware device assigned to the guest with VFIO, or a device emulated by QEMU? > In such a case, the kernel assumes the endpoint is > detached from the virtio-iommu domain, but QEMU actually does not detach it. > > This inconsistent view of the removed device sometimes prevents the VM from > correctly performing the following procedure, for example: > 1. Enable a VF. > 2. Disable the VF. > 3. Open a vfio container. > 4. Open the group which the PF belongs to. > 5. Add the group to the vfio container. > 6. Map some memory region. > 7. Close the group. > 8. Close the vfio container. > 9. Repeat 3-8 > > When the VF gets disabled, the kernel assumes the endpoint is detached from > the IOMMU domain, but QEMU actually doesn't detach it. Later, the domain > will be reused in step 3-8. > > In step 7, the PF will be detached, and the kernel thinks there is no > endpoint attached and the mapping the domain holds is cleared, but the VF > endpoint is still attached and the mapping is kept intact. > > In step 9, the same domain will be reused again, and the kernel requests to > create a new mapping, but it will conflict with the existing mapping and > result in -EINVAL. > > This problem can be fixed by either of: > - requesting the detachment of the endpoint from the guest when the PCI > device is unplugged (the VF is disabled) Yes, I think this is an issue in the virtio-iommu driver, which should be sending a DETACH request when the VF is disabled, likely from viommu_release_device(). I'll work on a fix unless you would like to do it > - detecting that the PCI device is gone and automatically detach it on > QEMU-side. > > It is not completely clear for me which solution is more appropriate as the > virtio-iommu specification is written in a way independent of the endpoint > mechanism and does not say what should be done when a PCI device is > unplugged. Yes, I'm not sure it's in scope for the specification, it's more about software guidance Thanks, Jean ___ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization