Hello, Roger! On 10.06.21 10:54, Roger Pau Monné wrote: > On Fri, Jun 04, 2021 at 06:37:27AM +0000, Oleksandr Andrushchenko wrote: >> Hi, all! >> >> While working on PCI SR-IOV support for ARM I started porting [1] on top >> of current PCI on ARM support [2]. The question I have for this series >> is if we really need emulating SR-IOV code in Xen? >> >> I have implemented a PoC for SR-IOV on ARM [3] (please see the top 2 >> patches) >> and it "works for me": MSI support is still WIP, but I was able to see that >> VFs are properly seen in the guest and BARs are properly programmed in p2m. >> >> What I can't fully understand is if we can live with this approach or there >> are use-cases I can't see. >> >> Previously I've been told that this approach might not work on FreeBSD >> running >> as Domain-0, but is seems that "PCI Passthrough is not supported >> (Xen/FreeBSD)" >> anyways [4]. > PCI passthorgh is not supported on FreeBSD dom0 because PCI > passthrough is not supported by Xen itself when using a PVH dom0, and > that's the only mode FreeBSD dom0 can use.
So, it is still not clear to me: how and if PCI passthrough is supported on FreeBSD, what are the scenarios and requirements for that? > > PHYSDEVOP_pci_device_add can be added to FreeBSD, so it could be made > to work. I however think this is not the proper way to implement > SR-IOV support. I was not able to find any support for PHYSDEVOP_XXX in FreeBSD code, could you please point me to where are these used? If they are not, so how Xen under FreeBSD knows about PCI devices? I am trying to extrapolate my knowledge of how Linux does that (during PCI enumeration in Domain-0 we use hypercalls) > >> I also see ACRN hypervisor [5] implements SR-IOV inside it which makes >> me think I >> miss some important use-case on x86 though. >> >> I would like to ask for any advise with SR-IOV in hypervisor respect, >> any pointers >> to documentation or any other source which might be handy in deciding if >> we do >> need SR-IOV complexity in Xen. >> >> And it does bring complexity if you compare [1] and [3])... >> >> A bit of technical details on the approach implemented [3]: >> 1. We rely on PHYSDEVOP_pci_device_add >> 2. We rely on Domain-0 SR-IOV drivers to instantiate VFs >> 3. BARs are programmed in p2m implementing guest view on those (we have >> extended >> vPCI code for that and this path is used for both "normal" devices and >> VFs the same way) >> 4. No need to trap PCI_SRIOV_CTRL >> 5. No need to wait 100ms in Xen before attempting to access VF registers >> when >> enabling virtual functions on the PF - this is handled by Domain-0 itself. > I think the SR-IOV capability should be handled like any other PCI > capability, ie: like we currently handle MSI or MSI-X in vPCI. > > It's likely that using some kind of hypercall in order to deal with > SR-IOV could make this easier to implement in Xen, but that just adds > more code to all OSes that want to run as the hardware domain. I didn't introduce any new, but PHYSDEVOP_pci_device_add was enough. The rest I did in Xen itself wrt SR-IOV. > > OTOH if we properly trap accesses to the SR-IOV capability (like it > was proposed in [1] from your references) we won't have to modify OSes > that want to run as hardware domains in order to handle SR-IOV devices. Out of curiosity, could you please name a few? I do understand that we do want to support unmodified OSes and this is indeed important. But, still what are the other OSes which do support Xen + PCI passthrough? > > IMO going for the hypercall option seems easier now, but adds a burden > to all OSes that want to manage SR-IOV devices that will hurt us long > term. Again, I was able to make it somewhat work with PHYSDEVOP_pci_device_add only. > > Thanks, Roger. Thank you for your valuable comments, Oleksandr > >> Thank you in advance, >> Oleksandr >> >> [1] >> https://urldefense.com/v3/__https://lists.xenproject.org/archives/html/xen-devel/2018-07/msg01494.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CezmvSZGg$ >> [lists[.]xenproject[.]org] >> [2] >> https://urldefense.com/v3/__https://gitlab.com/xen-project/fusa/xen-integration/-/tree/integration/pci-passthrough__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8Cf3Z8Yk7w$ >> [gitlab[.]com] >> [3] >> https://urldefense.com/v3/__https://github.com/xen-troops/xen/commits/pci_phase2__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CfncIdgvw$ >> [github[.]com] >> [4] >> https://urldefense.com/v3/__https://wiki.freebsd.org/Xen__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CceLOPdyg$ >> [wiki[.]freebsd[.]org] >> [5] >> https://urldefense.com/v3/__https://projectacrn.github.io/latest/tutorials/sriov_virtualization.html__;!!GF_29dbcQIUBPA!mIpLEvBmxRoFWUgYc7MFAF032kDGd897NP9t6d1fsc9uZoLHSW97GX8xAWKhvlFF8CdjqHPxZQ$ >> [projectacrn[.]github[.]io]
