Re: [ovirt-users] VM support for Infiniband
On 9/30/2016 10:10 AM, Kevin Hung wrote: oVirt sees the HCA as SR-IOV capable and it sees the VFs, so I believe I have set up SR-IOV correctly. I was able to do a generic passthrough of the HCA, but I was not able to pass through the VFs to more than one VM probably because they are all in the same IOMMU group. I first tried following the oVirt 3.6 Deep Dive video on SR-IOV [1] but was stuck because of the bridging problem. [1] https://www.youtube.com/watch?v=A-MROZ8D06Y I was able to pass through the VFs after fixing the IOMMU issue. I had to read up on IOMMU groups and discovered that the PCIe root ports on Xeon E3 processors do not support ACS [1], which is needed for the VFs to be in different IOMMU groups. I was able to plug the HCA into a different PCIe slot which was connected to the PCH instead, allowing VFIO to see the VFs in different IOMMU groups. The way SR-IOV works for Ethernet network adapters in oVirt seems to be really nice as the scheduler takes care of the pass-through. Is this possible for Infiniband HCAs or is that a technical/architectural limitation? [1] http://vfio.blogspot.com.ar/2014/08/iommu-groups-inside-and-out.html Thanks, Kevin Hung ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM support for Infiniband
On 9/30/2016 9:28 AM, Michal Skrivanek wrote: On 30 Sep 2016, at 03:56, Kevin Hung wrote: Hello, I have been trying (unsuccessfully) to add my Infiniband HCAs on my VM hosts to oVirt 4.0.4. I am not using IBoIP, just plain Infiniband. I would like to be able to pass through the card (or better yet the VF from SR-IOV) to the guest VM. SR-IOV should work (or general PCI passthrough), it’s then transparent to oVirt. Did you not succeed with setting up SR-IOV or generic passthrough of the HCA? oVirt sees the HCA as SR-IOV capable and it sees the VFs, so I believe I have set up SR-IOV correctly. I was able to do a generic passthrough of the HCA, but I was not able to pass through the VFs to more than one VM probably because they are all in the same IOMMU group. I first tried following the oVirt 3.6 Deep Dive video on SR-IOV [1] but was stuck because of the bridging problem. From what I can figure out, oVirt is trying to bridge the interface but I don't think that will work as shown here [1]. Does oVirt currently have support for passing through Infiniband to VMs? If not, are there any plans to support it? Well, at least for generic passthrough the PCI device would be detached from the host so should not be recognized/touched by oVirt networking code at all. Thanks, michal [1] http://lists.ovirt.org/pipermail/users/2016-July/040898.html Thanks, Kevin Hung ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users [1] https://www.youtube.com/watch?v=A-MROZ8D06Y ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] VM support for Infiniband
> On 30 Sep 2016, at 03:56, Kevin Hung wrote: > > Hello, > > I have been trying (unsuccessfully) to add my Infiniband HCAs on my VM hosts > to oVirt 4.0.4. I am not using IBoIP, just plain Infiniband. I would like to > be able to pass through the card (or better yet the VF from SR-IOV) to the > guest VM. SR-IOV should work (or general PCI passthrough), it’s then transparent to oVirt. Did you not succeed with setting up SR-IOV or generic passthrough of the HCA? > > From what I can figure out, oVirt is trying to bridge the interface but I > don't think that will work as shown here [1]. Does oVirt currently have > support for passing through Infiniband to VMs? If not, are there any plans to > support it? Well, at least for generic passthrough the PCI device would be detached from the host so should not be recognized/touched by oVirt networking code at all. Thanks, michal > > [1] http://lists.ovirt.org/pipermail/users/2016-July/040898.html > > Thanks, > Kevin Hung > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users