Re: vGPU support in CloudStack on Ubuntu KVM
or dev in :37:00.2 :86:00.2* > > *do * > > * echo -n "$dev" > /sys/bus/pci/drivers/xhci_hcd/unbind* > > * echo -n "$dev" > /sys/bus/pci/drivers/vfio-pci/bind* > > *done* > > > *exit 0* > > > Then # update-initramfs -u -k all > > > 9. Once everything is good, go ahead and reboot the machine, and your > devices should look something like the below, the *vfio-pci* kernel module > should be the "Kernel driver in use". > > > 37:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce > RTX 2080 Ti Rev. A] [10de:1e07] (rev a1) > > Subsystem: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] > [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: nouveau > > 37:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio > Controller [10de:10f7] (rev a1) > > Subsystem: NVIDIA Corporation TU102 High Definition Audio Controller > [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: snd_hda_intel > > 37:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host > Controller [10de:1ad6] (rev a1) > > Subsystem: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: xhci_pci > > 37:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C > UCSI Controller [10de:1ad7] (rev a1) > > Subsystem: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:12fa] > > Kernel driver in use: vfio-pci > > > 86:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce > RTX 2080 Ti Rev. A] [10de:1e07] (rev a1) > > Subsystem: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] > [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: nouveau > > 86:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio > Controller [10de:10f7] (rev a1) > > Subsystem: NVIDIA Corporation TU102 High Definition Audio Controller > [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: snd_hda_intel > > 86:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host > Controller [10de:1ad6] (rev a1) > > Subsystem: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:12fa] > > Kernel driver in use: vfio-pci > > Kernel modules: xhci_pci > > 86:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C > UCSI Controller [10de:1ad7] (rev a1) > > Subsystem: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:12fa] > > Kernel driver in use: vfio-pci > > > 10. Go to CloudStack, create a VM as per the normal workflow, then stop it > and insert the additional settings as below. This can be done from the > WebUI and API as well. > > > extraconfig-1 >name='vfio' /> function='0x0' /> > > extraconfig-2 > * name='vfio' /> function='0x0' /> * > > > Replace the device address as per in your environment. We'd have to put 2 > extraconfigs as we have 2 GPUs. Even extraconfig-1 can accomodate for both > devices as well. > > > 11. Once done, schedule the VM on the host, CloudStack will compose the XML > with the additional configuration (in extraconifg-n) specified, and the > qemu process for that VM will have additional arguments as > > > -device vfio-pci,host=:37:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device > vfio-pci,host=:86:00.0,id=hostdev1,bus=pci.0,addr=0x7 > > > > 12. Once the VM is up, check using *lspci *or similar and finally install > nvidia-smi and confirm everything is working fine. > > > > Thanks > > On Sun, Nov 13, 2022 at 3:25 PM Alex Mattioli > > wrote: > > > Hi Jay, > > I'd love to hear more about how you implemented the GPU pass-through, and > > I think it could be quite useful for the community as well. > > > > Cheers > > Alex > > > > > > > > > > -Original Message- > > From: Jayanth Reddy > > Sent: 13 November 2022 10:43 > > To: users@cloudstack.apache.org > > Cc: Emil Karlsson > > Subject: Re: vGPU support in CloudStack on Ubuntu KVM > > > > Hi, > > AFAIK, vGPU and GPU are only supported on Xen Hypervisor as per > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/GPU+and+vGPU+support+for+CloudStack+Guest+VMs > > . > > Not sure about the vGPU but we managed to do a full GPU passthrough to a > > VM running on Ubuntu KVM Host. Interested to discuss further? > > > > Thanks > > > > > > On Wed, Oct 12, 2022 at 6:40 PM Pierre Le Fevre wrote: > > > > > Hi all, > > > I am currently trying to get vGPU to work in some of our VMs in > > > cloudstack to enable GPU acceleration in Jupyter Notebooks. > > > Our current setup is using CloudStack 4.17.1.0 on Ubuntu 20.04 with > > > KVM as a hypervisor. > > > It seems like there is some support for vGPU in CloudStack but I can't > > > find proper documentation about compatibility with newer GPUs and > > > other hypervisors. > > > > > > I've tried installing all the proper drivers on the host machine with > > > a NVIDIA A6000 and it shows up properly in nvidia-smi. From the docs > > > available it seems like it should show up in the UI under the host > > > after adding it to our cluster, but no GPU appears. The dashboard also > > > reports 0 GPUs in the zone. > > > > > > Is this a limitation of KVM, or have some of you gotten this setup to > > work? > > > > > > All the best > > > Pierre > > > > > >
Re: vGPU support in CloudStack on Ubuntu KVM
TX 2080 Ti Rev. A] [10de:1e07] (rev a1) Subsystem: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] [10de:12fa] Kernel driver in use: vfio-pci Kernel modules: nouveau 86:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1) Subsystem: NVIDIA Corporation TU102 High Definition Audio Controller [10de:12fa] Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 86:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1) Subsystem: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:12fa] Kernel driver in use: vfio-pci Kernel modules: xhci_pci 86:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1) Subsystem: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:12fa] Kernel driver in use: vfio-pci 10. Go to CloudStack, create a VM as per the normal workflow, then stop it and insert the additional settings as below. This can be done from the WebUI and API as well. extraconfig-1 extraconfig-2 * * Replace the device address as per in your environment. We'd have to put 2 extraconfigs as we have 2 GPUs. Even extraconfig-1 can accomodate for both devices as well. 11. Once done, schedule the VM on the host, CloudStack will compose the XML with the additional configuration (in extraconifg-n) specified, and the qemu process for that VM will have additional arguments as -device vfio-pci,host=:37:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=:86:00.0,id=hostdev1,bus=pci.0,addr=0x7 12. Once the VM is up, check using *lspci *or similar and finally install nvidia-smi and confirm everything is working fine. Thanks On Sun, Nov 13, 2022 at 3:25 PM Alex Mattioli wrote: > Hi Jay, > I'd love to hear more about how you implemented the GPU pass-through, and > I think it could be quite useful for the community as well. > > Cheers > Alex > > > > > -Original Message- > From: Jayanth Reddy > Sent: 13 November 2022 10:43 > To: users@cloudstack.apache.org > Cc: Emil Karlsson > Subject: Re: vGPU support in CloudStack on Ubuntu KVM > > Hi, > AFAIK, vGPU and GPU are only supported on Xen Hypervisor as per > https://cwiki.apache.org/confluence/display/CLOUDSTACK/GPU+and+vGPU+support+for+CloudStack+Guest+VMs > . > Not sure about the vGPU but we managed to do a full GPU passthrough to a > VM running on Ubuntu KVM Host. Interested to discuss further? > > Thanks > > > On Wed, Oct 12, 2022 at 6:40 PM Pierre Le Fevre wrote: > > > Hi all, > > I am currently trying to get vGPU to work in some of our VMs in > > cloudstack to enable GPU acceleration in Jupyter Notebooks. > > Our current setup is using CloudStack 4.17.1.0 on Ubuntu 20.04 with > > KVM as a hypervisor. > > It seems like there is some support for vGPU in CloudStack but I can't > > find proper documentation about compatibility with newer GPUs and > > other hypervisors. > > > > I've tried installing all the proper drivers on the host machine with > > a NVIDIA A6000 and it shows up properly in nvidia-smi. From the docs > > available it seems like it should show up in the UI under the host > > after adding it to our cluster, but no GPU appears. The dashboard also > > reports 0 GPUs in the zone. > > > > Is this a limitation of KVM, or have some of you gotten this setup to > work? > > > > All the best > > Pierre > > >
RE: vGPU support in CloudStack on Ubuntu KVM
Hi Jay, I'd love to hear more about how you implemented the GPU pass-through, and I think it could be quite useful for the community as well. Cheers Alex -Original Message- From: Jayanth Reddy Sent: 13 November 2022 10:43 To: users@cloudstack.apache.org Cc: Emil Karlsson Subject: Re: vGPU support in CloudStack on Ubuntu KVM Hi, AFAIK, vGPU and GPU are only supported on Xen Hypervisor as per https://cwiki.apache.org/confluence/display/CLOUDSTACK/GPU+and+vGPU+support+for+CloudStack+Guest+VMs. Not sure about the vGPU but we managed to do a full GPU passthrough to a VM running on Ubuntu KVM Host. Interested to discuss further? Thanks On Wed, Oct 12, 2022 at 6:40 PM Pierre Le Fevre wrote: > Hi all, > I am currently trying to get vGPU to work in some of our VMs in > cloudstack to enable GPU acceleration in Jupyter Notebooks. > Our current setup is using CloudStack 4.17.1.0 on Ubuntu 20.04 with > KVM as a hypervisor. > It seems like there is some support for vGPU in CloudStack but I can't > find proper documentation about compatibility with newer GPUs and > other hypervisors. > > I've tried installing all the proper drivers on the host machine with > a NVIDIA A6000 and it shows up properly in nvidia-smi. From the docs > available it seems like it should show up in the UI under the host > after adding it to our cluster, but no GPU appears. The dashboard also > reports 0 GPUs in the zone. > > Is this a limitation of KVM, or have some of you gotten this setup to work? > > All the best > Pierre >
Re: vGPU support in CloudStack on Ubuntu KVM
Hi, AFAIK, vGPU and GPU are only supported on Xen Hypervisor as per https://cwiki.apache.org/confluence/display/CLOUDSTACK/GPU+and+vGPU+support+for+CloudStack+Guest+VMs. Not sure about the vGPU but we managed to do a full GPU passthrough to a VM running on Ubuntu KVM Host. Interested to discuss further? Thanks On Wed, Oct 12, 2022 at 6:40 PM Pierre Le Fevre wrote: > Hi all, > I am currently trying to get vGPU to work in some of our VMs in cloudstack > to enable GPU acceleration in Jupyter Notebooks. > Our current setup is using CloudStack 4.17.1.0 on Ubuntu 20.04 with KVM as > a hypervisor. > It seems like there is some support for vGPU in CloudStack but I can't find > proper documentation about compatibility with newer GPUs and other > hypervisors. > > I've tried installing all the proper drivers on the host machine with a > NVIDIA A6000 and it shows up properly in nvidia-smi. From the docs > available it seems like it should show up in the UI under the host after > adding it to our cluster, but no GPU appears. The dashboard also reports 0 > GPUs in the zone. > > Is this a limitation of KVM, or have some of you gotten this setup to work? > > All the best > Pierre >
