Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
Peter Xu writes: > On Fri, Oct 18, 2019 at 11:19:55PM -0700, Jintack Lim wrote: >> On Fri, Oct 18, 2019 at 8:37 PM Peter Xu wrote: >> > >> > On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote: >> > > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu wrote: >> > > > >> > > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: >> > > > > Hi, >> > > > >> > > > Hello, Jintack, >> > > > >> > > Hi Peter, >> > > >> > > > > >> > > > > I'm trying to pass through a physical network device to a nested VM >> > > > > using virtual IOMMU. While I was able to do it successfully using KVM >> > > > > and Xen guest hypervisors running in a VM respectively, I couldn't do >> > > > > it with Hyper-V as I described below. I wonder if anyone have >> > > > > successfully used virtual IOMMU in other hypervisors other than KVM >> > > > > and Xen? (like Hyper-V or VMware) >> > > > > >> > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the >> > > > > underlying hardware is not capable of doing passthrough. The exact >> > > > > error message is as follows. >> > > > > >> > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons >> > > > > The chipset on the system does not do DMA remapping, without which >> > > > > SR-IOV cannot be supported. >> > > > > >> > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have >> > > > > enabled iommu in windows boot loader[1], and I see differences when >> > > > > booing a Windows VM with and without virtual IOMMU. I also checked >> > > > > that virtual IOMMU traces are printed. >> > > > >> > > > What traces have you checked? More explicitly, have you seen DMAR >> > > > enabled and page table setup for that specific device to be >> > > > pass-throughed? >> > > >> > > Thanks for the pointers. I checked that DMAR is NOT enabled. The only >> > > registers that Windows guest accessed were Version Register, >> > > Capability Register, and Extended Capability Register. On the other >> > > hand, a Linux guest accessed other registers and enabled DMAR. >> > > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything >> > > interesting there? >> > > http://paste.ubuntu.com/p/YcSyxG9Z3x/ >> > >> > Then I feel like Windows is reluctant to enable DMAR due to lacking of >> > some caps. >> > >> > > >> > > > >> > > > > >> > > > > I have tried multiple KVM/QEMU versions including the latest ones >> > > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers >> > > > > (2016 and 2019), but I see the same result. [4] >> > > > > >> > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or >> > > > > VMware successfully, especially for passthrough. I also appreciate if >> > > > > somebody can point out any configuration errors I have. >> > > > > >> > > > > Here's the qemu command line I use, basically from the QEMU vt-d >> > > > > page[2] and Hyper-v on KVM from kvmforum [3]. >> > > > > >> > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device >> > > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M >> > > > >> > > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest >> > > > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered >> > > > issues when trying to pass the SVVP Windows test with this, in which >> > > > 4-level is required. I'm not sure whether whether that is required in >> > > > general usages of vIOMMU in Windows. >> > > >> > > I just tried the option you mentioned, but it didn't change anything. >> > > BTW, what version of Windows was it? >> > >> > Sorry I don't remember that. I didn't do the test but I was just >> > acknowledged that with it the test passed. I assume you're using the >> > latest QEMU here because I know Windows could require another >> > capability (DMA draining) and it should be on by default in latest >> > qemu master. >> >> Thanks. Yes, I plan to use v2.11.0 eventually, but I'm trying to make >> things work with the latest version first. >> >> > >> > At that time the complete cmdline to pass the test should be: >> > >> > -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on >> > >> > I also don't remember on why caching-mode needs to be off at that >> > time (otherwise SVVP fails too). >> >> Thanks for providing the cmdline. However, turning off the >> caching-mode with an assigned device resulted in the following error >> on VM boot. >> "We need to set caching-mode=on for intel-iommu to enable device assignment." >> Does this mean that we can't assign a physical device all the way to a >> nested VM with a Windows L1 hypervisor as of now? >> >> Without assigning a device, I was able to boot a Windows VM with the >> cmdline above and I see that DMAR in vIOMMU is enabled. Windows still >> complains about DMA remapping, though. I'll investigate further. > > We're going to have other ways to do device assignment in the future > leveraging the coming nested device page tables just like ARM, but > it's still a long way
Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
On Fri, Oct 18, 2019 at 11:19:55PM -0700, Jintack Lim wrote: > On Fri, Oct 18, 2019 at 8:37 PM Peter Xu wrote: > > > > On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote: > > > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu wrote: > > > > > > > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > > > > > Hi, > > > > > > > > Hello, Jintack, > > > > > > > Hi Peter, > > > > > > > > > > > > > I'm trying to pass through a physical network device to a nested VM > > > > > using virtual IOMMU. While I was able to do it successfully using KVM > > > > > and Xen guest hypervisors running in a VM respectively, I couldn't do > > > > > it with Hyper-V as I described below. I wonder if anyone have > > > > > successfully used virtual IOMMU in other hypervisors other than KVM > > > > > and Xen? (like Hyper-V or VMware) > > > > > > > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the > > > > > underlying hardware is not capable of doing passthrough. The exact > > > > > error message is as follows. > > > > > > > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons > > > > > The chipset on the system does not do DMA remapping, without which > > > > > SR-IOV cannot be supported. > > > > > > > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > > > > > enabled iommu in windows boot loader[1], and I see differences when > > > > > booing a Windows VM with and without virtual IOMMU. I also checked > > > > > that virtual IOMMU traces are printed. > > > > > > > > What traces have you checked? More explicitly, have you seen DMAR > > > > enabled and page table setup for that specific device to be > > > > pass-throughed? > > > > > > Thanks for the pointers. I checked that DMAR is NOT enabled. The only > > > registers that Windows guest accessed were Version Register, > > > Capability Register, and Extended Capability Register. On the other > > > hand, a Linux guest accessed other registers and enabled DMAR. > > > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything > > > interesting there? > > > http://paste.ubuntu.com/p/YcSyxG9Z3x/ > > > > Then I feel like Windows is reluctant to enable DMAR due to lacking of > > some caps. > > > > > > > > > > > > > > > > > > > I have tried multiple KVM/QEMU versions including the latest ones > > > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > > > > > (2016 and 2019), but I see the same result. [4] > > > > > > > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > > > > > VMware successfully, especially for passthrough. I also appreciate if > > > > > somebody can point out any configuration errors I have. > > > > > > > > > > Here's the qemu command line I use, basically from the QEMU vt-d > > > > > page[2] and Hyper-v on KVM from kvmforum [3]. > > > > > > > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > > > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M > > > > > > > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest > > > > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered > > > > issues when trying to pass the SVVP Windows test with this, in which > > > > 4-level is required. I'm not sure whether whether that is required in > > > > general usages of vIOMMU in Windows. > > > > > > I just tried the option you mentioned, but it didn't change anything. > > > BTW, what version of Windows was it? > > > > Sorry I don't remember that. I didn't do the test but I was just > > acknowledged that with it the test passed. I assume you're using the > > latest QEMU here because I know Windows could require another > > capability (DMA draining) and it should be on by default in latest > > qemu master. > > Thanks. Yes, I plan to use v2.11.0 eventually, but I'm trying to make > things work with the latest version first. > > > > > At that time the complete cmdline to pass the test should be: > > > > -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on > > > > I also don't remember on why caching-mode needs to be off at that > > time (otherwise SVVP fails too). > > Thanks for providing the cmdline. However, turning off the > caching-mode with an assigned device resulted in the following error > on VM boot. > "We need to set caching-mode=on for intel-iommu to enable device assignment." > Does this mean that we can't assign a physical device all the way to a > nested VM with a Windows L1 hypervisor as of now? > > Without assigning a device, I was able to boot a Windows VM with the > cmdline above and I see that DMAR in vIOMMU is enabled. Windows still > complains about DMA remapping, though. I'll investigate further. We're going to have other ways to do device assignment in the future leveraging the coming nested device page tables just like ARM, but it's still a long way until even the hardware is ready... And we also don't know whether Microsoft will be unhappy again on these new bits. :) So, I
Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
On Fri, Oct 18, 2019 at 8:37 PM Peter Xu wrote: > > On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote: > > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu wrote: > > > > > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > > > > Hi, > > > > > > Hello, Jintack, > > > > > Hi Peter, > > > > > > > > > > I'm trying to pass through a physical network device to a nested VM > > > > using virtual IOMMU. While I was able to do it successfully using KVM > > > > and Xen guest hypervisors running in a VM respectively, I couldn't do > > > > it with Hyper-V as I described below. I wonder if anyone have > > > > successfully used virtual IOMMU in other hypervisors other than KVM > > > > and Xen? (like Hyper-V or VMware) > > > > > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the > > > > underlying hardware is not capable of doing passthrough. The exact > > > > error message is as follows. > > > > > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons > > > > The chipset on the system does not do DMA remapping, without which > > > > SR-IOV cannot be supported. > > > > > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > > > > enabled iommu in windows boot loader[1], and I see differences when > > > > booing a Windows VM with and without virtual IOMMU. I also checked > > > > that virtual IOMMU traces are printed. > > > > > > What traces have you checked? More explicitly, have you seen DMAR > > > enabled and page table setup for that specific device to be > > > pass-throughed? > > > > Thanks for the pointers. I checked that DMAR is NOT enabled. The only > > registers that Windows guest accessed were Version Register, > > Capability Register, and Extended Capability Register. On the other > > hand, a Linux guest accessed other registers and enabled DMAR. > > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything > > interesting there? > > http://paste.ubuntu.com/p/YcSyxG9Z3x/ > > Then I feel like Windows is reluctant to enable DMAR due to lacking of > some caps. > > > > > > > > > > > > > > I have tried multiple KVM/QEMU versions including the latest ones > > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > > > > (2016 and 2019), but I see the same result. [4] > > > > > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > > > > VMware successfully, especially for passthrough. I also appreciate if > > > > somebody can point out any configuration errors I have. > > > > > > > > Here's the qemu command line I use, basically from the QEMU vt-d > > > > page[2] and Hyper-v on KVM from kvmforum [3]. > > > > > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M > > > > > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest > > > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered > > > issues when trying to pass the SVVP Windows test with this, in which > > > 4-level is required. I'm not sure whether whether that is required in > > > general usages of vIOMMU in Windows. > > > > I just tried the option you mentioned, but it didn't change anything. > > BTW, what version of Windows was it? > > Sorry I don't remember that. I didn't do the test but I was just > acknowledged that with it the test passed. I assume you're using the > latest QEMU here because I know Windows could require another > capability (DMA draining) and it should be on by default in latest > qemu master. Thanks. Yes, I plan to use v2.11.0 eventually, but I'm trying to make things work with the latest version first. > > At that time the complete cmdline to pass the test should be: > > -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on > > I also don't remember on why caching-mode needs to be off at that > time (otherwise SVVP fails too). Thanks for providing the cmdline. However, turning off the caching-mode with an assigned device resulted in the following error on VM boot. "We need to set caching-mode=on for intel-iommu to enable device assignment." Does this mean that we can't assign a physical device all the way to a nested VM with a Windows L1 hypervisor as of now? Without assigning a device, I was able to boot a Windows VM with the cmdline above and I see that DMAR in vIOMMU is enabled. Windows still complains about DMA remapping, though. I'll investigate further. > > -- > Peter Xu >
Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote: > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu wrote: > > > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > > > Hi, > > > > Hello, Jintack, > > > Hi Peter, > > > > > > > I'm trying to pass through a physical network device to a nested VM > > > using virtual IOMMU. While I was able to do it successfully using KVM > > > and Xen guest hypervisors running in a VM respectively, I couldn't do > > > it with Hyper-V as I described below. I wonder if anyone have > > > successfully used virtual IOMMU in other hypervisors other than KVM > > > and Xen? (like Hyper-V or VMware) > > > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the > > > underlying hardware is not capable of doing passthrough. The exact > > > error message is as follows. > > > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons > > > The chipset on the system does not do DMA remapping, without which > > > SR-IOV cannot be supported. > > > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > > > enabled iommu in windows boot loader[1], and I see differences when > > > booing a Windows VM with and without virtual IOMMU. I also checked > > > that virtual IOMMU traces are printed. > > > > What traces have you checked? More explicitly, have you seen DMAR > > enabled and page table setup for that specific device to be > > pass-throughed? > > Thanks for the pointers. I checked that DMAR is NOT enabled. The only > registers that Windows guest accessed were Version Register, > Capability Register, and Extended Capability Register. On the other > hand, a Linux guest accessed other registers and enabled DMAR. > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything > interesting there? > http://paste.ubuntu.com/p/YcSyxG9Z3x/ Then I feel like Windows is reluctant to enable DMAR due to lacking of some caps. > > > > > > > > > I have tried multiple KVM/QEMU versions including the latest ones > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > > > (2016 and 2019), but I see the same result. [4] > > > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > > > VMware successfully, especially for passthrough. I also appreciate if > > > somebody can point out any configuration errors I have. > > > > > > Here's the qemu command line I use, basically from the QEMU vt-d > > > page[2] and Hyper-v on KVM from kvmforum [3]. > > > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M > > > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest > > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered > > issues when trying to pass the SVVP Windows test with this, in which > > 4-level is required. I'm not sure whether whether that is required in > > general usages of vIOMMU in Windows. > > I just tried the option you mentioned, but it didn't change anything. > BTW, what version of Windows was it? Sorry I don't remember that. I didn't do the test but I was just acknowledged that with it the test passed. I assume you're using the latest QEMU here because I know Windows could require another capability (DMA draining) and it should be on by default in latest qemu master. At that time the complete cmdline to pass the test should be: -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on I also don't remember on why caching-mode needs to be off at that time (otherwise SVVP fails too). -- Peter Xu
Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
On Mon, Oct 14, 2019 at 7:50 PM Peter Xu wrote: > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > > Hi, > > Hello, Jintack, > Hi Peter, > > > > I'm trying to pass through a physical network device to a nested VM > > using virtual IOMMU. While I was able to do it successfully using KVM > > and Xen guest hypervisors running in a VM respectively, I couldn't do > > it with Hyper-V as I described below. I wonder if anyone have > > successfully used virtual IOMMU in other hypervisors other than KVM > > and Xen? (like Hyper-V or VMware) > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the > > underlying hardware is not capable of doing passthrough. The exact > > error message is as follows. > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons > > The chipset on the system does not do DMA remapping, without which > > SR-IOV cannot be supported. > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > > enabled iommu in windows boot loader[1], and I see differences when > > booing a Windows VM with and without virtual IOMMU. I also checked > > that virtual IOMMU traces are printed. > > What traces have you checked? More explicitly, have you seen DMAR > enabled and page table setup for that specific device to be > pass-throughed? Thanks for the pointers. I checked that DMAR is NOT enabled. The only registers that Windows guest accessed were Version Register, Capability Register, and Extended Capability Register. On the other hand, a Linux guest accessed other registers and enabled DMAR. Here's a link to the trace I got using QEMU 4.1.0. Do you see anything interesting there? http://paste.ubuntu.com/p/YcSyxG9Z3x/ > > > > > I have tried multiple KVM/QEMU versions including the latest ones > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > > (2016 and 2019), but I see the same result. [4] > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > > VMware successfully, especially for passthrough. I also appreciate if > > somebody can point out any configuration errors I have. > > > > Here's the qemu command line I use, basically from the QEMU vt-d > > page[2] and Hyper-v on KVM from kvmforum [3]. > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered > issues when trying to pass the SVVP Windows test with this, in which > 4-level is required. I'm not sure whether whether that is required in > general usages of vIOMMU in Windows. I just tried the option you mentioned, but it didn't change anything. BTW, what version of Windows was it? > > > q35,accel=kvm,kernel-irqchip=split -cpu > > host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -drive > > if=none,file=/vm/guest0.img,id=vda,cache=none,format=raw -device > > virtio-blk-pci,drive=vda --nographic -qmp > > unix:/var/run/qmp,server,nowait -serial > > telnet:127.0.0.1:,server,nowait -netdev > > user,id=net0,hostfwd=tcp::-:22 -device > > virtio-net-pci,netdev=net0,mac=de:ad:be:ef:f2:12 -netdev > > tap,id=net1,vhost=on,helper=/srv/vm/qemu/qemu-bridge-helper -device > > virtio-net-pci,netdev=net1,disable-modern=off,disable-legacy=on,mac=de:ad:be:ef:f2:11 > > -device vfio-pci,host=:06:10.0,id=net2 -monitor stdio -usb -device > > usb-tablet -rtc base=localtime,clock=host -vnc 127.0.0.1:4 --cdrom > > win19.iso --drive file=virtio-win.iso,index=3,media=cdrom > > -- > Peter Xu >
Re: Using virtual IOMMU in guest hypervisors other than KVM and Xen?
On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > Hi, Hello, Jintack, > > I'm trying to pass through a physical network device to a nested VM > using virtual IOMMU. While I was able to do it successfully using KVM > and Xen guest hypervisors running in a VM respectively, I couldn't do > it with Hyper-V as I described below. I wonder if anyone have > successfully used virtual IOMMU in other hypervisors other than KVM > and Xen? (like Hyper-V or VMware) > > The issue I have with Hyper-V is that Hyper-V gives an error that the > underlying hardware is not capable of doing passthrough. The exact > error message is as follows. > > Windows Power-shell > (Get-VMHost).IovSupportReasons > The chipset on the system does not do DMA remapping, without which > SR-IOV cannot be supported. > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > enabled iommu in windows boot loader[1], and I see differences when > booing a Windows VM with and without virtual IOMMU. I also checked > that virtual IOMMU traces are printed. What traces have you checked? More explicitly, have you seen DMAR enabled and page table setup for that specific device to be pass-throughed? > > I have tried multiple KVM/QEMU versions including the latest ones > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > (2016 and 2019), but I see the same result. [4] > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > VMware successfully, especially for passthrough. I also appreciate if > somebody can point out any configuration errors I have. > > Here's the qemu command line I use, basically from the QEMU vt-d > page[2] and Hyper-v on KVM from kvmforum [3]. > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered issues when trying to pass the SVVP Windows test with this, in which 4-level is required. I'm not sure whether whether that is required in general usages of vIOMMU in Windows. > q35,accel=kvm,kernel-irqchip=split -cpu > host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -drive > if=none,file=/vm/guest0.img,id=vda,cache=none,format=raw -device > virtio-blk-pci,drive=vda --nographic -qmp > unix:/var/run/qmp,server,nowait -serial > telnet:127.0.0.1:,server,nowait -netdev > user,id=net0,hostfwd=tcp::-:22 -device > virtio-net-pci,netdev=net0,mac=de:ad:be:ef:f2:12 -netdev > tap,id=net1,vhost=on,helper=/srv/vm/qemu/qemu-bridge-helper -device > virtio-net-pci,netdev=net1,disable-modern=off,disable-legacy=on,mac=de:ad:be:ef:f2:11 > -device vfio-pci,host=:06:10.0,id=net2 -monitor stdio -usb -device > usb-tablet -rtc base=localtime,clock=host -vnc 127.0.0.1:4 --cdrom > win19.iso --drive file=virtio-win.iso,index=3,media=cdrom -- Peter Xu
Using virtual IOMMU in guest hypervisors other than KVM and Xen?
Hi, I'm trying to pass through a physical network device to a nested VM using virtual IOMMU. While I was able to do it successfully using KVM and Xen guest hypervisors running in a VM respectively, I couldn't do it with Hyper-V as I described below. I wonder if anyone have successfully used virtual IOMMU in other hypervisors other than KVM and Xen? (like Hyper-V or VMware) The issue I have with Hyper-V is that Hyper-V gives an error that the underlying hardware is not capable of doing passthrough. The exact error message is as follows. Windows Power-shell > (Get-VMHost).IovSupportReasons The chipset on the system does not do DMA remapping, without which SR-IOV cannot be supported. I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have enabled iommu in windows boot loader[1], and I see differences when booing a Windows VM with and without virtual IOMMU. I also checked that virtual IOMMU traces are printed. I have tried multiple KVM/QEMU versions including the latest ones (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers (2016 and 2019), but I see the same result. [4] I'd love to hear if somebody is using virtual IOMMU in Hyper-V or VMware successfully, especially for passthrough. I also appreciate if somebody can point out any configuration errors I have. Here's the qemu command line I use, basically from the QEMU vt-d page[2] and Hyper-v on KVM from kvmforum [3]. ./qemu/x86_64-softmmu/qemu-system-x86_64 -device intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M q35,accel=kvm,kernel-irqchip=split -cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -drive if=none,file=/vm/guest0.img,id=vda,cache=none,format=raw -device virtio-blk-pci,drive=vda --nographic -qmp unix:/var/run/qmp,server,nowait -serial telnet:127.0.0.1:,server,nowait -netdev user,id=net0,hostfwd=tcp::-:22 -device virtio-net-pci,netdev=net0,mac=de:ad:be:ef:f2:12 -netdev tap,id=net1,vhost=on,helper=/srv/vm/qemu/qemu-bridge-helper -device virtio-net-pci,netdev=net1,disable-modern=off,disable-legacy=on,mac=de:ad:be:ef:f2:11 -device vfio-pci,host=:06:10.0,id=net2 -monitor stdio -usb -device usb-tablet -rtc base=localtime,clock=host -vnc 127.0.0.1:4 --cdrom win19.iso --drive file=virtio-win.iso,index=3,media=cdrom Thanks, Jintack [1] https://social.technet.microsoft.com/Forums/en-US/a7c2940a-af32-4dab-8b31-7a605e8cf075/a-hypervisor-feature-is-not-available-to-the-user?forum=WinServerPreview [2] https://wiki.qemu.org/Features/VT-d [3] https://www.linux-kvm.org/images/6/6a/HyperV-KVM.pdf [4] https://www.mail-archive.com/qemu-devel@nongnu.org/msg568963.html
