On Fri, Oct 18, 2019 at 8:37 PM Peter Xu <pet...@redhat.com> wrote: > > On Wed, Oct 16, 2019 at 03:01:22PM -0700, Jintack Lim wrote: > > On Mon, Oct 14, 2019 at 7:50 PM Peter Xu <pet...@redhat.com> wrote: > > > > > > On Mon, Oct 14, 2019 at 01:28:49PM -0700, Jintack Lim wrote: > > > > Hi, > > > > > > Hello, Jintack, > > > > > Hi Peter, > > > > > > > > > > I'm trying to pass through a physical network device to a nested VM > > > > using virtual IOMMU. While I was able to do it successfully using KVM > > > > and Xen guest hypervisors running in a VM respectively, I couldn't do > > > > it with Hyper-V as I described below. I wonder if anyone have > > > > successfully used virtual IOMMU in other hypervisors other than KVM > > > > and Xen? (like Hyper-V or VMware) > > > > > > > > The issue I have with Hyper-V is that Hyper-V gives an error that the > > > > underlying hardware is not capable of doing passthrough. The exact > > > > error message is as follows. > > > > > > > > Windows Power-shell > (Get-VMHost).IovSupportReasons > > > > The chipset on the system does not do DMA remapping, without which > > > > SR-IOV cannot be supported. > > > > > > > > I'm pretty sure that Hyper-V recognizes virtual IOMMU, though; I have > > > > enabled iommu in windows boot loader[1], and I see differences when > > > > booing a Windows VM with and without virtual IOMMU. I also checked > > > > that virtual IOMMU traces are printed. > > > > > > What traces have you checked? More explicitly, have you seen DMAR > > > enabled and page table setup for that specific device to be > > > pass-throughed? > > > > Thanks for the pointers. I checked that DMAR is NOT enabled. The only > > registers that Windows guest accessed were Version Register, > > Capability Register, and Extended Capability Register. On the other > > hand, a Linux guest accessed other registers and enabled DMAR. > > Here's a link to the trace I got using QEMU 4.1.0. Do you see anything > > interesting there? > > http://paste.ubuntu.com/p/YcSyxG9Z3x/ > > Then I feel like Windows is reluctant to enable DMAR due to lacking of > some caps. > > > > > > > > > > > > > > I have tried multiple KVM/QEMU versions including the latest ones > > > > (kernel v5.3, QEMU 4.1.0) as well as two different Windows servers > > > > (2016 and 2019), but I see the same result. [4] > > > > > > > > I'd love to hear if somebody is using virtual IOMMU in Hyper-V or > > > > VMware successfully, especially for passthrough. I also appreciate if > > > > somebody can point out any configuration errors I have. > > > > > > > > Here's the qemu command line I use, basically from the QEMU vt-d > > > > page[2] and Hyper-v on KVM from kvmforum [3]. > > > > > > > > ./qemu/x86_64-softmmu/qemu-system-x86_64 -device > > > > intel-iommu,intremap=on,caching-mode=on -smp 6 -m 24G -M > > > > > > Have you tried to use 4-level IOMMU page table (aw-bits=48 on latest > > > QEMU, or x-aw-bits=48 on some old ones)? IIRC we've encountered > > > issues when trying to pass the SVVP Windows test with this, in which > > > 4-level is required. I'm not sure whether whether that is required in > > > general usages of vIOMMU in Windows. > > > > I just tried the option you mentioned, but it didn't change anything. > > BTW, what version of Windows was it? > > Sorry I don't remember that. I didn't do the test but I was just > acknowledged that with it the test passed. I assume you're using the > latest QEMU here because I know Windows could require another > capability (DMA draining) and it should be on by default in latest > qemu master.
Thanks. Yes, I plan to use v2.11.0 eventually, but I'm trying to make things work with the latest version first. > > At that time the complete cmdline to pass the test should be: > > -device intel-iommu,intremap=on,aw-bits=48,caching-mode=off,eim=on > > I also don't remember on why caching-mode needs to be off at that > time (otherwise SVVP fails too). Thanks for providing the cmdline. However, turning off the caching-mode with an assigned device resulted in the following error on VM boot. "We need to set caching-mode=on for intel-iommu to enable device assignment." Does this mean that we can't assign a physical device all the way to a nested VM with a Windows L1 hypervisor as of now? Without assigning a device, I was able to boot a Windows VM with the cmdline above and I see that DMAR in vIOMMU is enabled. Windows still complains about DMA remapping, though. I'll investigate further. > > -- > Peter Xu >