> On Oct 6, 2015, at 4:43 PM, Rain Maker <[email protected]> wrote: > > Unfortunately, no difference. WIth or without -hypervisor doesn't make > any difference to that flag. > > But, experimenting on, I found something <very> odd. > The 0x3a register is 0 when the VM boots up. > Even when I start a L2 VM, 0x3a is still 0. > > However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See > this terminal session, which is executed within the L1 VM ("kvmtest"). > http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png > > I was only executing mini.iso (an Ubuntu Netinstaller), and closing it > at the boot prompt. I did not do anything in the L2 VM. Both the Qemu > VM as the -enable-kvm VMs do boot. > > Is this how the MSR is supposed to react? > > AFAIK, the MSR can only be modified from kernelspace (which also > explains why Qemu would only reset it with -enable-kvm, there are no > kernelspace components used without it if I understand correctly) > > Looking at this, I can imagine that Windows does not detect a correct > value. It will get 0. Would it make sense to cygwin KVM and see if > that changes the MSR register? >
You have to use kvm to run (hardware assisted) nested virtualization. I am not sure why you think Windows will read 0 for the feature control msr but you have to use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2, you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available hardware features available to it and attempt to enable hardware virtualization. > Sincerely, > Roel Brook. > > 2015-10-05 23:17 GMT+02:00 Bandan Das <[email protected]>: >> Rain Maker <[email protected]> writes: >> >>> Qemu on Linux works fine. I did not even have to explicitly set >>> -hypervisor. It simply works. >> >> Sorry, I meant running Linux with "-hypervisor" to see if specifying >> that is somehow messing with the feature flags. >> >>> As does VirtualBox FYI. >>> >>> Booting with UEFI didn't make any difference. >>> >>> After A LOT of Googling, I believe that Hyper-V actually checks bit >>> 0x3a of the MSR register (instead of, as the error would . >>> This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the >>> value returned by rdmsr is "0", while on my host it is "5". For >> >> That seems odd. Even Linux wouldn't work if that value is 0. >> >>> Hyper-V to work (from what I understand), it should be either 4 or 5. >>> >>> Googling that, funny enough, brought me back to this list: >>> https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html. >>> I guess it really IS important to know what you're googling for to >>> find things fast... >>> >>> That thread simply says that the kernel is too old. Well, my host is >>> running 4.2, so should be new enough. >>> I'm a bit stuck. Any ideas? >>> >>> Sincerely, >>> Roel Brook >>> >>> >>> >>> >>> 2015-10-05 21:18 GMT+02:00 Bandan Das <[email protected]>: >>>> Rain Maker <[email protected]> writes: >>>> >>>>> Thanks Bandan. >>>>> >>>>> That helped a bit. It got me to the next hurdle, as you suspected. >>>>> >>>>> I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed, >>>>> and the installation now reports "Hyper-V cannot be installed because >>>>> virtualization support is not enabled in the BIOS.". >>>> >>>> Thanks for trying this out. >>>> >>>>> I am sure that vmx is passed. but "systeminfo" does report "Hyper-V >>>>> cannot be installed because virtualization support is not enabled in >>>>> the BIOS." >>>>> >>>>> Apparently, Microsoft queries the BIOS to verify that the >>>> >>>> When kvm is initialized, it checks for TXT and VMX both being enabled. >>>> That too, only if the feature control msr is locked. I don't think there >>>> are actually any specific "bios calls" to find this out. I would assume >>>> Hyper-V should be doing the same thing but your testing says otherwise. >>>> >>>> Can you please run linux as L1 with "-hypervisor" and see if it works ? >>>> If it doesn't, please check dmesg for relevant messages. >>>> >>>> Thanks, >>>> Bandan >>>> >>>> >>>>> virtualization bit is actually enabled, instead of simply relying on >>>>> the VMX flag. >>>>> Unfortunately, VMs are still not starting either. The seabios in Qemu >>>>> seems to be pretty difficult to modify. I'll check whether I can >>>>> reinstall on UEFI, maybe that is going to make a difference. >>>>> >>>>> The way VMWare does this is actually semi-documented (it hasn't always >>>>> been in the product, and a workaround involving manually editing the >>>>> configuration has been used for a long time). I'll see if I can >>>>> correlate these to Qemu options, to see whether we can use those >>>>> instructions to get this working on Qemu. >>>>> >>>>> 1. Set 'vhv.enable = "TRUE" on the VM >>>>> It "enables virtual hardware virtualization". This seems equivalent >>>>> to the -hypervisor flag >>>>> >>>>> 2. Set 'monitor.virtual_exec = "hardware" on the VM. >>>>> This option seems to force hardware virtualization for both CPU and >>>>> MMU. Unsure whether there's an equivalent Qemu configuration option. >>>>> Unsure whether it's needed on Qemu. Details at >>>>> http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf >>>>> >>>>> 3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration >>>>> This seems synonymous to the +vmx flag >>>>> >>>>> 4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI" >>>>> I have not found any option to explicitly do this in Qemu. Looking >>>>> at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be >>>>> OK. >>>>> >>>>> 5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- >>>>> --H- ---- >>>>> Not sure how to do that in Qemu or what it does. Looking at >>>>> https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE >>>>> instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem >>>>> to make any difference whatsoever. >>>>> >>>>> Sincerely, >>>>> Roel Brook >>>>> >>>>> >>>>> >>>>> 2015-10-04 5:07 GMT+02:00 Bandan Das <[email protected]>: >>>>>> ... >>>>>>> Windows 2012 / 2016 technical preview 3 >>>>>>> -------------------------------------------------------- >>>>>>> The installation via the "default" method of Add/Remove Features does >>>>>>> not work. Hyper-V displays the error message "A hypervisor is already >>>>>>> running". >>>>>>> >>>>>>> This check can be skipped by using a different method of installation >>>>>>> (from PowerShell): >>>>>>> Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V >>>>>>> –All -NoRestart >>>>>>> >>>>>>> This results in (again) the server booting up, but being unable to run >>>>>>> any guest VMs. The error message is less clear then that in 2008, just >>>>>>> "The Virtual Machine Management Service failed to start the virtual >>>>>>> machine 'New Virtual Machine' because one of the Hyper-V components is >>>>>>> not running (Virtual machine ID >>>>>>> 0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)." >>>>>>> is what I could find. >>>>>>> >>>>>>> Other >>>>>>> -------- >>>>>>> Just to verify that "nesting" is actually working, I've also installed >>>>>>> a Ubuntu 15.10 VM and installed Qemu on it. >>>>>>> This combination CAN successfully run a VM. >>>>>>> >>>>>>> I've also installed VirtualBox on one of the Windows VMs. This >>>>>>> VirtualBox instance is also capable of running virtual machines. >>>>>>> According to the icon in the bottom right, VirtualBox IS using the >>>>>>> hardware virtualization. >>>>>>> >>>>>>> Is this a problem specific to Hyper-V? Is there a method to get >>>>>> >>>>>> Nesting a Hyper-V L1 hypervisor is largely untested. But one of the >>>>>> problems I recollect is that Hyper-V doesn’t like running in a >>>>>> virtualized environment. It checks the “hypervisor” feature flag that >>>>>> Qemu exports. You could try running qemu with “-cpu host,-hypervisor” >>>>>> or something similar and see if it makes any difference. I suspect there >>>>>> would be other roadblocks though, this is just one of them. >>>>>> >>>>>> >>>>>> >>>>>>> Hyper-V working including running guests? I know for a fact that >>>>>>> VMWare Workstation / ESX is able to run Hyper-V fully, so it should >>>>>> >>>>>> Yes, IIRC one of the things ESX does is hide the hypervisor flag >>>>>> specifically for Hyper-V. >>>>>> >>>>>> Bandan >>>>>> >>>>>>> not be completely impossible (but I dislike VMWare for different >>>>>>> reasons). >>>>>>> >>>>>>> My Qemu command line (generated by virt-manager). Except for disks and >>>>>>> domain names, all are identical: >>>>>>> >>>>>>> qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine >>>>>>> pc-i440fx-vivid,accel=kvm,usb=off -cpu >>>>>>> SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme >>>>>>> -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid >>>>>>> 54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults >>>>>>> -chardev >>>>>>> socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait >>>>>>> -mon chardev=charmonitor,id=monitor,mode=control -rtc >>>>>>> base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard >>>>>>> -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global >>>>>>> PIIX4_PM.disable_s4=1 -boot strict=on -device >>>>>>> ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device >>>>>>> ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 >>>>>>> -device >>>>>>> ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 >>>>>>> -device >>>>>>> ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 >>>>>>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive >>>>>>> file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads >>>>>>> -device >>>>>>> ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 >>>>>>> -drive >>>>>>> file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw >>>>>>> -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 >>>>>>> -netdev tap,fd=24,id=hostnet0 -device >>>>>>> rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3 >>>>>>> -chardev pty,id=charserial0 -device >>>>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>>>> spicevmc,id=charchannel0,name=vdagent -device >>>>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 >>>>>>> -device usb-tablet,id=input0 -spice >>>>>>> port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on >>>>>>> -device >>>>>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 >>>>>>> -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device >>>>>>> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev >>>>>>> spicevmc,id=charredir0,name=usbredir -device >>>>>>> usb-redir,chardev=charredir0,id=redir0 -chardev >>>>>>> spicevmc,id=charredir1,name=usbredir -device >>>>>>> usb-redir,chardev=charredir1,id=redir1 -chardev >>>>>>> spicevmc,id=charredir2,name=usbredir -device >>>>>>> usb-redir,chardev=charredir2,id=redir2 -chardev >>>>>>> spicevmc,id=charredir3,name=usbredir -device >>>>>>> usb-redir,chardev=charredir3,id=redir3 -device >>>>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on >>>>>>> >>>>>>> Thank you in advance for response. >>>>>>> >>>>>>> Sincerely, >>>>>>> Roel Brook >>>>>>> >>>>>>
