Hi, all I am trying to use nested virtualization setup and experiencing L2 guest (Windows) hanging during boot. I have seen some threads about it, but did not manage to make sense of how can I work around it.
Would appreciate any guidance for a solution or workaround. Thanks, Ariel Some details are attached. *L0 logs look like this:* [117402.803802] kvm [11772]: vcpu0 unhandled rdmsr: 0x345 [117402.823629] kvm_set_msr_common: 54 callbacks suppressed [117402.823631] kvm [11772]: vcpu0 unhandled wrmsr: 0x680 data 0 [117402.833796] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c0 data 0 [117402.843967] kvm [11772]: vcpu0 unhandled wrmsr: 0x681 data 0 [117402.854106] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c1 data 0 [117402.864272] kvm [11772]: vcpu0 unhandled wrmsr: 0x682 data 0 [117402.874429] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c2 data 0 [117402.884613] kvm [11772]: vcpu0 unhandled wrmsr: 0x683 data 0 [117402.893675] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c3 data 0 [117402.902550] kvm [11772]: vcpu0 unhandled wrmsr: 0x684 data 0 [117402.911442] kvm [11772]: vcpu0 unhandled wrmsr: 0x6c4 data 0 At the time of the hanging this line is added: nested_vmx_run: VMCS MSR_{LOAD,STORE} unsupported *L1 logs look like this:* Jan 9 11:34:43 hpvm---2 kernel: [ 217.975781] kvm [2106]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff *The setup details are as follows:* ------------------------------------------------ *L0 - Ubuntu 14.04, kernel 3.13.0-43:* *CPUID* Vendor ID: "GenuineIntel"; CPUID level 13 Intel-specific functions: Version 000206d7: Type 0 - Original OEM Family 6 - Pentium Pro Model 13 - Stepping 7 Reserved 8 Extended brand string: " Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz" CLFLUSH instruction cache line size: 8 Initial APIC ID: 39 Hyper threading siblings: 32 Feature flags bfebfbff: FPU Floating Point Unit VME Virtual 8086 Mode Enhancements DE Debugging Extensions PSE Page Size Extensions TSC Time Stamp Counter MSR Model Specific Registers PAE Physical Address Extension MCE Machine Check Exception CX8 COMPXCHG8B Instruction APIC On-chip Advanced Programmable Interrupt Controller present and enabled SEP Fast System Call MTRR Memory Type Range Registers PGE PTE Global Flag MCA Machine Check Architecture CMOV Conditional Move and Compare Instructions FGPAT Page Attribute Table PSE-36 36-bit Page Size Extension CLFSH CFLUSH instruction DS Debug store ACPI Thermal Monitor and Clock Ctrl MMX MMX instruction set FXSR Fast FP/MMX Streaming SIMD Extensions save/restore SSE Streaming SIMD Extensions instruction set SSE2 SSE2 extensions SS Self Snoop HT Hyper Threading TM Thermal monitor 31 reserved TLB and cache info: 5a: unknown TLB/cache descriptor 03: Data TLB: 4KB pages, 4-way set assoc, 64 entries 76: unknown TLB/cache descriptor ff: unknown TLB/cache descriptor b2: unknown TLB/cache descriptor f0: unknown TLB/cache descriptor ca: unknown TLB/cache descriptor Processor serial: 0002-06D7-0000-0000-0000-0000 *L1 - Ubuntu 14.04, kernel 3.13.0-43* *CPUID* CPU 0: vendor_id = "GenuineIntel" version information (1/eax): processor type = primary processor (0) family = Intel Pentium Pro/II/III/Celeron/Core/Core 2/Atom, AMD Athlon/Duron, Cyrix M2, VIA C3 (6) model = 0xa (10) stepping id = 0x1 (1) extended family = 0x0 (0) extended model = 0x2 (2) (simple synth) = Intel Core i5-2000 / Core i7-2000 / Mobile Core i3-2000 / Mobile Core i5-2000 / Mobile Core i7-2000 / Pentium G500/G600/G800 / Celeron G400/G500/700/800/B800 / Xeon E3-1200 (Sandy Bridge), 32nm miscellaneous (1/ebx): process local APIC physical ID = 0x0 (0) cpu count = 0x0 (0) CLFLUSH line size = 0x8 (8) brand index = 0x0 (0) brand id = 0x00 (0): unknown feature information (1/edx): x87 FPU on chip = true virtual-8086 mode enhancement = true debugging extensions = true page size extensions = true time stamp counter = true RDMSR and WRMSR support = true physical address extensions = true machine check exception = true CMPXCHG8B inst. = true APIC on chip = true SYSENTER and SYSEXIT = true memory type range registers = true PTE global bit = true machine check architecture = true conditional move/compare instruction = true page attribute table = true page size extension = true processor serial number = false CLFLUSH instruction = true debug store = false thermal monitor and clock ctrl = false MMX Technology = true FXSAVE/FXRSTOR = true SSE extensions = true SSE2 extensions = true self snoop = true hyper-threading / multi-core supported = false therm. monitor = false IA64 = false pending break event = false feature information (1/ecx): PNI/SSE3: Prescott New Instructions = true PCLMULDQ instruction = true 64-bit debug store = false MONITOR/MWAIT = false CPL-qualified debug store = false VMX: virtual machine extensions = true SMX: safer mode extensions = false Enhanced Intel SpeedStep Technology = false thermal monitor 2 = false SSSE3 extensions = true context ID: adaptive or shared L1 data = false FMA instruction = false CMPXCHG16B instruction = true xTPR disable = false perfmon and debug = false process context identifiers = true direct cache access = false SSE4.1 extensions = true SSE4.2 extensions = true extended xAPIC support = true MOVBE instruction = false POPCNT instruction = true time stamp counter deadline = true AES instruction = true XSAVE/XSTOR states = true OS-enabled XSAVE/XSTOR = true AVX: advanced vector extensions = true F16C half-precision convert instruction = false RDRAND instruction = false hypervisor guest status = true cache and TLB information (2): 0x7d: L2 cache: 2M, 8-way, sectored, 64 byte lines 0x30: L1 cache: 32K, 8-way, 64 byte lines 0x2c: L1 data cache: 32K, 8-way, 64 byte lines processor serial number: 0002-06A1-0000-0000-0000-0000 deterministic cache parameters (4): --- cache 0 --- cache type = data cache (1) cache level = 0x1 (1) self-initializing cache level = true fully associative cache = false extra threads sharing this cache = 0x0 (0) extra processor cores on this die = 0x0 (0) system coherency line size = 0x3f (63) physical line partitions = 0x0 (0) ways of associativity = 0x7 (7) WBINVD/INVD behavior on lower caches = true inclusive to lower caches = false complex cache indexing = false number of sets - 1 (s) = 63 --- cache 1 --- cache type = instruction cache (2) cache level = 0x1 (1) self-initializing cache level = true fully associative cache = false extra threads sharing this cache = 0x0 (0) extra processor cores on this die = 0x0 (0) system coherency line size = 0x3f (63) physical line partitions = 0x0 (0) ways of associativity = 0x7 (7) WBINVD/INVD behavior on lower caches = true inclusive to lower caches = false complex cache indexing = false number of sets - 1 (s) = 63 --- cache 2 --- cache type = unified cache (3) cache level = 0x2 (2) self-initializing cache level = true fully associative cache = false extra threads sharing this cache = 0x0 (0) extra processor cores on this die = 0x0 (0) system coherency line size = 0x3f (63) physical line partitions = 0x0 (0) ways of associativity = 0xf (15) WBINVD/INVD behavior on lower caches = true inclusive to lower caches = false complex cache indexing = false number of sets - 1 (s) = 4095 MONITOR/MWAIT (5): smallest monitor-line size (bytes) = 0x0 (0) largest monitor-line size (bytes) = 0x0 (0) enum of Monitor-MWAIT exts supported = true supports intrs as break-event for MWAIT = true number of C0 sub C-states using MWAIT = 0x0 (0) number of C1 sub C-states using MWAIT = 0x0 (0) number of C2 sub C-states using MWAIT = 0x0 (0) number of C3 sub C-states using MWAIT = 0x0 (0) number of C4 sub C-states using MWAIT = 0x0 (0) number of C5 sub C-states using MWAIT = 0x0 (0) number of C6 sub C-states using MWAIT = 0x0 (0) number of C7 sub C-states using MWAIT = 0x0 (0) Thermal and Power Management Features (6): digital thermometer = false Intel Turbo Boost Technology = false ARAT always running APIC timer = false PLN power limit notification = false ECMD extended clock modulation duty = false PTM package thermal management = false digital thermometer thresholds = 0x0 (0) ACNT/MCNT supported performance measure = false ACNT2 available = false performance-energy bias capability = false extended feature flags (7): FSGSBASE instructions = false IA32_TSC_ADJUST MSR supported = false BMI instruction = false HLE hardware lock elision = false AVX2: advanced vector extensions 2 = false SMEP supervisor mode exec protection = false BMI2 instructions = false enhanced REP MOVSB/STOSB = false INVPCID instruction = false RTM: restricted transactional memory = false QM: quality of service monitoring = false deprecated FPU CS/DS = false intel memory protection extensions = false AVX512F: AVX-512 foundation instructions = false RDSEED instruction = false ADX instructions = false SMAP: supervisor mode access prevention = false Intel processor trace = false AVX512PF: prefetch instructions = false AVX512ER: exponent & reciprocal instrs = false AVX512CD: conflict detection instrs = false SHA instructions = false PREFETCHWT1 = false Direct Cache Access Parameters (9): PLATFORM_DCA_CAP MSR bits = 0 Architecture Performance Monitoring Features (0xa/eax): version ID = 0x2 (2) number of counters per logical processor = 0x4 (4) bit width of counter = 0x30 (48) length of EBX bit vector = 0x7 (7) Architecture Performance Monitoring Features (0xa/ebx): core cycle event not available = false instruction retired event not available = false reference cycles event not available = false last-level cache ref event not available = false last-level cache miss event not avail = false branch inst retired event not available = false branch mispred retired event not avail = false Architecture Performance Monitoring Features (0xa/edx): number of fixed counters = 0x3 (3) bit width of fixed counters = 0x30 (48) XSAVE features (0xd/0): XCR0 lower 32 bits valid bit field mask = 0x00000007 bytes required by fields in XCR0 = 0x00000340 (832) bytes required by XSAVE/XRSTOR area = 0x00000340 (832) XCR0 upper 32 bits valid bit field mask = 0x00000000 YMM features (0xd/2): YMM save state byte size = 0x00000100 (256) YMM save state byte offset = 0x00000240 (576) LWP features (0xd/0x3e): LWP save state byte size = 0x00000000 (0) LWP save state byte offset = 0x00000000 (0) hypervisor_id = "KVMKVMKVM " hypervisor features (0x40000001/eax): kvmclock available at MSR 0x11 = true delays unnecessary for PIO ops = true mmu_op = false kvmclock available a MSR 0x4b564d00 = true async pf enable available by MSR = true steal clock supported = true guest EOI optimization enabled = true stable: no guest per-cpu warps expected = true extended feature flags (0x80000001/edx): SYSCALL and SYSRET instructions = true execution disable = true 1-GB large page support = true RDTSCP = true 64-bit extensions technology available = true Intel feature flags (0x80000001/ecx): LAHF/SAHF supported in 64-bit mode = true LZCNT advanced bit manipulation = false 3DNow! PREFETCH/PREFETCHW instructions = false brand = "Intel Xeon E312xx (Sandy Bridge)" L1 TLB/cache information: 2M/4M pages & L1 TLB (0x80000005/eax): instruction # entries = 0xff (255) instruction associativity = 0x1 (1) data # entries = 0xff (255) data associativity = 0x1 (1) L1 TLB/cache information: 4K pages & L1 TLB (0x80000005/ebx): instruction # entries = 0xff (255) instruction associativity = 0x1 (1) data # entries = 0xff (255) data associativity = 0x1 (1) L1 data cache information (0x80000005/ecx): line size (bytes) = 0x40 (64) lines per tag = 0x1 (1) associativity = 0x2 (2) size (Kb) = 0x40 (64) L1 instruction cache information (0x80000005/edx): line size (bytes) = 0x40 (64) lines per tag = 0x1 (1) associativity = 0x2 (2) size (Kb) = 0x40 (64) L2 TLB/cache information: 2M/4M pages & L2 TLB (0x80000006/eax): instruction # entries = 0x0 (0) instruction associativity = L2 off (0) data # entries = 0x0 (0) data associativity = L2 off (0) L2 TLB/cache information: 4K pages & L2 TLB (0x80000006/ebx): instruction # entries = 0x200 (512) instruction associativity = 4-way (4) data # entries = 0x200 (512) data associativity = 4-way (4) L2 unified cache information (0x80000006/ecx): line size (bytes) = 0x40 (64) lines per tag = 0x1 (1) associativity = 16-way (8) size (Kb) = 0x200 (512) L3 cache information (0x80000006/edx): line size (bytes) = 0x0 (0) lines per tag = 0x0 (0) associativity = L2 off (0) size (in 512Kb units) = 0x0 (0) Advanced Power Management Features (0x80000007/edx): temperature sensing diode = false frequency ID (FID) control = false voltage ID (VID) control = false thermal trip (TTP) = false thermal monitor (TM) = false software thermal control (STC) = false 100 MHz multiplier control = false hardware P-State control = false TscInvariant = false Physical Address and Linear Address Size (0x80000008/eax): maximum physical address bits = 0x28 (40) maximum linear (virtual) address bits = 0x30 (48) maximum guest physical address bits = 0x0 (0) Logical CPU cores (0x80000008/ecx): number of CPU cores - 1 = 0x0 (0) ApicIdCoreIdSize = 0x0 (0) SVM Secure Virtual Machine (0x8000000a/eax): SvmRev: SVM revision = 0x0 (0) SVM Secure Virtual Machine (0x8000000a/edx): nested paging = false LBR virtualization = false SVM lock = false NRIP save = false MSR based TSC rate control = false VMCB clean bits support = false flush by ASID = false decode assists = false SSSE3/SSE5 opcode set disable = false pause intercept filter = false pause filter threshold = false NASID: number of address space identifiers = 0x0 (0): (multi-processing synth): none (multi-processing method): Intel leaf 1/4 (APIC widths synth): CORE_width=0 SMT_width=16 (APIC synth): PKG_ID=8 CORE_ID=8 SMT_ID=0 (synth) = Intel Xeon E3-1200 (Sandy Bridge), 32nm L2 - Windows 2003 (but experience same hanging, although not in boot time, in L2 - Linux guest) *The execution command for L1 is:* qemu-system-x86_64 -machine accel=kvm:tcg -name instance-00000001 -S -machine pc-i440fx-1.5,accel=kvm,usb=off -cpu SandyBridge,+pdpe1gb,+osxsave, +dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+ monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 16384 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid 897abedb-a2c2-4140-b994-1b4ee303a54b -smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.2.3,serial=44454c4c-4800-1030-8057- b3c04f395931,uuid=897abedb-a2c2-4140-b994-1b4ee303a54b -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/ var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/instances/897abedb-a2c2-4140-b994- 1b4ee303a54b/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev= hostnet0,id=net0,mac=fa:16:3e:30:0e:85,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/897abedb- a2c2-4140-b994-1b4ee303a54b/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 *The execution command for L2 is:* /usr/bin/kvm -name SP-Containers -S -machine pc-i440fx-2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 431ce7cc-173e-3bd0-988d-94f504b7ca4c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/SP- Containers.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/hdd/linux_dsm.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=22,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:33:50:56,bus=pci.0,addr=0x6 -netdev tap,fd=23,id=hostnet1 -device rtl8139,netdev=hostnet1,id= net1,mac=52:54:00:35:34:50,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x3 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on