[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
This bug was fixed in the package linux - 5.4.0-66.74 --- linux (5.4.0-66.74) focal; urgency=medium * focal/linux: 5.4.0-66.74 -proposed tracker (LP: #1913152) * Add support for selective build of special drivers (LP: #1912789) - [Packaging] Add support for ODM drivers - [Packaging] Turn on ODM support for amd64 * Packaging resync (LP: #1786013) - update dkms package versions - update dkms package versions * Introduce the new NVIDIA 460-server series and update the 460 series (LP: #1913200) - [Config] dkms-versions -- drop NVIDIA 435 455 and 440-server - [Config] dkms-versions -- add the 460-server nvidia driver * Enable mute and micmute LED on HP EliteBook 850 G7 (LP: #1910102) - ALSA: hda/realtek: Enable mute and micmute LED on HP EliteBook 850 G7 * SYNA30B4:00 06CB:CE09 Mouse on HP EliteBook 850 G7 not working at all (LP: #1908992) - HID: multitouch: Enable multi-input for Synaptics pointstick/touchpad device * HD Audio Device PCI ID for the Intel Cometlake-R platform (LP: #1912427) - SAUCE: ALSA: hda: Add Cometlake-R PCI ID * switch to an autogenerated nvidia series based core via dkms-versions (LP: #1912803) - [Packaging] nvidia -- use dkms-versions to define versions built - [Packaging] update-version-dkms -- maintain flags fields - [Config] dkms-versions -- add transitional/skip information for nvidia packages * udpgro.sh in net from ubuntu_kernel_selftests seems not reflecting sub-test result (LP: #1908499) - selftests: fix the return value for UDP GRO test * qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP tx csum offload (LP: #1909062) - qede: fix offload for IPIP tunnel packets * Use DCPD to control HP DreamColor panel (LP: #1911001) - SAUCE: drm/dp: Another HP DreamColor panel brigntness fix * kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts (LP: #1911848) - KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set * Ubuntu 20.10 four needed fixes to 'Add driver for Mellanox Connect-IB adapters' (LP: #1905574) - net/mlx5: Fix a race when moving command interface to polling mode * Fix right sounds and mute/micmute LEDs for HP ZBook Fury 15/17 G7 Mobile Workstation (LP: #1910561) - ALSA: hda/realtek: fix right sounds and mute/micmute LEDs for HP machines * Ubuntu 20.04 - multicast counter is not increased in ip -s (LP: #1901842) - net/mlx5e: Fix multicast counter not up-to-date in "ip -s" * eeh-basic.sh in powerpc from ubuntu_kernel_selftests timeout with 5.4 P8 / P9 (LP: #1882503) - selftests/powerpc/eeh: disable kselftest timeout setting for eeh-basic * DMI entry syntax fix for Pegatron / ByteSpeed C15B (LP: #1910639) - Input: i8042 - unbreak Pegatron C15B * CVE-2020-29372 - mm: check that mm is still valid in madvise() * update ENA driver, incl. new ethtool stats (LP: #1910291) - net: ena: Change WARN_ON expression in ena_del_napi_in_range() - net: ena: ethtool: convert stat_offset to 64 bit resolution - net: ena: ethtool: Add new device statistics - net: ena: ethtool: add stats printing to XDP queues - net: ena: xdp: add queue counters for xdp actions - net: ena: Change license into format to SPDX in all files - net: ena: Change log message to netif/dev function - net: ena: Capitalize all log strings and improve code readability - net: ena: Remove redundant print of placement policy - net: ena: Change RSS related macros and variables names - net: ena: Fix all static chekers' warnings - drivers/net/ethernet: remove incorrectly formatted doc - net: ena: handle bad request id in ena_netdev - net: ena: fix packet's addresses for rx_offset feature * s390x broken with unknown syscall number on kernels < 5.8 (LP: #1895132) - s390/ptrace: return -ENOSYS when invalid syscall is supplied * Focal update: v5.4.86 upstream stable release (LP: #1910822) - ARM: dts: sun7i: bananapi: Enable RGMII RX/TX delay on Ethernet PHY - ARM: dts: sun8i: r40: bananapi-m2-berry: Fix dcdc1 regulator - ARM: dts: sun8i: v40: bananapi-m2-berry: Fix ethernet node - pinctrl: merrifield: Set default bias in case no particular value given - pinctrl: baytrail: Avoid clearing debounce value when turning it off - ARM: dts: sun8i: v3s: fix GIC node memory range - ARM: dts: sun7i: pcduino3-nano: enable RGMII RX/TX delay on PHY - ARM: dts: imx6qdl-wandboard-revd1: Remove PAD_GPIO_6 from enetgrp - ARM: dts: imx6qdl-kontron-samx6i: fix I2C_PM scl pin - PM: runtime: Add pm_runtime_resume_and_get to deal with usage counter - gpio: zynq: fix reference leak in zynq_gpio functions - gpio: mvebu: fix potential user-after-free on probe - scsi: bnx2i: Requires MMU - xsk: Fix xsk_poll()'s return type - xsk: Replace
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
Attached is reproducing the problem with 5.4.0-66-generic. Note that CPU is low, and system boots to desktop correctly. ** Attachment added: "5.4.0-66-generic" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1911848/+attachment/5462097/+files/5.4.0-66-generic.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance tab. The CPU will be stuck at 100%. The windows guest will be non responsive. A patched kernel is available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test If you install this kernel and boot the Windows 2k19 guest, it will come up normally when the hyper-v role is enabled, and you will be able to log in. [Where problems could occur] This is a change to a core part of the kvm subsystem, so there is potential for regression which could affect all users of KVM. If a regression were to occur, there are no workarounds. Users would need to downgrade their kernel while a fix is developed. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1911848/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
Performing verification for Focal. I installed 5.4.0-65-generic to a c5.metal AWS machine, and installed ubuntu-desktop and a KVM stack. I install xrdp and verified I could reach the VMs desktop. I installed virt-manager, and configured the Windows 2k19 VM in accordance with the testcase section, carefully making sure the Cascadelake-Server-noTSX processor is selected and the features in place. I installed Windows, and installed the virtio drivers. From there I enabled Hyper-V and restarted. The machine got stuck during reboot, after the hyper-v feature had installed and configured. Looking at the performance tab, the machine is stuck at 100% CPU, see 5.4.0-65-generic.png attached. I then enabled -proposed and installed 5.4.0-66-generic, and kexec rebooted. From there I started the windows VM, which came up properly to the desktop. Looking at the performance tab, CPU usage is low, and is what is expected. See 5.4.0-66-generic.png attached. The 5.4.0-66-generic kernel fixes the issue, happy to mark as verified. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
Attached is reproducing the problem with 5.4.0-65-generic, note, see how CPU reaches 100% and is stuck. ** Attachment added: "5.4.0-65-generic" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1911848/+attachment/5462096/+files/5.4.0-65-generic.png -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance tab. The CPU will be stuck at 100%. The windows guest will be non responsive. A patched kernel is available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test If you install this kernel and boot the Windows 2k19 guest, it will come up normally when the hyper-v role is enabled, and you will be able to log in. [Where problems could occur] This is a change to a core part of the kvm subsystem, so there is potential for regression which could affect all users of KVM. If a regression were to occur, there are no workarounds. Users would need to downgrade their kernel while a fix is developed. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1911848/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance tab. The CPU will be stuck at 100%. The windows guest will be non responsive. A patched kernel is available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test If you install this kernel and boot the Windows 2k19 guest, it will come up normally when the hyper-v role is enabled, and you will be able to log in. [Where problems could occur] This is a change to a core part of the kvm subsystem, so there is potential for
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
** Changed in: linux (Ubuntu Focal) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake-Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance tab. The CPU will be stuck at 100%. The windows guest will be non responsive. A patched kernel is available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test If you install this kernel and boot the Windows 2k19 guest, it will come up normally when the hyper-v role is enabled, and you will be able to log in. [Where problems could occur] This is a change to a core part of the kvm subsystem, so there is potential for regression which could affect all users of KVM. If a regression were to occur, there are no workarounds. Users would need to downgrade their kernel while a fix is developed. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1911848/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1911848] Re: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts
** Description changed: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before the graphical login screen appears, on Windows Server systems with the desktop environment installed. If you look at performance metrics for the guest, the CPU will appear to be stuck at 100%, and it never changes from 100%. The Windows Server guest is unresponsive. The KVM settings use Cascadelake-Server-noTSX virtual CPUs, with some very specific settings needed for nested virtualisation. See testcase section. If you use any other vcpu type, the problem does not reproduce. Known workarounds are to install the 5.8 HWE kernel, in which case the server will come up as expected. [Fix] The following commit fixes the issue, and landed in mainline 5.8-rc1: commit 8081ad06b68a728e676d3b08e9ab70ce4039747b Author: Sean Christopherson Date: Wed Apr 22 19:25:40 2020 -0700 Subject: KVM: x86: Set KVM_REQ_EVENT if run is canceled with req_immediate_exit set Link: https://github.com/torvalds/linux/commit/8081ad06b68a728e676d3b08e9ab70ce4039747b It appears that pending requests to the hypervisor can be lost or delayed if an immediate exit was requested in vcpu_enter_guest(). As the commit message mentions, only the !injected case is affected, so we add a check at the cancel_injection label to see if we got there as a result of an immediate exit, and then re-issue a KVM_REQ_EVENT request if we are. The Windows guest is waiting for an event to be processed, which never happens, and so gets stuck. Even though the above commit has a Fixes: tag to a commit in 3.15-rc1, in my testing the 4.15 kernel with a Bionic-ussuri userspace does not reproduce the issue, so SRU to Bionic will not be needed. [Testcase] A cascadelake based Xeon server is required. Anything else and the bug will not reproduce. I used a c5.metal server on AWS. It has the following processor: Intel(R) Xeon(R) Platinum 8275CL CPU @ 3.00GHz Install a KVM stack, and ubuntu-desktop. Set up xrdp and confirm you can reach the desktop. Copy a Windows Server 2k19 image to the destination server, as well as a recent ISO image of virtio drivers. Install virt-manager. Create a new virtual machine using the Windows 2k19 defaults. Use 8 vcpus, 16gb ram. Click customise button to change settings before install. - Change the hard disk to be SATA, attach a new cd rom driver for the + Change the hard disk to be SATA, attach a new cd rom drive for the virtio drivers. Change networking to virtio. Change CPU to Cascadelake- Server-noTSX. Edit the virsh xml, and ensure you set the following features for CPU: Cascadelake-Server-noTSX Those settings are an absolute must. Boot the VM, and install Windows 2k19 with the desktop environment. Once it is installed, open up computer management > device manager and install drivers from the virtio ISO for missing hardware, likely the network and balloon devices. From there, go to server manager, and install the hyper-v role. Reboot the server. It will reboot a few times, and on the final time, it will lock up before it reaches the log in screen. In virt-manager, go to the performance tab. The CPU will be stuck at 100%. The windows guest will be non responsive. A patched kernel is available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf296306-test If you install this kernel and boot the Windows 2k19 guest, it will come up normally when the hyper-v role is enabled, and you will be able to log in. [Where problems could occur] This is a change to a core part of the kvm subsystem, so there is potential for regression which could affect all users of KVM. If a regression were to occur, there are no workarounds. Users would need to downgrade their kernel while a fix is developed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1911848 Title: kvm: Windows 2k19 with Hyper-v role gets stuck on pending hypervisor requests on cascadelake based kvm hosts Status in linux package in Ubuntu: Fix Released Status in linux source package in Focal: In Progress Bug description: BugLink: https://bugs.launchpad.net/bugs/1911848 [Impact] On CascadeLake based KVM hosts, Windows Server 2k16 and 2k19 guests will fail to start once they have enabled the hyper-v role for nested virtualisation. The Windows Server guests will get stuck in the late stages of boot, before
