[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
Got similar behavior with windows server 2012r2 VMs. Environment: uname -a Linux ubuntu-q87 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ii linux-image-4.13.0-36-generic 4.13.0-36.40~16.04.1 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP apt policy qemu qemu: Installed: (none) Candidate: 1:2.11+dfsg-1ubuntu5~cloud0 Version table: 1:2.11+dfsg-1ubuntu5~cloud0 500 Windows VMs error out and create a memory dump: The computer has rebooted from a bugcheck. The bugcheck was: 0x0109 (0xa3a01f5891f186c5, 0xb3b72bdee47188a0, 0x0320, 0x0017). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 033018-31234-01. Based on microsoft docs: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0x109---critical-structure-corruption CRITICAL_STRUCTURE_CORRUPTION Parameters Parameter Description 1 Reserved 2 Reserved 3 Reserved 4 The type of the corrupted region. (See the following table later on this page.) ... 0x17 Local APIC modification <--- this Which is the same as with other reports out there: https://www.spinics.net/lists/kvm/msg159977.html https://forum.proxmox.com/threads/new-windows-vm-keeps-dying.39145/#post-193639 >From what I see the change was backported but there was no new build yet. http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/arch/x86/kvm/x86.c?h=hwe&id=78d2542b88d16 See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1738972 I suggest this is marked as a duplicate to 1738972 -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Status in linux package in Ubuntu: Confirmed Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Status in linux package in Ubuntu: Confirmed Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Status in linux package in Ubuntu: Confirmed Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
I just reported the bug in the kernel: https://bugzilla.kernel.org/show_bug.cgi?id=197951 If you reported or commented on the bug here, please go comment on that report confirming as well. A lot of open-source bugzilla projects tend to rarely pay attention to bug reports that only one person has confirmed/reported. ** Bug watch added: Linux Kernel Bug Tracker #197951 https://bugzilla.kernel.org/show_bug.cgi?id=197951 -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
I have yet to try disabling swap, but in the 5 days since I downgraded the kernel to 4.12.12 from 4.12.13, I have not had a single BSOD. I think 4.12.13 is the culprit. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
I am on Arch as well, using a customized kernel using the vfio patchset (in this case 4.13.11). I was having the same issue as you guys, where my Windows 10 VM with an NVIDIA card passed in was getting the CRITICAL_STRUCTURE_CORRUPTION blue screen error message after running for a while. Usually I saw this when hitting some form of memory (GPU or system RAM), and it was quick (~3 hours) to crash while mining on the GPU (as that hits the GPU memory hard). It looks like what Jimi said above about swap seeming to be a contributing factor seems to be correct. I have disabled swap on my host and have seen no instability thus far. Windows 7 also may be seeing similar issues, though it was just crashing though without displaying an error as far as I could see. This VM has an AMD card in it. Same goes for it, where it also has not crashed after more than a day after disabling swap. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
Specifically 4.12.12, because it seems that was the last version I was running before this issue started (I was on 4.12.13 when it started). I, too, can't find any other package upgrade that could have possibly been the culprit. The timing of my upgrades of any qemu or libvirt packages rule them out. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
I'm experiencing this BSOD issue as well. I'm also on Arch x64, and the same versions of everything (though not minimal qemu--just the normal package in the main repos). I also passthru a GPU and a USB card, but not an SSD. It will happen randomly, anytime, at least once a day, and it seems like demanding games make it much more frequent. Since you're on btrfs, I'll see what happens if I downgrade the kernel to 4.12. If that doesn't work, I guess I'll try to confirm the swapoff fix, but my host only gets 4GB of RAM when the VM is running, so no swap would hurt real bad. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
[Qemu-devel] [Bug 1728256] Re: Memory corruption in Windows 10 guest / amd64
I've also had the exact symptoms and issues you've described. I have also noticed that the VM would BSOD with the CRITICAL_STRUCTURE_CORRUPTION message when the host system would read VM memory from swap. After disabling swap on the host system I've completely managed to eliminate this BSOD issue. Hopefully it's also applicable to your system so you can atleast figure out how to move forward. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1728256 Title: Memory corruption in Windows 10 guest / amd64 Status in QEMU: New Bug description: I have a Win 10 Pro x64 guest inside a qemu/kvm running on an Arch x86_64 host. The VM has a physical GPU passed through, as well as the physical USB controllers, as well as a dedicated SSD attached via SATA; you can find the complete libvirt xml here: https://pastebin.com/U1ZAXBNg I built qemu from source using the qemu-minimal-git AUR package; you can find the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=qemu-minimal-git (if you aren't familiar with Arch, this is essentially a bash script where build() and package() are run to build the files, and then install them into the $pkgdir to later tar them up.) Starting with qemu v2.10.0, Windows crashes randomly with a bluescreen about CRITICAL_STRUCTURE_CORRUPTION. I also tested the git heads f90ea7ba7c, 861cd431c9 and e822e81e35, before I went back to v2.9.0, which is running stable for over 50 hours right now. During my tests I found that locking the memory pages alleviates the problem somewhat, but never completely avoids it. However, with the crashes occuring randomly, that could as well be false conclusions; I had crashes within minutes after boot with that too. I will now start `git bisect`ing; if you have any other suggestions on what I could try or possible patches feel free to leave them with me. To manage notifications about this bug go to: https://bugs.launchpad.net/qemu/+bug/1728256/+subscriptions
