Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 24, 2015, at 9:50 AM, Paolo Bonzini pbonz...@redhat.com wrote: On 23/06/2015 00:08, John Nielsen wrote: I’m resurrecting an old thread since I haven’t heard anything in a while. Has anyone looked in to the KVM+apicv bug documented above as well as here: https://bugs.launchpad.net/qemu/+bug/1329956 ? If appropriate, where should I go to file a KVM bug (since this isn’t really Qemu’s problem)? Hi John, does this happen with the latest upstream kernel version ? I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.-- To unsubscribe from this list: send the line unsubscribe kvm in Can you reproduce it with 10.1? I did this: 1) download http://download.pcbsd.org/iso/10.1-RELEASE/amd64/PCBSD10.1.2-x64-trueos-server.raw.xz and unpack it 2) run it with qemu-kvm -drive if=virtio,PCBSD10.1.2-x64-trueos-server.raw -smp 2 3) login as root/pcbsd, type reboot I would like to know if I'm doing anything wrong. My machine is a Xeon E5 v3 (Haswell). My SeaBIOS build doesn't have the atkbd0 bug, but just to rule that out, can you send me your Seabios binary (/usr/share/qemu/bios*.bin) as well? Interesting. Using the same PC-BSD image I am able to reproduce on a server running slightly older software but I can not reproduce running bleeding edge. I verified enable_apicv=Y on both. In both cases I ran qemu-kvm -drive if=virtio,file=PCBSD10.1.2-x64-trueos-server.raw -smp 2 -vnc 0.0.0.0:0 Specifically: Breaks (VM hangs during boot after pressing ctrl-alt-del): kernel 3.12.22 qemu-kvm-1.7.0-3.el6.x86_64 seabios-1.7.3.1-1.el6.noarch Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz Works (VM reboots normally): kernel 4.0.4 qemu-kvm-2.3.0-6.el7.centos.x86_64 seabios-bin-1.8.1-1.el7.centos.noarch Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz Unfortunately I no longer have the test environment I used a few days ago to reproduce this issue so I can’t verify the software versions that were in use. It’s possible I was mistaken about the kernel version (I thought it was 4.0.4). Perhaps it really is fixed in the newer kernel? In any case, this is great news! I would be interested in identifying the patch(es) that fixed the issue to make back-porting them easier, but I won’t have time to do a binary search anytime soon. Thanks for looking in to this again. If anyone else is interested in identifying what specifically fixed the issue please let me know if I can do anything to help. JN -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 22, 2015, at 3:48 PM, Bandan Das b...@redhat.com wrote: John Nielsen li...@jnielsen.net writes: On Jun 17, 2014, at 10:48 AM, John Nielsen li...@jnielsen.net wrote: On Jun 17, 2014, at 12:05 AM, Gleb Natapov g...@kernel.org wrote: On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote: Il 16/06/2014 18:47, John Nielsen ha scritto: On Jun 16, 2014, at 10:39 AM, Paolo Bonzini pbonz...@redhat.com wrote: Il 16/06/2014 18:09, John Nielsen ha scritto: The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, while the hosts that don't show the problem use the prior revision, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz. Can you do grep . /sys/module/kvm_intel/parameters/* on both hosts please? No differences that I can see. Output below. Not really: Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:N Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:Y So we have a clue. Let me study the code more, I'll try to get back with a suggestion. Wow, can't believe I missed that. Good catch! Does disabling apicv on E5-2650 v2 make reboot problem go away? Yes it does! # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem occurs] # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N /sys/module/kvm_intel/parameters/enable_apicv:N # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem does not occur] Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list. I’m resurrecting an old thread since I haven’t heard anything in a while. Has anyone looked in to the KVM+apicv bug documented above as well as here: https://bugs.launchpad.net/qemu/+bug/1329956 ? If appropriate, where should I go to file a KVM bug (since this isn’t really Qemu’s problem)? Hi John, does this happen with the latest upstream kernel version ? I know for sure it happens with 4.0.4 and I’m not aware of any newer changes that would affect it.-- To unsubscribe from this list: send the line unsubscribe kvm in
Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 17, 2014, at 10:48 AM, John Nielsen li...@jnielsen.net wrote: On Jun 17, 2014, at 12:05 AM, Gleb Natapov g...@kernel.org wrote: On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote: Il 16/06/2014 18:47, John Nielsen ha scritto: On Jun 16, 2014, at 10:39 AM, Paolo Bonzini pbonz...@redhat.com wrote: Il 16/06/2014 18:09, John Nielsen ha scritto: The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, while the hosts that don't show the problem use the prior revision, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz. Can you do grep . /sys/module/kvm_intel/parameters/* on both hosts please? No differences that I can see. Output below. Not really: Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:N Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:Y So we have a clue. Let me study the code more, I'll try to get back with a suggestion. Wow, can't believe I missed that. Good catch! Does disabling apicv on E5-2650 v2 make reboot problem go away? Yes it does! # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem occurs] # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N /sys/module/kvm_intel/parameters/enable_apicv:N # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem does not occur] Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list. I’m resurrecting an old thread since I haven’t heard anything in a while. Has anyone looked in to the KVM+apicv bug documented above as well as here: https://bugs.launchpad.net/qemu/+bug/1329956 ? If appropriate, where should I go to file a KVM bug (since this isn’t really Qemu’s problem)? Thanks, JN-- To unsubscribe from this list: send the line unsubscribe kvm in
Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 16, 2014, at 10:21 PM, Paolo Bonzini pbonz...@redhat.com wrote: Il 16/06/2014 18:47, John Nielsen ha scritto: On Jun 16, 2014, at 10:39 AM, Paolo Bonzini pbonz...@redhat.com wrote: Il 16/06/2014 18:09, John Nielsen ha scritto: The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, while the hosts that don't show the problem use the prior revision, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz. Can you do grep . /sys/module/kvm_intel/parameters/* on both hosts please? No differences that I can see. Output below. Not really: Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:N Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:Y So we have a clue. Let me study the code more, I'll try to get back with a suggestion. Paolo, have you had an opportunity to look in to this some more? Thanks, JN -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 20, 2014, at 1:53 PM, Paolo Bonzini pbonz...@redhat.com wrote: Il 20/06/2014 17:41, John Nielsen ha scritto: So we have a clue. Let me study the code more, I'll try to get back with a suggestion. Paolo, have you had an opportunity to look in to this some more? Not yet, sorry. One possibility is this though. Can you try migrating (or saving/restoring) the guest when it's hung, and see if it resuscitates? The guest is still hung after a save/restore. # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio QEMU 2.0.50 monitor - type 'help' for more information (qemu) stop (qemu) savevm smphang (qemu) q # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none -monitor stdio -loadvm smphang QEMU 2.0.50 monitor - type 'help' for more information (qemu) [The VNC console shows the same hung kernel screen as when I ran savevm] JN -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Hang on reboot in FreeBSD guest on Linux KVM host
On Jun 17, 2014, at 12:05 AM, Gleb Natapov g...@kernel.org wrote: On Tue, Jun 17, 2014 at 06:21:23AM +0200, Paolo Bonzini wrote: Il 16/06/2014 18:47, John Nielsen ha scritto: On Jun 16, 2014, at 10:39 AM, Paolo Bonzini pbonz...@redhat.com wrote: Il 16/06/2014 18:09, John Nielsen ha scritto: The only substantial difference on the hardware side is the CPU. The hosts where the problem occurs use Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz, while the hosts that don't show the problem use the prior revision, Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz. Can you do grep . /sys/module/kvm_intel/parameters/* on both hosts please? No differences that I can see. Output below. Not really: Working host: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:N Problem host: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz # grep . /sys/module/kvm_intel/parameters/* /sys/module/kvm_intel/parameters/enable_apicv:Y So we have a clue. Let me study the code more, I'll try to get back with a suggestion. Wow, can't believe I missed that. Good catch! Does disabling apicv on E5-2650 v2 make reboot problem go away? Yes it does! # modprobe kvm_intel /sys/module/kvm_intel/parameters/enable_apicv:Y # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem occurs] # rmmod kvm_intel # modprobe kvm_intel enable_apicv=N /sys/module/kvm_intel/parameters/enable_apicv:N # /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none [problem does not occur] Thank you. This both narrows the problem considerably and provides an acceptable workaround. It would still be nice to see it fixed, of course. Keep me CC'ed as I'm not on the KVM list. JN -- To unsubscribe from this list: send the line unsubscribe kvm in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
