[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu Wily) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu Wily) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu Yakkety) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu Yakkety) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu Xenial) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu Xenial) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu Vivid) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu Vivid) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu Trusty) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu Trusty) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Expired Status in linux source package in Trusty: Expired Status in linux source package in Vivid: Expired Status in linux source package in Wily: Expired Status in linux source package in Xenial: Expired Status in linux source package in Yakkety: Expired Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
** Changed in: linux (Ubuntu) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu Trusty) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu Vivid) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu Wily) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu Xenial) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu Yakkety) Status: In Progress => Incomplete ** Changed in: linux (Ubuntu) Assignee: Rafael David Tinoco (inaddy) => (unassigned) ** Changed in: linux (Ubuntu Vivid) Assignee: Rafael David Tinoco (inaddy) => (unassigned) ** Changed in: linux (Ubuntu Trusty) Assignee: Rafael David Tinoco (inaddy) => (unassigned) ** Changed in: linux (Ubuntu Wily) Assignee: Rafael David Tinoco (inaddy) => (unassigned) ** Changed in: linux (Ubuntu Yakkety) Assignee: Rafael David Tinoco (inaddy) => (unassigned) ** Changed in: linux (Ubuntu Xenial) Assignee: Rafael David Tinoco (inaddy) => (unassigned) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: Incomplete Status in linux source package in Trusty: Incomplete Status in linux source package in Vivid: Incomplete Status in linux source package in Wily: Incomplete Status in linux source package in Xenial: Incomplete Status in linux source package in Yakkety: Incomplete Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX:
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
** Also affects: linux (Ubuntu Trusty) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Vivid) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Wily) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Yakkety) Importance: High Assignee: Rafael David Tinoco (inaddy) Status: In Progress ** Also affects: linux (Ubuntu Xenial) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Xenial) Status: New => In Progress ** Changed in: linux (Ubuntu Wily) Status: New => In Progress ** Changed in: linux (Ubuntu Vivid) Status: New => In Progress ** Changed in: linux (Ubuntu Trusty) Status: New => In Progress ** Changed in: linux (Ubuntu Trusty) Importance: Undecided => High ** Changed in: linux (Ubuntu Wily) Importance: Undecided => High ** Changed in: linux (Ubuntu Vivid) Importance: Undecided => High ** Changed in: linux (Ubuntu Xenial) Importance: Undecided => High ** Changed in: linux (Ubuntu Xenial) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: linux (Ubuntu Wily) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: linux (Ubuntu Vivid) Assignee: (unassigned) => Rafael David Tinoco (inaddy) ** Changed in: linux (Ubuntu Trusty) Assignee: (unassigned) => Rafael David Tinoco (inaddy) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: In Progress Status in linux source package in Trusty: In Progress Status in linux source package in Vivid: In Progress Status in linux source package in Wily: In Progress Status in linux source package in Xenial: In Progress Status in linux source package in Yakkety: In Progress Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
** Tags added: canonical-bootstack -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: In Progress Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: 7f8c3aeba370 RCX: 7f8bdea18000 RDX: 22c18020 RSI: 1000 RDI: 7f8bdea18000 RBP: 7f8c3ab82e40 R8: 7f8bb80c9a40 R9: 7f8c3aeba498 R10: R11: 22c18020 R12: 22c18020 R13: 00022c18 R14: R15: ORIG_RAX: CS: 0033 SS: 002b ## kernel 4.4 # kern.log 544 [ 360.282132] INFO: task qemu-system-x86:1592 blocked for more than 120 seconds. 545 [ 360.282984] Not tainted 4.4.0-27-generic #46~14.04.1-Ubuntu 546 [ 360.283581] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 547 [ 360.284439] qemu-system-x86 D 8800bb833e90 0 1592 1 0x 548 [ 360.284443] 8800bb833e90 88023151c4c0 8802345eb700 8800bb834000 549 [ 360.28] 0010 81efe6d0 55ac8fa05520 7f88fc7f7d88 550 [ 360.284445] 8800bb833ea8 817ed5f5 8800bb833ef0 8800bb833f38 551 [ 360.284447] Call Trace: 552 [ 360.284472] [] schedule+0x35/0x80 553 [ 360.284481] [] kvm_async_pf_task_wait+0x1a3/0x1f0 554 [ 360.284487] [] ? prepare_to_wait_event+0xf0/0xf0 555 [ 360.284494] [] ? do_sendfile+0x360/0x380 556 [ 360.284495]
[Kernel-packages] [Bug 1596941] Re: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive)
The following upstream patch: >From 25fb213d873977290caf374234df496ad158ec1e Mon Sep 17 00:00:00 2001 From: Rik van RielDate: Mon, 21 Mar 2016 15:13:27 +0100 Subject: [PATCH 2/2] kvm, rt: change async pagefault code locking for PREEMPT_RT The async pagefault wake code can run from the idle task in exception context, so everything here needs to be made non-preemptible. Conversion to a simple wait queue and raw spinlock does the trick. Signed-off-by: Rik van Riel Signed-off-by: Paolo Bonzini Fixes the issue by not letting async pagefault code to be preempted due to waitqueues. Backport for Trusty needs: >From 25fb213d873977290caf374234df496ad158ec1e Mon Sep 17 00:00:00 2001 From: Rik van Riel Date: Mon, 21 Mar 2016 15:13:27 +0100 Subject: [PATCH 2/2] kvm, rt: change async pagefault code locking for PREEMPT_RT >From 6b9cf536987c69825f91af9478109aa7bcbebc94 Mon Sep 17 00:00:00 2001 From: "Peter Zijlstra (Intel)" Date: Fri, 19 Feb 2016 09:46:37 +0100 Subject: [PATCH 1/2] wait.[ch]: Introduce the simple waitqueue (swait) implementation If adding simple waitqueue interface to Trusty is not acceptable as SRU I'll have to come up with something else. I'm sure that problem goes away when using these 2 patches. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1596941 Title: KVM deadlock on KVM guest migration with latest QEMU (mitaka) from Xenial (or Mitaka Ubuntu Cloud Archive) Status in linux package in Ubuntu: In Progress Bug description: It was brought to my knowledge that qemu-kvm live migration (with full storage copy) on Trusty + Mitaka Ubuntu Cloud Archive was broken. When investigating I stepped into the following situation: crash> sys KERNEL: /usr/lib/debug/boot/vmlinux-3.13.0-86-generic DUMPFILE: ./201606241546/dump.201606241546 [PARTIAL DUMP] CPUS: 4 DATE: Fri Jun 24 15:46:39 2016 UPTIME: 00:06:00 LOAD AVERAGE: 1.00, 0.60, 0.26 TASKS: 146 NODENAME: vmqemulivefail1 RELEASE: 3.13.0-86-generic VERSION: #131-Ubuntu SMP Thu May 12 23:33:13 UTC 2016 MACHINE: x86_64 (2494 Mhz) MEMORY: 8 GB PANIC: "Kernel panic - not syncing: hung_task: blocked tasks" Full backtrace doesn't have anything useful since i've configured kernel.softlockup_panic. From scheduled-out tasks (and from kern.log) I was able to see that in more than one occasion I had the qemu process possibly dead-locked when dealing with asynchronous page faults: ## kernel 3.13 # dump 1 PID: 1604 TASK: 8800374be000 CPU: 3 COMMAND: "qemu-system-x86" #0 [8800ba115e28] __schedule at 8172e379 #1 [8800ba115e90] schedule at 8172e859 #2 [8800ba115ea0] kvm_async_pf_task_wait at 8105060f #3 [8800ba115f38] do_async_page_fault at 81736090 #4 [8800ba115f50] async_page_fault at 81732cd8 RIP: 7fb4eff0a4b3 RSP: 7fb4713facb0 RFLAGS: 00010206 RAX: 7fb4cb9cf000 RBX: 7fb4f166d8f0 RCX: 0010 RDX: 1fff RSI: 7fb4cb9deff8 RDI: 4000 RBP: R8: R9: 0002601b R10: 00fffe00 R11: 1fff R12: 0008 R13: 7fb4713fad84 R14: 7fb4f1665290 R15: 7fb4713fad88 ORIG_RAX: CS: 0033 SS: 002b # dump 2 PID: 1735 TASK: 8800b9bcb000 CPU: 2 COMMAND: "qemu-system-x86" #0 [8802333c9e28] __schedule at 8172e379 #1 [8802333c9e90] schedule at 8172e859 #2 [8802333c9ea0] kvm_async_pf_task_wait at 8105060f #3 [8802333c9f38] do_async_page_fault at 81736090 #4 [8802333c9f50] async_page_fault at 81732cd8 RIP: 7f631399d3b0 RSP: 7f62912c7990 RFLAGS: 00010206 RAX: RBX: 7f6315f9e370 RCX: 7f62ca714000 RDX: 32914020 RSI: 1000 RDI: 7f62ca714000 RBP: 7f6315c66e40 R8: 7f62912c7a40 R9: 7f6315f9e3e0 R10: R11: 32914020 R12: 32914020 R13: 00032914 R14: R15: ORIG_RAX: CS: 0033 SS: 002b # dump 3 PID: 1617 TASK: 880232834800 CPU: 3 COMMAND: "qemu-system-x86" #0 [880232a6de28] __schedule at 8172e379 #1 [880232a6de90] schedule at 8172e859 #2 [880232a6dea0] kvm_async_pf_task_wait at 8105060f #3 [880232a6df38] do_async_page_fault at 81736090 #4 [880232a6df50] async_page_fault at 81732cd8 RIP: 7f8c39e8b3b0 RSP: 7f8bb80c9990 RFLAGS: 00010206 RAX: RBX: