Re: host stalls when qemu-system-aarch64 with kvm and pflash
On 03/30/2017 05:51 AM, Marc Zyngier wrote: > On 29/03/17 19:56, Christoffer Dall wrote: >> On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >>> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dallwrote: Hi Radha, On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: > Hi, > I am seeing an issue with qemu-system-aarch64 when using pflash > (booting kernel via UEFI bios). > > Host kernel: 4.11.0-rc3-next-20170323 > Qemu version: v2.9.0-rc1 > > Command used: > ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M > virt,gic_version=3 -nographic -smp 1 -m 2048 -drive > if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device > virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash > /root/flash1.img > > > As soon as the guest kernel boots the host starts to stall and prints > the below messages. And the system never recovers. I can neither > poweroff the guest nor the host. So I have resort to external power > reset of the host. > > == > [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! > [kworker/25:1:454] > [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk > shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce > sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai > d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm > drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s > ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv > mdio_thunder mdio_cavium > > [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted > 4.11.0-rc3-next-20170323 #1 > [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb > 23 2017 > [ 116.207010] Workqueue: events netstamp_clear > [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 > [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 > [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 > [ 116.207026] pc : [] lr : [] > pstate: 8145 > [ 116.207028] sp : 801f901a7be0 > [ 116.207030] x29: 801f901a7be0 x28: 09139000 > [ 116.207036] x27: 09139434 x26: 0080 > [ 116.207041] x25: x24: 081565d0 > [ 116.207047] x23: 0001 x22: 08e11e00 > [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 > [ 116.207057] x19: 09138e38 x18: 0a03 > [ 116.207063] x17: b77c9028 x16: 082e81d8 > [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a > [ 116.207073] x13: 58dabe4c x12: 0018 > [ 116.207079] x11: 366e2f04 x10: 09f0 > [ 116.207084] x9 : 801f901a7d30 x8 : 0002 > [ 116.207089] x7 : x6 : > [ 116.207095] x5 : x4 : 0020 > [ 116.207100] x3 : 0020 x2 : > [ 116.207105] x1 : 801f6d682578 x0 : 0003 > > [ 150.443116] INFO: rcu_sched self-detected stall on CPU > [ 150.448261] 25-...: (14997 ticks this GP) > idle=47a/141/0 softirq=349/349 fqs=7495 > [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: > [ 150.451123] 25-...: (14997 ticks this GP) > idle=47a/141/0 softirq=349/349 fqs=7495 > [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) > [ 150.451136] Task dump for CPU 25: > [ 150.451138] kworker/25:1R running task0 454 2 > 0x0002 > [ 150.451155] Workqueue: events netstamp_clear > [ 150.451158] Call trace: > [ 150.451164] [] __switch_to+0x90/0xa8 > [ 150.451172] [] static_key_slow_inc+0x128/0x138 > [ 150.451175] [] static_key_enable+0x34/0x60 > [ 150.451178] [] netstamp_clear+0x68/0x80 > [ 150.451181] [] process_one_work+0x158/0x478 > [ 150.451183] [] worker_thread+0x50/0x4a8 > [ 150.451187] [] kthread+0x108/0x138 > [ 150.451190] [] ret_from_fork+0x10/0x50 > [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) > [ 150.482242] Task dump for CPU 25: > [ 150.482245] kworker/25:1R running task0 454 2 > 0x0002 > [ 150.482259] Workqueue: events netstamp_clear > [ 150.482264] Call trace: > [ 150.482271] [] dump_backtrace+0x0/0x2b0 > [ 150.482277] [] show_stack+0x24/0x30 > [ 150.482281] [] sched_show_task+0x128/0x178 > [ 150.482285] [] dump_cpu_task+0x48/0x58 > [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 > [ 150.482297] [] rcu_check_callbacks+0x774/0x938 > [ 150.482305] [] update_process_times+0x34/0x60 > [
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Thu, Mar 30, 2017 at 9:47 AM, Laszlo Ersekwrote: > On 03/29/17 20:56, Christoffer Dall wrote: >> On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >>> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dall wrote: Hi Radha, On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: > Hi, > I am seeing an issue with qemu-system-aarch64 when using pflash > (booting kernel via UEFI bios). > > Host kernel: 4.11.0-rc3-next-20170323 > Qemu version: v2.9.0-rc1 > > Command used: > ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M > virt,gic_version=3 -nographic -smp 1 -m 2048 -drive > if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device > virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash > /root/flash1.img > > > As soon as the guest kernel boots the host starts to stall and prints > the below messages. And the system never recovers. I can neither > poweroff the guest nor the host. So I have resort to external power > reset of the host. > > == > [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! > [kworker/25:1:454] > [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk > shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce > sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai > d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm > drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s > ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv > mdio_thunder mdio_cavium > > [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted > 4.11.0-rc3-next-20170323 #1 > [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb > 23 2017 > [ 116.207010] Workqueue: events netstamp_clear > [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 > [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 > [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 > [ 116.207026] pc : [] lr : [] > pstate: 8145 > [ 116.207028] sp : 801f901a7be0 > [ 116.207030] x29: 801f901a7be0 x28: 09139000 > [ 116.207036] x27: 09139434 x26: 0080 > [ 116.207041] x25: x24: 081565d0 > [ 116.207047] x23: 0001 x22: 08e11e00 > [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 > [ 116.207057] x19: 09138e38 x18: 0a03 > [ 116.207063] x17: b77c9028 x16: 082e81d8 > [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a > [ 116.207073] x13: 58dabe4c x12: 0018 > [ 116.207079] x11: 366e2f04 x10: 09f0 > [ 116.207084] x9 : 801f901a7d30 x8 : 0002 > [ 116.207089] x7 : x6 : > [ 116.207095] x5 : x4 : 0020 > [ 116.207100] x3 : 0020 x2 : > [ 116.207105] x1 : 801f6d682578 x0 : 0003 > > [ 150.443116] INFO: rcu_sched self-detected stall on CPU > [ 150.448261] 25-...: (14997 ticks this GP) > idle=47a/141/0 softirq=349/349 fqs=7495 > [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: > [ 150.451123] 25-...: (14997 ticks this GP) > idle=47a/141/0 softirq=349/349 fqs=7495 > [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) > [ 150.451136] Task dump for CPU 25: > [ 150.451138] kworker/25:1R running task0 454 2 > 0x0002 > [ 150.451155] Workqueue: events netstamp_clear > [ 150.451158] Call trace: > [ 150.451164] [] __switch_to+0x90/0xa8 > [ 150.451172] [] static_key_slow_inc+0x128/0x138 > [ 150.451175] [] static_key_enable+0x34/0x60 > [ 150.451178] [] netstamp_clear+0x68/0x80 > [ 150.451181] [] process_one_work+0x158/0x478 > [ 150.451183] [] worker_thread+0x50/0x4a8 > [ 150.451187] [] kthread+0x108/0x138 > [ 150.451190] [] ret_from_fork+0x10/0x50 > [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) > [ 150.482242] Task dump for CPU 25: > [ 150.482245] kworker/25:1R running task0 454 2 > 0x0002 > [ 150.482259] Workqueue: events netstamp_clear > [ 150.482264] Call trace: > [ 150.482271] [] dump_backtrace+0x0/0x2b0 > [ 150.482277] [] show_stack+0x24/0x30 > [ 150.482281] [] sched_show_task+0x128/0x178 > [ 150.482285] [] dump_cpu_task+0x48/0x58 > [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 > [ 150.482297] [] rcu_check_callbacks+0x774/0x938 > [ 150.482305] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On 03/29/17 20:56, Christoffer Dall wrote: > On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dallwrote: >>> Hi Radha, >>> >>> On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: Hi, I am seeing an issue with qemu-system-aarch64 when using pflash (booting kernel via UEFI bios). Host kernel: 4.11.0-rc3-next-20170323 Qemu version: v2.9.0-rc1 Command used: ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M virt,gic_version=3 -nographic -smp 1 -m 2048 -drive if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash /root/flash1.img As soon as the guest kernel boots the host starts to stall and prints the below messages. And the system never recovers. I can neither poweroff the guest nor the host. So I have resort to external power reset of the host. == [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [kworker/25:1:454] [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv mdio_thunder mdio_cavium [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted 4.11.0-rc3-next-20170323 #1 [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb 23 2017 [ 116.207010] Workqueue: events netstamp_clear [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 [ 116.207026] pc : [] lr : [] pstate: 8145 [ 116.207028] sp : 801f901a7be0 [ 116.207030] x29: 801f901a7be0 x28: 09139000 [ 116.207036] x27: 09139434 x26: 0080 [ 116.207041] x25: x24: 081565d0 [ 116.207047] x23: 0001 x22: 08e11e00 [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 [ 116.207057] x19: 09138e38 x18: 0a03 [ 116.207063] x17: b77c9028 x16: 082e81d8 [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a [ 116.207073] x13: 58dabe4c x12: 0018 [ 116.207079] x11: 366e2f04 x10: 09f0 [ 116.207084] x9 : 801f901a7d30 x8 : 0002 [ 116.207089] x7 : x6 : [ 116.207095] x5 : x4 : 0020 [ 116.207100] x3 : 0020 x2 : [ 116.207105] x1 : 801f6d682578 x0 : 0003 [ 150.443116] INFO: rcu_sched self-detected stall on CPU [ 150.448261] 25-...: (14997 ticks this GP) idle=47a/141/0 softirq=349/349 fqs=7495 [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: [ 150.451123] 25-...: (14997 ticks this GP) idle=47a/141/0 softirq=349/349 fqs=7495 [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) [ 150.451136] Task dump for CPU 25: [ 150.451138] kworker/25:1R running task0 454 2 0x0002 [ 150.451155] Workqueue: events netstamp_clear [ 150.451158] Call trace: [ 150.451164] [] __switch_to+0x90/0xa8 [ 150.451172] [] static_key_slow_inc+0x128/0x138 [ 150.451175] [] static_key_enable+0x34/0x60 [ 150.451178] [] netstamp_clear+0x68/0x80 [ 150.451181] [] process_one_work+0x158/0x478 [ 150.451183] [] worker_thread+0x50/0x4a8 [ 150.451187] [] kthread+0x108/0x138 [ 150.451190] [] ret_from_fork+0x10/0x50 [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) [ 150.482242] Task dump for CPU 25: [ 150.482245] kworker/25:1R running task0 454 2 0x0002 [ 150.482259] Workqueue: events netstamp_clear [ 150.482264] Call trace: [ 150.482271] [] dump_backtrace+0x0/0x2b0 [ 150.482277] [] show_stack+0x24/0x30 [ 150.482281] [] sched_show_task+0x128/0x178 [ 150.482285] [] dump_cpu_task+0x48/0x58 [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 [ 150.482297] [] rcu_check_callbacks+0x774/0x938 [ 150.482305] [] update_process_times+0x34/0x60 [ 150.482314] [] tick_sched_handle.isra.7+0x38/0x70 [ 150.482319] [] tick_sched_timer+0x4c/0x98 [ 150.482324] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On 29/03/17 19:56, Christoffer Dall wrote: > On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dallwrote: >>> Hi Radha, >>> >>> On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: Hi, I am seeing an issue with qemu-system-aarch64 when using pflash (booting kernel via UEFI bios). Host kernel: 4.11.0-rc3-next-20170323 Qemu version: v2.9.0-rc1 Command used: ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M virt,gic_version=3 -nographic -smp 1 -m 2048 -drive if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash /root/flash1.img As soon as the guest kernel boots the host starts to stall and prints the below messages. And the system never recovers. I can neither poweroff the guest nor the host. So I have resort to external power reset of the host. == [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [kworker/25:1:454] [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv mdio_thunder mdio_cavium [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted 4.11.0-rc3-next-20170323 #1 [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb 23 2017 [ 116.207010] Workqueue: events netstamp_clear [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 [ 116.207026] pc : [] lr : [] pstate: 8145 [ 116.207028] sp : 801f901a7be0 [ 116.207030] x29: 801f901a7be0 x28: 09139000 [ 116.207036] x27: 09139434 x26: 0080 [ 116.207041] x25: x24: 081565d0 [ 116.207047] x23: 0001 x22: 08e11e00 [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 [ 116.207057] x19: 09138e38 x18: 0a03 [ 116.207063] x17: b77c9028 x16: 082e81d8 [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a [ 116.207073] x13: 58dabe4c x12: 0018 [ 116.207079] x11: 366e2f04 x10: 09f0 [ 116.207084] x9 : 801f901a7d30 x8 : 0002 [ 116.207089] x7 : x6 : [ 116.207095] x5 : x4 : 0020 [ 116.207100] x3 : 0020 x2 : [ 116.207105] x1 : 801f6d682578 x0 : 0003 [ 150.443116] INFO: rcu_sched self-detected stall on CPU [ 150.448261] 25-...: (14997 ticks this GP) idle=47a/141/0 softirq=349/349 fqs=7495 [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: [ 150.451123] 25-...: (14997 ticks this GP) idle=47a/141/0 softirq=349/349 fqs=7495 [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) [ 150.451136] Task dump for CPU 25: [ 150.451138] kworker/25:1R running task0 454 2 0x0002 [ 150.451155] Workqueue: events netstamp_clear [ 150.451158] Call trace: [ 150.451164] [] __switch_to+0x90/0xa8 [ 150.451172] [] static_key_slow_inc+0x128/0x138 [ 150.451175] [] static_key_enable+0x34/0x60 [ 150.451178] [] netstamp_clear+0x68/0x80 [ 150.451181] [] process_one_work+0x158/0x478 [ 150.451183] [] worker_thread+0x50/0x4a8 [ 150.451187] [] kthread+0x108/0x138 [ 150.451190] [] ret_from_fork+0x10/0x50 [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) [ 150.482242] Task dump for CPU 25: [ 150.482245] kworker/25:1R running task0 454 2 0x0002 [ 150.482259] Workqueue: events netstamp_clear [ 150.482264] Call trace: [ 150.482271] [] dump_backtrace+0x0/0x2b0 [ 150.482277] [] show_stack+0x24/0x30 [ 150.482281] [] sched_show_task+0x128/0x178 [ 150.482285] [] dump_cpu_task+0x48/0x58 [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 [ 150.482297] [] rcu_check_callbacks+0x774/0x938 [ 150.482305] [] update_process_times+0x34/0x60 [ 150.482314] [] tick_sched_handle.isra.7+0x38/0x70 [ 150.482319] [] tick_sched_timer+0x4c/0x98 [ 150.482324] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Wed, Mar 29, 2017 at 2:06 PM, Christoffer Dallwrote: > On Wed, Mar 29, 2017 at 01:51:19PM -0700, Radha Mohan wrote: >> On Wed, Mar 29, 2017 at 11:56 AM, Christoffer Dall wrote: >> > On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >> >> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dall >> >> wrote: >> >> > Hi Radha, >> >> > >> >> > On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: >> >> >> Hi, >> >> >> I am seeing an issue with qemu-system-aarch64 when using pflash >> >> >> (booting kernel via UEFI bios). >> >> >> >> >> >> Host kernel: 4.11.0-rc3-next-20170323 >> >> >> Qemu version: v2.9.0-rc1 >> >> >> >> >> >> Command used: >> >> >> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M >> >> >> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive >> >> >> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device >> >> >> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash >> >> >> /root/flash1.img >> >> >> >> >> >> >> >> >> As soon as the guest kernel boots the host starts to stall and prints >> >> >> the below messages. And the system never recovers. I can neither >> >> >> poweroff the guest nor the host. So I have resort to external power >> >> >> reset of the host. >> >> >> >> >> >> == >> >> >> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! >> >> >> [kworker/25:1:454] >> >> >> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk >> >> >> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce >> >> >> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai >> >> >> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor >> >> >> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm >> >> >> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s >> >> >> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv >> >> >> mdio_thunder mdio_cavium >> >> >> >> >> >> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted >> >> >> 4.11.0-rc3-next-20170323 #1 >> >> >> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 >> >> >> Feb 23 2017 >> >> >> [ 116.207010] Workqueue: events netstamp_clear >> >> >> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 >> >> >> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 >> >> >> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 >> >> >> [ 116.207026] pc : [] lr : [] >> >> >> pstate: 8145 >> >> >> [ 116.207028] sp : 801f901a7be0 >> >> >> [ 116.207030] x29: 801f901a7be0 x28: 09139000 >> >> >> [ 116.207036] x27: 09139434 x26: 0080 >> >> >> [ 116.207041] x25: x24: 081565d0 >> >> >> [ 116.207047] x23: 0001 x22: 08e11e00 >> >> >> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 >> >> >> [ 116.207057] x19: 09138e38 x18: 0a03 >> >> >> [ 116.207063] x17: b77c9028 x16: 082e81d8 >> >> >> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a >> >> >> [ 116.207073] x13: 58dabe4c x12: 0018 >> >> >> [ 116.207079] x11: 366e2f04 x10: 09f0 >> >> >> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 >> >> >> [ 116.207089] x7 : x6 : >> >> >> [ 116.207095] x5 : x4 : 0020 >> >> >> [ 116.207100] x3 : 0020 x2 : >> >> >> [ 116.207105] x1 : 801f6d682578 x0 : 0003 >> >> >> >> >> >> [ 150.443116] INFO: rcu_sched self-detected stall on CPU >> >> >> [ 150.448261] 25-...: (14997 ticks this GP) >> >> >> idle=47a/141/0 softirq=349/349 fqs=7495 >> >> >> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: >> >> >> [ 150.451123] 25-...: (14997 ticks this GP) >> >> >> idle=47a/141/0 softirq=349/349 fqs=7495 >> >> >> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) >> >> >> [ 150.451136] Task dump for CPU 25: >> >> >> [ 150.451138] kworker/25:1R running task0 454 2 >> >> >> 0x0002 >> >> >> [ 150.451155] Workqueue: events netstamp_clear >> >> >> [ 150.451158] Call trace: >> >> >> [ 150.451164] [] __switch_to+0x90/0xa8 >> >> >> [ 150.451172] [] static_key_slow_inc+0x128/0x138 >> >> >> [ 150.451175] [] static_key_enable+0x34/0x60 >> >> >> [ 150.451178] [] netstamp_clear+0x68/0x80 >> >> >> [ 150.451181] [] process_one_work+0x158/0x478 >> >> >> [ 150.451183] [] worker_thread+0x50/0x4a8 >> >> >> [ 150.451187] [] kthread+0x108/0x138 >> >> >> [ 150.451190] [] ret_from_fork+0x10/0x50 >> >> >> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) >> >> >> [ 150.482242] Task dump for CPU 25: >> >> >> [ 150.482245] kworker/25:1R running task0 454 2 >> >> >> 0x0002 >> >> >> [ 150.482259] Workqueue: events netstamp_clear >> >> >>
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Wed, Mar 29, 2017 at 01:51:19PM -0700, Radha Mohan wrote: > On Wed, Mar 29, 2017 at 11:56 AM, Christoffer Dallwrote: > > On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: > >> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dall wrote: > >> > Hi Radha, > >> > > >> > On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: > >> >> Hi, > >> >> I am seeing an issue with qemu-system-aarch64 when using pflash > >> >> (booting kernel via UEFI bios). > >> >> > >> >> Host kernel: 4.11.0-rc3-next-20170323 > >> >> Qemu version: v2.9.0-rc1 > >> >> > >> >> Command used: > >> >> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M > >> >> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive > >> >> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device > >> >> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash > >> >> /root/flash1.img > >> >> > >> >> > >> >> As soon as the guest kernel boots the host starts to stall and prints > >> >> the below messages. And the system never recovers. I can neither > >> >> poweroff the guest nor the host. So I have resort to external power > >> >> reset of the host. > >> >> > >> >> == > >> >> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! > >> >> [kworker/25:1:454] > >> >> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk > >> >> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce > >> >> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai > >> >> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > >> >> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm > >> >> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s > >> >> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv > >> >> mdio_thunder mdio_cavium > >> >> > >> >> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted > >> >> 4.11.0-rc3-next-20170323 #1 > >> >> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 > >> >> Feb 23 2017 > >> >> [ 116.207010] Workqueue: events netstamp_clear > >> >> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 > >> >> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 > >> >> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 > >> >> [ 116.207026] pc : [] lr : [] > >> >> pstate: 8145 > >> >> [ 116.207028] sp : 801f901a7be0 > >> >> [ 116.207030] x29: 801f901a7be0 x28: 09139000 > >> >> [ 116.207036] x27: 09139434 x26: 0080 > >> >> [ 116.207041] x25: x24: 081565d0 > >> >> [ 116.207047] x23: 0001 x22: 08e11e00 > >> >> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 > >> >> [ 116.207057] x19: 09138e38 x18: 0a03 > >> >> [ 116.207063] x17: b77c9028 x16: 082e81d8 > >> >> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a > >> >> [ 116.207073] x13: 58dabe4c x12: 0018 > >> >> [ 116.207079] x11: 366e2f04 x10: 09f0 > >> >> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 > >> >> [ 116.207089] x7 : x6 : > >> >> [ 116.207095] x5 : x4 : 0020 > >> >> [ 116.207100] x3 : 0020 x2 : > >> >> [ 116.207105] x1 : 801f6d682578 x0 : 0003 > >> >> > >> >> [ 150.443116] INFO: rcu_sched self-detected stall on CPU > >> >> [ 150.448261] 25-...: (14997 ticks this GP) > >> >> idle=47a/141/0 softirq=349/349 fqs=7495 > >> >> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: > >> >> [ 150.451123] 25-...: (14997 ticks this GP) > >> >> idle=47a/141/0 softirq=349/349 fqs=7495 > >> >> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) > >> >> [ 150.451136] Task dump for CPU 25: > >> >> [ 150.451138] kworker/25:1R running task0 454 2 > >> >> 0x0002 > >> >> [ 150.451155] Workqueue: events netstamp_clear > >> >> [ 150.451158] Call trace: > >> >> [ 150.451164] [] __switch_to+0x90/0xa8 > >> >> [ 150.451172] [] static_key_slow_inc+0x128/0x138 > >> >> [ 150.451175] [] static_key_enable+0x34/0x60 > >> >> [ 150.451178] [] netstamp_clear+0x68/0x80 > >> >> [ 150.451181] [] process_one_work+0x158/0x478 > >> >> [ 150.451183] [] worker_thread+0x50/0x4a8 > >> >> [ 150.451187] [] kthread+0x108/0x138 > >> >> [ 150.451190] [] ret_from_fork+0x10/0x50 > >> >> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) > >> >> [ 150.482242] Task dump for CPU 25: > >> >> [ 150.482245] kworker/25:1R running task0 454 2 > >> >> 0x0002 > >> >> [ 150.482259] Workqueue: events netstamp_clear > >> >> [ 150.482264] Call trace: > >> >> [ 150.482271] [] dump_backtrace+0x0/0x2b0 > >> >> [ 150.482277] [] show_stack+0x24/0x30 > >> >> [ 150.482281] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Wed, Mar 29, 2017 at 11:56 AM, Christoffer Dallwrote: > On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: >> On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dall wrote: >> > Hi Radha, >> > >> > On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: >> >> Hi, >> >> I am seeing an issue with qemu-system-aarch64 when using pflash >> >> (booting kernel via UEFI bios). >> >> >> >> Host kernel: 4.11.0-rc3-next-20170323 >> >> Qemu version: v2.9.0-rc1 >> >> >> >> Command used: >> >> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M >> >> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive >> >> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device >> >> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash >> >> /root/flash1.img >> >> >> >> >> >> As soon as the guest kernel boots the host starts to stall and prints >> >> the below messages. And the system never recovers. I can neither >> >> poweroff the guest nor the host. So I have resort to external power >> >> reset of the host. >> >> >> >> == >> >> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! >> >> [kworker/25:1:454] >> >> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk >> >> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce >> >> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai >> >> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor >> >> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm >> >> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s >> >> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv >> >> mdio_thunder mdio_cavium >> >> >> >> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted >> >> 4.11.0-rc3-next-20170323 #1 >> >> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb >> >> 23 2017 >> >> [ 116.207010] Workqueue: events netstamp_clear >> >> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 >> >> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 >> >> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 >> >> [ 116.207026] pc : [] lr : [] >> >> pstate: 8145 >> >> [ 116.207028] sp : 801f901a7be0 >> >> [ 116.207030] x29: 801f901a7be0 x28: 09139000 >> >> [ 116.207036] x27: 09139434 x26: 0080 >> >> [ 116.207041] x25: x24: 081565d0 >> >> [ 116.207047] x23: 0001 x22: 08e11e00 >> >> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 >> >> [ 116.207057] x19: 09138e38 x18: 0a03 >> >> [ 116.207063] x17: b77c9028 x16: 082e81d8 >> >> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a >> >> [ 116.207073] x13: 58dabe4c x12: 0018 >> >> [ 116.207079] x11: 366e2f04 x10: 09f0 >> >> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 >> >> [ 116.207089] x7 : x6 : >> >> [ 116.207095] x5 : x4 : 0020 >> >> [ 116.207100] x3 : 0020 x2 : >> >> [ 116.207105] x1 : 801f6d682578 x0 : 0003 >> >> >> >> [ 150.443116] INFO: rcu_sched self-detected stall on CPU >> >> [ 150.448261] 25-...: (14997 ticks this GP) >> >> idle=47a/141/0 softirq=349/349 fqs=7495 >> >> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: >> >> [ 150.451123] 25-...: (14997 ticks this GP) >> >> idle=47a/141/0 softirq=349/349 fqs=7495 >> >> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) >> >> [ 150.451136] Task dump for CPU 25: >> >> [ 150.451138] kworker/25:1R running task0 454 2 >> >> 0x0002 >> >> [ 150.451155] Workqueue: events netstamp_clear >> >> [ 150.451158] Call trace: >> >> [ 150.451164] [] __switch_to+0x90/0xa8 >> >> [ 150.451172] [] static_key_slow_inc+0x128/0x138 >> >> [ 150.451175] [] static_key_enable+0x34/0x60 >> >> [ 150.451178] [] netstamp_clear+0x68/0x80 >> >> [ 150.451181] [] process_one_work+0x158/0x478 >> >> [ 150.451183] [] worker_thread+0x50/0x4a8 >> >> [ 150.451187] [] kthread+0x108/0x138 >> >> [ 150.451190] [] ret_from_fork+0x10/0x50 >> >> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) >> >> [ 150.482242] Task dump for CPU 25: >> >> [ 150.482245] kworker/25:1R running task0 454 2 >> >> 0x0002 >> >> [ 150.482259] Workqueue: events netstamp_clear >> >> [ 150.482264] Call trace: >> >> [ 150.482271] [] dump_backtrace+0x0/0x2b0 >> >> [ 150.482277] [] show_stack+0x24/0x30 >> >> [ 150.482281] [] sched_show_task+0x128/0x178 >> >> [ 150.482285] [] dump_cpu_task+0x48/0x58 >> >> [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 >> >> [ 150.482297] [] rcu_check_callbacks+0x774/0x938 >> >> [ 150.482305] [] update_process_times+0x34/0x60 >> >> [ 150.482314] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Tue, Mar 28, 2017 at 01:24:15PM -0700, Radha Mohan wrote: > On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dallwrote: > > Hi Radha, > > > > On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: > >> Hi, > >> I am seeing an issue with qemu-system-aarch64 when using pflash > >> (booting kernel via UEFI bios). > >> > >> Host kernel: 4.11.0-rc3-next-20170323 > >> Qemu version: v2.9.0-rc1 > >> > >> Command used: > >> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M > >> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive > >> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device > >> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash > >> /root/flash1.img > >> > >> > >> As soon as the guest kernel boots the host starts to stall and prints > >> the below messages. And the system never recovers. I can neither > >> poweroff the guest nor the host. So I have resort to external power > >> reset of the host. > >> > >> == > >> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! > >> [kworker/25:1:454] > >> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk > >> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce > >> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai > >> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor > >> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm > >> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s > >> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv > >> mdio_thunder mdio_cavium > >> > >> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted > >> 4.11.0-rc3-next-20170323 #1 > >> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb > >> 23 2017 > >> [ 116.207010] Workqueue: events netstamp_clear > >> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 > >> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 > >> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 > >> [ 116.207026] pc : [] lr : [] > >> pstate: 8145 > >> [ 116.207028] sp : 801f901a7be0 > >> [ 116.207030] x29: 801f901a7be0 x28: 09139000 > >> [ 116.207036] x27: 09139434 x26: 0080 > >> [ 116.207041] x25: x24: 081565d0 > >> [ 116.207047] x23: 0001 x22: 08e11e00 > >> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 > >> [ 116.207057] x19: 09138e38 x18: 0a03 > >> [ 116.207063] x17: b77c9028 x16: 082e81d8 > >> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a > >> [ 116.207073] x13: 58dabe4c x12: 0018 > >> [ 116.207079] x11: 366e2f04 x10: 09f0 > >> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 > >> [ 116.207089] x7 : x6 : > >> [ 116.207095] x5 : x4 : 0020 > >> [ 116.207100] x3 : 0020 x2 : > >> [ 116.207105] x1 : 801f6d682578 x0 : 0003 > >> > >> [ 150.443116] INFO: rcu_sched self-detected stall on CPU > >> [ 150.448261] 25-...: (14997 ticks this GP) > >> idle=47a/141/0 softirq=349/349 fqs=7495 > >> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: > >> [ 150.451123] 25-...: (14997 ticks this GP) > >> idle=47a/141/0 softirq=349/349 fqs=7495 > >> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) > >> [ 150.451136] Task dump for CPU 25: > >> [ 150.451138] kworker/25:1R running task0 454 2 > >> 0x0002 > >> [ 150.451155] Workqueue: events netstamp_clear > >> [ 150.451158] Call trace: > >> [ 150.451164] [] __switch_to+0x90/0xa8 > >> [ 150.451172] [] static_key_slow_inc+0x128/0x138 > >> [ 150.451175] [] static_key_enable+0x34/0x60 > >> [ 150.451178] [] netstamp_clear+0x68/0x80 > >> [ 150.451181] [] process_one_work+0x158/0x478 > >> [ 150.451183] [] worker_thread+0x50/0x4a8 > >> [ 150.451187] [] kthread+0x108/0x138 > >> [ 150.451190] [] ret_from_fork+0x10/0x50 > >> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) > >> [ 150.482242] Task dump for CPU 25: > >> [ 150.482245] kworker/25:1R running task0 454 2 > >> 0x0002 > >> [ 150.482259] Workqueue: events netstamp_clear > >> [ 150.482264] Call trace: > >> [ 150.482271] [] dump_backtrace+0x0/0x2b0 > >> [ 150.482277] [] show_stack+0x24/0x30 > >> [ 150.482281] [] sched_show_task+0x128/0x178 > >> [ 150.482285] [] dump_cpu_task+0x48/0x58 > >> [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 > >> [ 150.482297] [] rcu_check_callbacks+0x774/0x938 > >> [ 150.482305] [] update_process_times+0x34/0x60 > >> [ 150.482314] [] tick_sched_handle.isra.7+0x38/0x70 > >> [ 150.482319] [] tick_sched_timer+0x4c/0x98 > >> [ 150.482324] [] __hrtimer_run_queues+0xd8/0x2b8 > >> [ 150.482328] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On 29 March 2017 at 19:17, Radha Mohanwrote: >> I will also try with qemu 2.8 and see. > > Same issue with 2.8.0 and 2.7.1 qemu version. Thanks for checking -- good to know it's not a regression on the QEMU side. (If you can soft-lockup the host it's clearly a host kernel bug, but it might have been that we changed something recently to provoke it.) thanks -- PMM ___ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Tue, Mar 28, 2017 at 1:24 PM, Radha Mohanwrote: > On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dall wrote: >> Hi Radha, >> >> On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: >>> Hi, >>> I am seeing an issue with qemu-system-aarch64 when using pflash >>> (booting kernel via UEFI bios). >>> >>> Host kernel: 4.11.0-rc3-next-20170323 >>> Qemu version: v2.9.0-rc1 >>> >>> Command used: >>> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M >>> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive >>> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device >>> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash >>> /root/flash1.img >>> >>> >>> As soon as the guest kernel boots the host starts to stall and prints >>> the below messages. And the system never recovers. I can neither >>> poweroff the guest nor the host. So I have resort to external power >>> reset of the host. >>> >>> == >>> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! >>> [kworker/25:1:454] >>> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk >>> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce >>> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai >>> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor >>> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm >>> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s >>> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv >>> mdio_thunder mdio_cavium >>> >>> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted >>> 4.11.0-rc3-next-20170323 #1 >>> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb 23 >>> 2017 >>> [ 116.207010] Workqueue: events netstamp_clear >>> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 >>> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 >>> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 >>> [ 116.207026] pc : [] lr : [] >>> pstate: 8145 >>> [ 116.207028] sp : 801f901a7be0 >>> [ 116.207030] x29: 801f901a7be0 x28: 09139000 >>> [ 116.207036] x27: 09139434 x26: 0080 >>> [ 116.207041] x25: x24: 081565d0 >>> [ 116.207047] x23: 0001 x22: 08e11e00 >>> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 >>> [ 116.207057] x19: 09138e38 x18: 0a03 >>> [ 116.207063] x17: b77c9028 x16: 082e81d8 >>> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a >>> [ 116.207073] x13: 58dabe4c x12: 0018 >>> [ 116.207079] x11: 366e2f04 x10: 09f0 >>> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 >>> [ 116.207089] x7 : x6 : >>> [ 116.207095] x5 : x4 : 0020 >>> [ 116.207100] x3 : 0020 x2 : >>> [ 116.207105] x1 : 801f6d682578 x0 : 0003 >>> >>> [ 150.443116] INFO: rcu_sched self-detected stall on CPU >>> [ 150.448261] 25-...: (14997 ticks this GP) >>> idle=47a/141/0 softirq=349/349 fqs=7495 >>> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: >>> [ 150.451123] 25-...: (14997 ticks this GP) >>> idle=47a/141/0 softirq=349/349 fqs=7495 >>> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) >>> [ 150.451136] Task dump for CPU 25: >>> [ 150.451138] kworker/25:1R running task0 454 2 >>> 0x0002 >>> [ 150.451155] Workqueue: events netstamp_clear >>> [ 150.451158] Call trace: >>> [ 150.451164] [] __switch_to+0x90/0xa8 >>> [ 150.451172] [] static_key_slow_inc+0x128/0x138 >>> [ 150.451175] [] static_key_enable+0x34/0x60 >>> [ 150.451178] [] netstamp_clear+0x68/0x80 >>> [ 150.451181] [] process_one_work+0x158/0x478 >>> [ 150.451183] [] worker_thread+0x50/0x4a8 >>> [ 150.451187] [] kthread+0x108/0x138 >>> [ 150.451190] [] ret_from_fork+0x10/0x50 >>> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) >>> [ 150.482242] Task dump for CPU 25: >>> [ 150.482245] kworker/25:1R running task0 454 2 >>> 0x0002 >>> [ 150.482259] Workqueue: events netstamp_clear >>> [ 150.482264] Call trace: >>> [ 150.482271] [] dump_backtrace+0x0/0x2b0 >>> [ 150.482277] [] show_stack+0x24/0x30 >>> [ 150.482281] [] sched_show_task+0x128/0x178 >>> [ 150.482285] [] dump_cpu_task+0x48/0x58 >>> [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 >>> [ 150.482297] [] rcu_check_callbacks+0x774/0x938 >>> [ 150.482305] [] update_process_times+0x34/0x60 >>> [ 150.482314] [] tick_sched_handle.isra.7+0x38/0x70 >>> [ 150.482319] [] tick_sched_timer+0x4c/0x98 >>> [ 150.482324] [] __hrtimer_run_queues+0xd8/0x2b8 >>> [ 150.482328] [] hrtimer_interrupt+0xa8/0x228 >>> [ 150.482334] [] arch_timer_handler_phys+0x3c/0x50 >>> [ 150.482341] []
Re: host stalls when qemu-system-aarch64 with kvm and pflash
On Tue, Mar 28, 2017 at 1:16 PM, Christoffer Dallwrote: > Hi Radha, > > On Tue, Mar 28, 2017 at 12:58:24PM -0700, Radha Mohan wrote: >> Hi, >> I am seeing an issue with qemu-system-aarch64 when using pflash >> (booting kernel via UEFI bios). >> >> Host kernel: 4.11.0-rc3-next-20170323 >> Qemu version: v2.9.0-rc1 >> >> Command used: >> ./aarch64-softmmu/qemu-system-aarch64 -cpu host -enable-kvm -M >> virt,gic_version=3 -nographic -smp 1 -m 2048 -drive >> if=none,id=hd0,file=/root/zesty-server-cloudimg-arm64.img,id=0 -device >> virtio-blk-device,drive=hd0 -pflash /root/flash0.img -pflash >> /root/flash1.img >> >> >> As soon as the guest kernel boots the host starts to stall and prints >> the below messages. And the system never recovers. I can neither >> poweroff the guest nor the host. So I have resort to external power >> reset of the host. >> >> == >> [ 116.199077] NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! >> [kworker/25:1:454] >> [ 116.206901] Modules linked in: binfmt_misc nls_iso8859_1 aes_ce_blk >> shpchp crypto_simd gpio_keys cryptd aes_ce_cipher ghash_ce sha2_ce >> sha1_ce uio_pdrv_genirq uio autofs4 btrfs raid10 rai >> d456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor >> raid6_pq libcrc32c raid1 raid0 multipath linear ast i2c_algo_bit ttm >> drm_kms_helper syscopyarea sysfillrect sysimgblt fb_s >> ys_fops drm nicvf ahci nicpf libahci thunder_bgx thunder_xcv >> mdio_thunder mdio_cavium >> >> [ 116.206995] CPU: 25 PID: 454 Comm: kworker/25:1 Not tainted >> 4.11.0-rc3-next-20170323 #1 >> [ 116.206997] Hardware name: www.cavium.com crb-1s/crb-1s, BIOS 0.3 Feb 23 >> 2017 >> [ 116.207010] Workqueue: events netstamp_clear >> [ 116.207015] task: 801f906b5400 task.stack: 801f901a4000 >> [ 116.207020] PC is at smp_call_function_many+0x284/0x2e8 >> [ 116.207023] LR is at smp_call_function_many+0x244/0x2e8 >> [ 116.207026] pc : [] lr : [] >> pstate: 8145 >> [ 116.207028] sp : 801f901a7be0 >> [ 116.207030] x29: 801f901a7be0 x28: 09139000 >> [ 116.207036] x27: 09139434 x26: 0080 >> [ 116.207041] x25: x24: 081565d0 >> [ 116.207047] x23: 0001 x22: 08e11e00 >> [ 116.207052] x21: 801f6d5cff00 x20: 801f6d5cff08 >> [ 116.207057] x19: 09138e38 x18: 0a03 >> [ 116.207063] x17: b77c9028 x16: 082e81d8 >> [ 116.207068] x15: 3d0d6dd44d08 x14: 0036312196549b4a >> [ 116.207073] x13: 58dabe4c x12: 0018 >> [ 116.207079] x11: 366e2f04 x10: 09f0 >> [ 116.207084] x9 : 801f901a7d30 x8 : 0002 >> [ 116.207089] x7 : x6 : >> [ 116.207095] x5 : x4 : 0020 >> [ 116.207100] x3 : 0020 x2 : >> [ 116.207105] x1 : 801f6d682578 x0 : 0003 >> >> [ 150.443116] INFO: rcu_sched self-detected stall on CPU >> [ 150.448261] 25-...: (14997 ticks this GP) >> idle=47a/141/0 softirq=349/349 fqs=7495 >> [ 150.451115] INFO: rcu_sched detected stalls on CPUs/tasks: >> [ 150.451123] 25-...: (14997 ticks this GP) >> idle=47a/141/0 softirq=349/349 fqs=7495 >> [ 150.451124] (detected by 13, t=15002 jiffies, g=805, c=804, q=8384) >> [ 150.451136] Task dump for CPU 25: >> [ 150.451138] kworker/25:1R running task0 454 2 >> 0x0002 >> [ 150.451155] Workqueue: events netstamp_clear >> [ 150.451158] Call trace: >> [ 150.451164] [] __switch_to+0x90/0xa8 >> [ 150.451172] [] static_key_slow_inc+0x128/0x138 >> [ 150.451175] [] static_key_enable+0x34/0x60 >> [ 150.451178] [] netstamp_clear+0x68/0x80 >> [ 150.451181] [] process_one_work+0x158/0x478 >> [ 150.451183] [] worker_thread+0x50/0x4a8 >> [ 150.451187] [] kthread+0x108/0x138 >> [ 150.451190] [] ret_from_fork+0x10/0x50 >> [ 150.477451] (t=15008 jiffies g=805 c=804 q=8384) >> [ 150.482242] Task dump for CPU 25: >> [ 150.482245] kworker/25:1R running task0 454 2 >> 0x0002 >> [ 150.482259] Workqueue: events netstamp_clear >> [ 150.482264] Call trace: >> [ 150.482271] [] dump_backtrace+0x0/0x2b0 >> [ 150.482277] [] show_stack+0x24/0x30 >> [ 150.482281] [] sched_show_task+0x128/0x178 >> [ 150.482285] [] dump_cpu_task+0x48/0x58 >> [ 150.482288] [] rcu_dump_cpu_stacks+0xa0/0xe8 >> [ 150.482297] [] rcu_check_callbacks+0x774/0x938 >> [ 150.482305] [] update_process_times+0x34/0x60 >> [ 150.482314] [] tick_sched_handle.isra.7+0x38/0x70 >> [ 150.482319] [] tick_sched_timer+0x4c/0x98 >> [ 150.482324] [] __hrtimer_run_queues+0xd8/0x2b8 >> [ 150.482328] [] hrtimer_interrupt+0xa8/0x228 >> [ 150.482334] [] arch_timer_handler_phys+0x3c/0x50 >> [ 150.482341] [] handle_percpu_devid_irq+0x8c/0x230 >> [ 150.482344] [] generic_handle_irq+0x34/0x50 >> [ 150.482347] [] __handle_domain_irq+0x68/0xc0 >> [ 150.482351] []