------- Comment From hasri...@in.ibm.com 2018-07-18 14:09 EDT------- Able to reproduce the issue with the latest proposed kernel on Tuleta machine. Seems the issue is not resolved with the provided patches.
[ 155.121412] Watchdog CPU:65 Hard LOCKUP [ 155.121416] Modules linked in: rmd256 rmd160 rmd128 vhost_net md4 vhost dccp_ipv4(+) sctp(+) unix_diag algif_hash tap dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter kvm_hv kvm binfmt_misc dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua uio_pdrv_genirq uio ipmi_powernv ipmi_devintf ipmi_msghandler powernv_rng ibmpowernv vmx_crypto leds_powernv powernv_op_panel crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq uas usb_storage crc32c_vpmsum ipr tg3 [ 155.121531] CPU: 65 PID: 7391 Comm: stress-ng-hrtim Not tainted 4.15.0-29-generic #31-Ubuntu [ 155.121537] NIP: c000000000d01444 LR: c0000000001269c8 CTR: 0000000000000000 [ 155.121544] REGS: c000000007c97d80 TRAP: 0900 Not tainted (4.15.0-29-generic) [ 155.121545] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28000448 XER: 00000000 [ 155.121560] CFAR: c0000000001269c4 SOFTE: 0 GPR00: c0000000001269c8 c000001fe581fc70 c0000000016eaf00 c000001fe57ad408 GPR04: c000001fe581fd30 0000000000000000 0000007cd2603a7c 0000007cd261c940 GPR08: 0000000000000021 0000000000000000 0000000080000041 0000000042000448 GPR12: 0000000000000001 c000000007a4cb00 0000000000000000 0000000000000000 GPR16: 00000000000186a0 00007fffd75e1ef0 0000007cd25c7040 00007fffd75e1ee8 GPR20: 0000007cd25c7048 00007fffd75e21ab ffffffffffffffff 00007fffd75e21aa GPR24: 0000000000010000 00007896e1ca0000 00007fffd75e1da8 00007fffd75e1ef8 GPR28: 0000000000000000 0000000000040002 c000001fe581fcf0 c000001fe57ad408 [ 155.121625] NIP [c000000000d01444] _raw_spin_lock_irq+0x44/0xe0 [ 155.121631] LR [c0000000001269c8] __set_current_blocked+0x48/0xb0 [ 155.121633] Call Trace: [ 155.121640] [c000001fe581fca0] [0000007cd25c7048] 0x7cd25c7048 [ 155.121648] [c000001fe581fcd0] [c000000000126ae4] signal_setup_done+0x84/0xe0 [ 155.121657] [c000001fe581fd10] [c00000000001da20] do_signal+0x110/0x2c0 [ 155.121665] [c000001fe581fe00] [c00000000001dd70] do_notify_resume+0xd0/0x100 [ 155.121672] [c000001fe581fe30] [c00000000000b8c4] ret_from_except_lite+0x70/0x74 [ 155.121673] Instruction dump: [ 155.121677] f821ffd1 7c7f1b78 39400000 892d028b 994d028b 39400000 994d028d 814d0008 [ 155.121692] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac 2fa90000 409e0010 # uname -a Linux lep8d 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:37:15 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Thanks, Harish -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1777194 Title: Hard LOCKUP observed on stressing Ubuntu 18 04 Status in The Ubuntu-power-systems project: In Progress Status in linux package in Ubuntu: In Progress Status in linux source package in Bionic: Fix Committed Bug description: == SRU Justification == IBM has seen hard lockups when running stress tests against Bionic. These test sometimes lead to rcu_stalls. IBM found that this bug is resolved with commits 6bed3237624e and ac61c1156623. == Fixes == 6bed3237624e ("powerpc: use NMI IPI for smp_send_stop") ac61c1156623 ("powerpc: Fix smp_send_stop NMI IPI handling") == Regression Potential == Low. Limited to powerpc. == Test Case == A test kernel was built with these patches and tested by the original bug reporter. The bug reporter states the test kernel resolved the bug. --Problem Description-- Hard LOCKUP on stressing Ubuntu 18 04 ---Issue observed--- Hard LOCKUP on stressing Ubuntu 18 04 using Ubuntu 18 04, sometimes leads to rcu_stalls. Apr 17 00:00:23 lep8d kernel: [ 4309.786755] Watchdog CPU:3 Hard LOCKUP Apr 17 00:00:23 lep8d kernel: [ 4309.786759] Modules linked in: algif_rng salsa20_generic userio camellia_generic cast6_generic cast_common snd_seq snd_seq_device snd_timer snd soundcore vhost_net serpent_generic tap twofish_generic twofish_common vhost_vsock vmw_vsock_virtio_transport_common vhost vsock lrw unix_diag algif_skcipher cuse sctp tgr192 wp512 rmd320 rmd256 rmd160 hci_vhci rmd128 bluetooth ecdh_generic dccp_ipv4 md4 uhid hid algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc uio_pdrv_genirq uio vmx_crypto ibmpowernv powernv_op_panel ipmi_powernv Apr 17 00:00:23 lep8d kernel: [ 4309.786899] ipmi_devintf ipmi_msghandler powernv_rng leds_powernv crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq uas usb_storage crc32c_vpmsum tg3 ipr Apr 17 00:00:23 lep8d kernel: [ 4309.786944] CPU: 3 PID: 28361 Comm: stress-ng-hrtim Not tainted 4.15.0-15-generic #16-Ubuntu Apr 17 00:00:23 lep8d kernel: [ 4309.786950] NIP: c000000000d0c8b8 LR: c000000000120dbc CTR: c000000000024480 Apr 17 00:00:23 lep8d kernel: [ 4309.786956] REGS: c000000007f7fd80 TRAP: 0900 Not tainted (4.15.0-15-generic) Apr 17 00:00:23 lep8d kernel: [ 4309.786957] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28000442 XER: 20000000 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] CFAR: c000000000120db8 SOFTE: 0 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR00: c000000000120dbc c000002d51377ba0 c0000000016eb400 c000002d512b0f88 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR04: 0000000000000000 0000000000000001 0000000001f40668 0000000000000001 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR08: 0000000000000001 0000000000000000 0000000080000003 0000000000000000 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR12: c000000000024480 c000000007a22100 0000000000000000 00000000000186a0 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR16: 00007fffed3687e0 00000abda98e5db8 0000000000008005 0000000000040100 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR20: 0000000000000000 00000000418004fc 00000000003c0000 0000000008430000 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR24: c000002d512b0f88 0000000000000000 c000002d51377d30 c000002d52d47c00 Apr 17 00:00:23 lep8d kernel: [ 4309.786972] GPR28: c000002d51377d50 c000002d51377d50 c000002d52f04500 c000002d512b0f88 Apr 17 00:00:23 lep8d kernel: [ 4309.787035] NIP [c000000000d0c8b8] _raw_spin_lock+0x38/0xe0 Apr 17 00:00:23 lep8d kernel: [ 4309.787045] LR [c000000000120dbc] dequeue_signal+0xcc/0x260 Apr 17 00:00:23 lep8d kernel: [ 4309.787046] Call Trace: Apr 17 00:00:23 lep8d kernel: [ 4309.787052] [c000002d51377bd0] [c000000000120dac] dequeue_signal+0xbc/0x260 Apr 17 00:00:23 lep8d kernel: [ 4309.787059] [c000002d51377c20] [c00000000012459c] get_signal+0x13c/0x7a0 Apr 17 00:00:23 lep8d kernel: [ 4309.787066] [c000002d51377d10] [c00000000001dacc] do_signal+0x7c/0x2c0 Apr 17 00:00:23 lep8d kernel: [ 4309.787072] [c000002d51377e00] [c00000000001deb0] do_notify_resume+0xd0/0x100 Apr 17 00:00:23 lep8d kernel: [ 4309.787083] [c000002d51377e30] [c00000000000b7c4] ret_from_except_lite+0x70/0x74 Apr 17 00:00:23 lep8d kernel: [ 4309.787085] Instruction dump: Apr 17 00:00:23 lep8d kernel: [ 4309.787090] 7c0802a6 60000000 fbe1fff8 f821ffd1 7c7f1b78 39400000 994d028c 814d0008 Apr 17 00:00:23 lep8d kernel: [ 4309.787102] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac 2fa90000 409e001c Apr 17 00:00:23 lep8d kernel: [ 4313.015781] kauditd_printk_skb: 13 callbacks suppressed ---uname output--- # uname -a Linux lep8d 4.15.0-15-generic #16-Ubuntu SMP Wed Apr 4 13:57:51 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Machine Type = Power 8 BML/Tuleta ----Additional Info----- Hard LOCKUP is also seen on garri BML. syslog is attached. Reproducible : 90% ---Steps to Reproduce--- 1. wget https://github.com/ColinIanKing/stress-ng/archive/master.zip 2. unzip master.zip; cd stress-ng-master; 3. make; make install; 4. Run the following command multiple times stress-ng --all <nr_cpus> --vm-bytes 80% --aggressive --maximize --oomable --timeout 300 --verify --syslog --metrics --times Issue is observed on Power 9 BML machines as well. [Tue Apr 17 04:13:42 2018] Watchdog CPU:37 Hard LOCKUP [Tue Apr 17 04:13:42 2018] Modules linked in: vsock lrw algif_skcipher tgr192 wp512 rmd320 rmd256 hci_vhci unix_diag bluetooth rmd160 sctp rmd128 ecdh_generic md4 dccp_ipv4 algif_hash cuse dccp af_alg vhost_net vhost tap xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc kvm_hv kvm dm_crypt dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua idt_89hpesx joydev input_leds mac_hid ofpart cmdlinepart vmx_crypto ipmi_powernv ipmi_devintf at24 uio_pdrv_genirq ibmpowernv opal_prd crct10dif_vpmsum powernv_flash ipmi_msghandler mtd uio sch_fq_codel ib_iser rdma_cm iw_cm ib_cm [Tue Apr 17 04:13:42 2018] ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi jc42 ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq ses enclosure scsi_transport_sas hid_generic ast i2c_algo_bit ttm drm_kms_helper usbhid hid syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e aacraid [Tue Apr 17 04:13:42 2018] CPU: 37 PID: 11524 Comm: stress-ng-hrtim Not tainted 4.15.0-15-generic #16-Ubuntu [Tue Apr 17 04:13:42 2018] NIP: c00000000012058c LR: c000000000120554 CTR: c00000000002bd30 [Tue Apr 17 04:13:42 2018] REGS: c000000007debd80 TRAP: 0900 Not tainted (4.15.0-15-generic) [Tue Apr 17 04:13:42 2018] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 22000442 XER: 00000000 [Tue Apr 17 04:13:42 2018] CFAR: c00000000011f8c4 SOFTE: 0 GPR00: c000000000120554 c0000004cfc7bd10 c0000000016eb400 c0000004cfb54200 GPR04: c0000004cfc7be00 0000000042000442 0000000000007338 0000000000040000 GPR08: c0000004cfc78080 c0000004cfc78000 0000000000000002 c000000000d10f78 GPR12: c00000000002bd30 c000000007a39700 0000000000000000 00000000000186a0 GPR16: 00007fffd2cab1d0 00000edac67c5db8 00007fffd2cab1c8 00000edac67c5dc0 GPR20: 00007fffd2cab1cc 00007fffd2cab48b ffffffffffffffff 00007fffd2cab48a GPR24: 0000000000010000 000072164cd80000 00007fffd2cab0c4 00007fffd2cab1d8 GPR28: 0000000000000000 c0000004cfc7be00 c0000004cfc7be00 c0000004cfb54200 [Tue Apr 17 04:13:42 2018] NIP [c00000000012058c] recalc_sigpending+0x5c/0x90 [Tue Apr 17 04:13:42 2018] LR [c000000000120554] recalc_sigpending+0x24/0x90 [Tue Apr 17 04:13:42 2018] Call Trace: [Tue Apr 17 04:13:42 2018] [c0000004cfc7bd10] [c00000000001db60] do_signal+0x110/0x2c0 (unreliable) [Tue Apr 17 04:13:42 2018] [c0000004cfc7bd30] [c000000000121658] __set_task_blocked+0x48/0x90 [Tue Apr 17 04:13:42 2018] [c0000004cfc7bd70] [c000000000124ed8] __set_current_blocked+0x58/0xb0 [Tue Apr 17 04:13:42 2018] [c0000004cfc7bda0] [c00000000002be18] sys_rt_sigreturn+0xe8/0x270 [Tue Apr 17 04:13:42 2018] [c0000004cfc7be30] [c00000000000b184] system_call+0x58/0x6c [Tue Apr 17 04:13:42 2018] Instruction dump: [Tue Apr 17 04:13:42 2018] e86d0260 3d220020 3929def8 81290000 2f890000 409e0030 78290464 39400002 [Tue Apr 17 04:13:42 2018] 39090080 7ce040a8 7ce75078 7ce041ad <40c2fff4> 38210020 e8010010 7c0803a6 Continuous lockups are observed. - Harish == == Hardware: P9 Boston/ P8 Tuleta DD revision: P9 DD2.2 Operating Env.: BML PNOR: version-SUPERMICRO-P9DSU-V1.10-20180413-imp Host OS: Ubuntu 18.04 ==== (In reply to comment #5) > hi Harish, > > Does this bug happen if you set powersave=off? I am wondering if this > problem might be related to the stop state issue. Following Issue is seen with powersave=off. This is on Power 8 BML. [ 517.480153] Modules linked in: salsa20_generic(+) camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid vhost_net tap sctp unix_diag userio rmd128 vhost_vsock vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto uio_pdrv_genirq ipmi_msghandler [ 517.480305] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage [ 517.480343] CPU: 73 PID: 4388 Comm: stress-ng-dev Not tainted 4.15.0-15-generic #16-Ubuntu [ 517.480347] NIP: c00000000000a724 LR: c000000000016e74 CTR: 0000000030061154 [ 517.480352] REGS: c000001fef1db890 TRAP: 0901 Not tainted (4.15.0-15-generic) [ 517.480353] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48002224 XER: 20000000 [ 517.480368] CFAR: c000000000d0c8fc SOFTE: 1 [ 517.480368] GPR00: c00000000018c5bc c000001fef1dbb10 c0000000016eb400 0000000000000500 [ 517.480368] GPR04: 0000000000000000 c0000000000a2608 9000000000001033 0000000000000004 [ 517.480368] GPR08: c000000007a52300 0000000000000000 0000000080000049 9000000000001003 [ 517.480368] GPR12: 0000000000000040 c000000007a52300 [ 517.480406] NIP [c00000000000a724] replay_interrupt_return+0x0/0x4 [ 517.480411] LR [c000000000016e74] arch_local_irq_restore+0x74/0x90 [ 517.480412] Call Trace: [ 517.480420] [c000001fef1dbb10] [c0000000018bd940] log_first_seq+0x0/0x8 (unreliable) [ 517.480427] [c000001fef1dbb30] [c00000000018c5bc] console_unlock+0x2fc/0x6c0 [ 517.480432] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420 [ 517.480437] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0 [ 517.480442] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54 [ 517.480455] [c000001fef1dbcd0] [d00000001efc2f20] vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock] [ 517.480462] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00 [ 517.480467] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130 [ 517.480473] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c [ 517.480475] Instruction dump: [ 517.480480] 7d8000a6 e9628008 7d200026 618c8000 2c030900 4182e7f8 2c030500 4182e310 [ 517.480491] 2c030a00 4182ffa4 2c030e60 4182f090 <4e800020> 7c781b78 48000359 48000371 which leads to the Hard LOCKUP and rcu_stalls. [ 629.383369] Watchdog CPU:73 Hard LOCKUP [ 629.383372] Modules linked in: salsa20_generic(+) camellia_generic cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid vhost_net tap sctp unix_diag userio rmd128 vhost_vsock vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto uio_pdrv_genirq ipmi_msghandler [ 629.383451] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage [ 629.383475] CPU: 73 PID: 4388 Comm: stress-ng-dev Tainted: G L 4.15.0-15-generic #16-Ubuntu [ 629.383478] NIP: c0000000000a259c LR: c00000000009df5c CTR: 0000000030036830 [ 629.383481] REGS: c000000007c37d80 TRAP: 0900 Tainted: G L (4.15.0-15-generic) [ 629.383482] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48002222 XER: 20000000 [ 629.383494] CFAR: c00000000009df48 SOFTE: 0 [ 629.383497] GPR00: 0000000030005128 c000001fef1dba30 c0000000016eb400 0000000000000000 [ 629.383504] GPR04: 0000000048002222 c0000000000a259c 9000000000009033 00000000000000f1 [ 629.383510] GPR08: 0000000000000000 00000000300b0218 c00000000009df70 9000000000001003 [ 629.383517] GPR12: c00000000009df48 c000000007a52300 0000714a9655a560 0000000000000000 [ 629.383524] GPR16: 0000000000000027 0000000000000027 c000000001572a00 0000000000000000 [ 629.383530] GPR20: 20c49ba5e353f7cf 0000000000000000 0000000000000017 000000000000000d [ 629.383537] GPR24: fffffffffffffff5 0000000000000000 0000000000000010 c0000000018a8d18 [ 629.383544] GPR28: 0000000000000000 0000000000000010 c000001fef1dbad0 0000000000000010 [ 629.383551] NIP [c0000000000a259c] opal_put_chars+0x19c/0x280 [ 629.383553] LR [c00000000009df5c] opal_return+0x14/0x48 [ 629.383554] Call Trace: [ 629.383556] [c000001fef1dba30] [c0000000000a259c] opal_put_chars+0x19c/0x280 (unreliable) [ 629.383561] [c000001fef1dbab0] [c0000000008089b0] hvc_console_print+0xd0/0x210 [ 629.383564] [c000001fef1dbb30] [c00000000018c59c] console_unlock+0x2dc/0x6c0 [ 629.383568] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420 [ 629.383571] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0 [ 629.383574] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54 [ 629.383577] [c000001fef1dbcd0] [d00000001efc2f20] vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock] [ 629.383581] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00 [ 629.383584] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130 [ 629.383587] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c [ 629.383589] Instruction dump: [ 629.383591] 7f03c378 38210080 eb01ffc0 eb61ffd8 4e800020 7f63db78 7f24cb78 48c6a5e1 [ 629.383600] 60000000 38600000 3b00fff5 4bffc01d <60000000> e8010090 eb210048 eb810060 [ 638.478556] INFO: rcu_sched detected stalls on CPUs/tasks: [ 638.478581] 73-....: (149 ticks this GP) idle=0aa/140000000000000/0 softirq=14950/14950 fqs=15015 [ 638.478586] (detected by 83, t=37345 jiffies, g=2733, c=2732, q=9230463) [ 638.478645] Sending NMI from CPU 83 to CPUs 73: So, actual issue might be with powersave. These issue are due to huge data dumped to console. - Harish == Harish Sriram <hasri...@in.ibm.com> == With powersave=off on a P9 DD 2.2 Boston LC, the following is observed. [ 119.200587] Watchdog CPU:60 Hard LOCKUP [ 119.204932] Watchdog CPU:48 Hard LOCKUP [ 119.207911] Watchdog CPU:52 Hard LOCKUP [ 119.208454] Watchdog CPU:46 Hard LOCKUP .... .... syslog will be attached. - Harish == MAHESH J. SALGAONKAR <mahesh.salgaon...@in.ibm.com> == > > hi Harish, > > > > Does this bug happen if you set powersave=off? I am wondering if this > > problem might be related to the stop state issue. > > Following Issue is seen with powersave=off. This is on Power 8 BML. > > [ 517.480153] Modules linked in: salsa20_generic(+) camellia_generic > cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw > algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 > ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid > vhost_net tap sctp unix_diag userio rmd128 vhost_vsock > vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp > af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 > iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack > nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc > devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter > kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto > uio_pdrv_genirq ipmi_msghandler > [ 517.480305] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel > sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas > btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage > [ 517.480343] CPU: 73 PID: 4388 Comm: stress-ng-dev Not tainted > 4.15.0-15-generic #16-Ubuntu > [ 517.480347] NIP: c00000000000a724 LR: c000000000016e74 CTR: > 0000000030061154 > [ 517.480352] REGS: c000001fef1db890 TRAP: 0901 Not tainted > (4.15.0-15-generic) > [ 517.480353] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: > 48002224 XER: 20000000 > [ 517.480368] CFAR: c000000000d0c8fc SOFTE: 1 > [ 517.480368] GPR00: c00000000018c5bc c000001fef1dbb10 c0000000016eb400 > 0000000000000500 > [ 517.480368] GPR04: 0000000000000000 c0000000000a2608 9000000000001033 > 0000000000000004 > [ 517.480368] GPR08: c000000007a52300 0000000000000000 0000000080000049 > 9000000000001003 > [ 517.480368] GPR12: 0000000000000040 c000000007a52300 > [ 517.480406] NIP [c00000000000a724] replay_interrupt_return+0x0/0x4 > [ 517.480411] LR [c000000000016e74] arch_local_irq_restore+0x74/0x90 > [ 517.480412] Call Trace: > [ 517.480420] [c000001fef1dbb10] [c0000000018bd940] log_first_seq+0x0/0x8 > (unreliable) > [ 517.480427] [c000001fef1dbb30] [c00000000018c5bc] > console_unlock+0x2fc/0x6c0 > [ 517.480432] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420 > [ 517.480437] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0 > [ 517.480442] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54 > [ 517.480455] [c000001fef1dbcd0] [d00000001efc2f20] > vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock] > [ 517.480462] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00 > [ 517.480467] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130 > [ 517.480473] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c > [ 517.480475] Instruction dump: > [ 517.480480] 7d8000a6 e9628008 7d200026 618c8000 2c030900 4182e7f8 > 2c030500 4182e310 > [ 517.480491] 2c030a00 4182ffa4 2c030e60 4182f090 <4e800020> 7c781b78 > 48000359 48000371 > > which leads to the Hard LOCKUP and rcu_stalls. > > [ 629.383369] Watchdog CPU:73 Hard LOCKUP > [ 629.383372] Modules linked in: salsa20_generic(+) camellia_generic > cast6_generic cast_common serpent_generic twofish_generic twofish_common lrw > algif_skcipher cuse hci_vhci bluetooth snd_seq snd_seq_device tgr192 > ecdh_generic snd_timer snd wp512 soundcore rmd320 rmd256 rmd160 uhid hid > vhost_net tap sctp unix_diag userio rmd128 vhost_vsock > vmw_vsock_virtio_transport_common md4 dccp_ipv4 vhost vsock algif_hash dccp > af_alg xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 > iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack > nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc > devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter > kvm_hv kvm binfmt_misc ipmi_powernv leds_powernv ipmi_devintf vmx_crypto > uio_pdrv_genirq ipmi_msghandler > [ 629.383451] powernv_rng crct10dif_vpmsum uio ibmpowernv powernv_op_panel > sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas > btrfs xor zstd_compress raid6_pq tg3 uas crc32c_vpmsum ipr usb_storage > [ 629.383475] CPU: 73 PID: 4388 Comm: stress-ng-dev Tainted: G > L 4.15.0-15-generic #16-Ubuntu > [ 629.383478] NIP: c0000000000a259c LR: c00000000009df5c CTR: > 0000000030036830 > [ 629.383481] REGS: c000000007c37d80 TRAP: 0900 Tainted: G L > (4.15.0-15-generic) > [ 629.383482] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: > 48002222 XER: 20000000 > [ 629.383494] CFAR: c00000000009df48 SOFTE: 0 > [ 629.383497] GPR00: 0000000030005128 c000001fef1dba30 c0000000016eb400 > 0000000000000000 > [ 629.383504] GPR04: 0000000048002222 c0000000000a259c 9000000000009033 > 00000000000000f1 > [ 629.383510] GPR08: 0000000000000000 00000000300b0218 c00000000009df70 > 9000000000001003 > [ 629.383517] GPR12: c00000000009df48 c000000007a52300 0000714a9655a560 > 0000000000000000 > [ 629.383524] GPR16: 0000000000000027 0000000000000027 c000000001572a00 > 0000000000000000 > [ 629.383530] GPR20: 20c49ba5e353f7cf 0000000000000000 0000000000000017 > 000000000000000d > [ 629.383537] GPR24: fffffffffffffff5 0000000000000000 0000000000000010 > c0000000018a8d18 > [ 629.383544] GPR28: 0000000000000000 0000000000000010 c000001fef1dbad0 > 0000000000000010 > [ 629.383551] NIP [c0000000000a259c] opal_put_chars+0x19c/0x280 > [ 629.383553] LR [c00000000009df5c] opal_return+0x14/0x48 > [ 629.383554] Call Trace: > [ 629.383556] [c000001fef1dba30] [c0000000000a259c] > opal_put_chars+0x19c/0x280 (unreliable) > [ 629.383561] [c000001fef1dbab0] [c0000000008089b0] > hvc_console_print+0xd0/0x210 > [ 629.383564] [c000001fef1dbb30] [c00000000018c59c] > console_unlock+0x2dc/0x6c0 > [ 629.383568] [c000001fef1dbc20] [c00000000018ccec] vprintk_emit+0x36c/0x420 > [ 629.383571] [c000001fef1dbc90] [c00000000018ec54] vprintk_func+0x64/0xf0 > [ 629.383574] [c000001fef1dbcb0] [c00000000018e354] printk+0x40/0x54 > [ 629.383577] [c000001fef1dbcd0] [d00000001efc2f20] > vsock_dev_do_ioctl.isra.4+0xb8/0xe0 [vsock] > [ 629.383581] [c000001fef1dbd40] [c0000000003efc34] do_vfs_ioctl+0xd4/0xa00 > [ 629.383584] [c000001fef1dbde0] [c0000000003f0624] SyS_ioctl+0xc4/0x130 > [ 629.383587] [c000001fef1dbe30] [c00000000000b184] system_call+0x58/0x6c > [ 629.383589] Instruction dump: > [ 629.383591] 7f03c378 38210080 eb01ffc0 eb61ffd8 4e800020 7f63db78 > 7f24cb78 48c6a5e1 > [ 629.383600] 60000000 38600000 3b00fff5 4bffc01d <60000000> e8010090 > eb210048 eb810060 > [ 638.478556] INFO: rcu_sched detected stalls on CPUs/tasks: > [ 638.478581] 73-....: (149 ticks this GP) idle=0aa/140000000000000/0 > softirq=14950/14950 fqs=15015 > [ 638.478586] (detected by 83, t=37345 jiffies, g=2733, c=2732, > q=9230463) > [ 638.478645] Sending NMI from CPU 83 to CPUs 73: > > So, actual issue might be with powersave. These issue are due to huge data > dumped to console. > > - Harish == Breno Leitao <bren...@br.ibm.com> == > I think we probably need below upstream commit: > > commit 15b4dd7981496f51c5f9262a5e0761e48de6655f > Author: Nicholas Piggin <npig...@gmail.com> > Date: Tue Mar 27 01:01:03 2018 +1000 > > powerpc/64s: return more carefully from sreset NMI This patch apply cleanly on top of bionic kernel, but we definitely need to test it before sending to canonical for inclusion. Holding the mirror until we have an ack that this is the patch we need for this issue. == Gustavo Luiz Ferreira Walbon <gwal...@br.ibm.com> == Breno, Two patches are missing, 6bed3237624e3faad1592543952907cd01a42c83 powerpc: use NMI IPI for smp_send_stop ac61c1156623455c46701654abd8c99720bceea1 powerpc: Fix smp_send_stop NMI IPI handling I have rebased the tree with latest proposed kernel and built the packages again : http://pokgsa.ibm.com/gsa/pokgsa/home/g/w/gwalbon/web/public/Bug166791/ Patches : https://github.com/walbon/ubuntu-bionic/commits/bug166791 == Harish Sriram <hasri...@in.ibm.com>== > > Hard Lockups are observed with stressing the 18.04.1 kernel. > > > > # uname -a > > Linux ltc-wspoon12 4.15.0-23-generic #25-Ubuntu SMP Wed May 23 17:59:00 UTC > > 2018 ppc64le ppc64le ppc64le GNU/Linux > > > > [ 5295.047807] Watchdog CPU:64 Hard LOCKUP > > [ 5295.047817] Modules linked in: snd_seq snd_seq_device snd_timer snd > > soundcore twofish_generic twofish_common lrw uhid hid algif_skcipher > > vhost_net tgr192 vhost tap wp512 hci_vhci bluetooth rmd320 ecdh_generic > > rmd256 cuse rmd160 rmd128 btrfs zstd_compress md4 algif_hash xor unix_diag > > binfmt_misc raid6_pq sctp af_alg userio dccp_ipv4 dccp xt_CHECKSUM > > iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 > > nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c > > ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables > > ip6table_filter ip6_tables iptable_filter idt_89hpesx at24 ofpart > > uio_pdrv_genirq ipmi_powernv ipmi_devintf ipmi_msghandler opal_prd uio > > cmdlinepart powernv_flash mtd ibmpowernv vmx_crypto crct10dif_vpmsum kvm_hv > > kvm sch_fq_codel > > [ 5295.047928] ip_tables x_tables autofs4 mlx5_ib ib_core nouveau ast > > i2c_algo_bit ttm mlx5_core drm_kms_helper syscopyarea sysfillrect sysimgblt > > fb_sys_fops ahci mlxfw crc32c_vpmsum drm tg3 libahci devlink > > [ 5295.047956] CPU: 64 PID: 15874 Comm: stress-ng-hrtim Not tainted > > 4.15.0-23-generic #25-Ubuntu > > [ 5295.047958] NIP: c000000000cffc04 LR: c000000000126468 CTR: > > 0000000000000000 > > [ 5295.047962] REGS: c00000003fcffd80 TRAP: 0900 Not tainted > > (4.15.0-23-generic) > > [ 5295.047963] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: > > 28000448 XER: 00000000 > > [ 5295.047973] CFAR: c000000000126464 SOFTE: 0 > > GPR00: c000000000126468 c0002036131f7c70 c0000000016eaf00 > > c0002038f4c4b208 > > GPR04: c0002036131f7d30 0000000000000000 00000ebedb3a39a4 > > 00000ebedb3bc840 > > GPR08: 0000000000000021 0000000000000000 0000000080000040 > > 0000000052000442 > > GPR12: 0000000000000001 c00000000faac000 0000000000000000 > > 00000000000186a0 > > GPR16: 00007ffff15aaec0 00000ebedb366800 00007ffff15aaeb8 > > 00000ebedb366808 > > GPR20: 00007ffff15aaebc 00007ffff15ab17b ffffffffffffffff > > 00007ffff15ab17a > > GPR24: 0000000000010000 00007ab613340000 00007ffff15aad68 > > 00007ffff15aaec8 > > GPR28: 0000000000000000 0000000000040002 c0002036131f7cf0 > > c0002038f4c4b208 > > [ 5295.048016] NIP [c000000000cffc04] _raw_spin_lock_irq+0x44/0xe0 > > [ 5295.048021] LR [c000000000126468] __set_current_blocked+0x48/0xb0 > > [ 5295.048022] Call Trace: > > [ 5295.048027] [c0002036131f7ca0] [00007ffff15aaebc] 0x7ffff15aaebc > > [ 5295.048031] [c0002036131f7cd0] [c000000000126584] > > signal_setup_done+0x84/0xe0 > > [ 5295.048036] [c0002036131f7d10] [c00000000001dc60] do_signal+0x110/0x2c0 > > [ 5295.048040] [c0002036131f7e00] [c00000000001dfb0] > > do_notify_resume+0xd0/0x100 > > [ 5295.048045] [c0002036131f7e30] [c00000000000b8c4] > > ret_from_except_lite+0x70/0x74 > > [ 5295.048046] Instruction dump: > > [ 5295.048050] f821ffd1 7c7f1b78 39400000 892d028b 994d028b 39400000 > > 994d028d 814d0008 > > [ 5295.048074] 7d201829 2c090000 40c20010 7d40192d <40c2fff0> 7c2004ac > > 2fa90000 409e0010 > > [ 5295.272204] Watchdog CPU:64 became unstuck > > > > This is on WSP DD 2.2. > > PNOR: > > Version of System Firmware : > > Product Name : OpenPOWER Firmware > > Product Extra : version-witherspoon-ibm-OP9-v2.0-2.14 > > Product Extra : occ-77bb5e6 > > Product Extra : skiboot-v6.0.1 > > Product Extra : buildroot-2018.02.1-6-ga8d1126 > > It would be interesting to run an irqsoff latency tracer with this stress > test. It might require kernel recompile to add the irqsoff tracer option. > > Where is this hrtimer stress test from? This test from stress ng. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1777194/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
