hello again list, We seem to be having issues on more servers where according to the linux developers here: "the kernel is stuck in a spin lock during a disk operation."
The call traces are below, I see a lot of ocfs in the call traces, but I don't know how to read them, please tell me does the issue come from ocfs? thanks --Jim 2018-01-06T17:10:02.194362+00:00 node-115 kernel: [87885.155288] Modules linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath 2018-01-06T17:10:02.194364+00:00 node-115 kernel: [87885.157143] CPU: 15 PID: 11936 Comm: qemu-system-x86 Not tainted 4.4.0-98-generic #121-Ubuntu 2018-01-06T17:10:02.194366+00:00 node-115 kernel: [87885.157144] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017 2018-01-06T17:10:02.194367+00:00 node-115 kernel: [87885.157280] task: ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000 2018-01-06T17:10:02.194400+00:00 node-115 kernel: [87885.157281] RIP: 0010:[<ffffffff810cb27c>] [<ffffffff810cb27c>] native_queued_spin_lock_slowpath+0x15c/0x170 2018-01-06T17:10:02.194414+00:00 node-115 kernel: [87885.157566] RSP: 0018:ffff88203f143c30 EFLAGS: 00000202 2018-01-06T17:10:02.194416+00:00 node-115 kernel: [87885.157567] RAX: 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001 2018-01-06T17:10:02.194418+00:00 node-115 kernel: [87885.157705] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec 2018-01-06T17:10:02.194440+00:00 node-115 kernel: [87885.157705] RBP: ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7 2018-01-06T17:10:02.194442+00:00 node-115 kernel: [87885.157706] R10: ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec 2018-01-06T17:10:02.194443+00:00 node-115 kernel: [87885.157707] R13: 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157708] FS: 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000 2018-01-06T17:10:02.194444+00:00 node-115 kernel: [87885.157709] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2018-01-06T17:10:02.194445+00:00 node-115 kernel: [87885.157710] CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0 2018-01-06T17:10:02.194446+00:00 node-115 kernel: [87885.157711] Stack: 2018-01-06T17:10:02.194448+00:00 node-115 kernel: [87885.157712] ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535 2018-01-06T17:10:02.194449+00:00 node-115 kernel: [87885.157714] ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157716] ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800 2018-01-06T17:10:02.194450+00:00 node-115 kernel: [87885.157717] Call Trace: 2018-01-06T17:10:02.194451+00:00 node-115 kernel: [87885.157718] <IRQ> 2018-01-06T17:10:02.194453+00:00 node-115 kernel: [87885.157725] [<ffffffff81844421>] _raw_spin_lock+0x21/0x30 2018-01-06T17:10:02.194454+00:00 node-115 kernel: [87885.157727] [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50 2018-01-06T17:10:02.194456+00:00 node-115 kernel: [87885.157729] [<ffffffff8184257b>] mutex_unlock+0x1b/0x20 2018-01-06T17:10:02.194457+00:00 node-115 kernel: [87885.157766] [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2] 2018-01-06T17:10:02.194458+00:00 node-115 kernel: [87885.157770] [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0 2018-01-06T17:10:02.194460+00:00 node-115 kernel: [87885.157771] [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100 2018-01-06T17:10:02.194461+00:00 node-115 kernel: [87885.157774] [<ffffffff813c3edf>] bio_endio+0x3f/0x60 2018-01-06T17:10:02.194463+00:00 node-115 kernel: [87885.157777] [<ffffffff813cb897>] blk_update_request+0x87/0x310 2018-01-06T17:10:02.194464+00:00 node-115 kernel: [87885.157780] [<ffffffff816bbd66>] end_clone_bio+0x46/0x70 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157782] [<ffffffff813c3edf>] bio_endio+0x3f/0x60 2018-01-06T17:10:02.194465+00:00 node-115 kernel: [87885.157783] [<ffffffff813cb897>] blk_update_request+0x87/0x310 2018-01-06T17:10:02.194467+00:00 node-115 kernel: [87885.157786] [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0 2018-01-06T17:10:02.194468+00:00 node-115 kernel: [87885.157788] [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690 2018-01-06T17:10:02.194469+00:00 node-115 kernel: [87885.157792] [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0 2018-01-06T17:10:02.194470+00:00 node-115 kernel: [87885.157795] [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157796] [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150 2018-01-06T17:10:02.194471+00:00 node-115 kernel: [87885.157799] [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0 2018-01-06T17:10:02.194480+00:00 node-115 kernel: [87885.157803] [<ffffffff81085dc1>] __do_softirq+0x101/0x290 2018-01-06T17:10:02.194481+00:00 node-115 kernel: [87885.157805] [<ffffffff810860c3>] irq_exit+0xa3/0xb0 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157809] [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40 2018-01-06T17:10:02.194483+00:00 node-115 kernel: [87885.157811] [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157812] <EOI> 2018-01-06T17:10:02.194484+00:00 node-115 kernel: [87885.157814] [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30 2018-01-06T17:10:02.194486+00:00 node-115 kernel: [87885.157815] [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130 2018-01-06T17:10:02.194487+00:00 node-115 kernel: [87885.157829] [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2] 2018-01-06T17:10:02.194488+00:00 node-115 kernel: [87885.157831] [<ffffffff818424ff>] mutex_lock+0x1f/0x30 2018-01-06T17:10:02.194489+00:00 node-115 kernel: [87885.157843] [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2] 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157847] [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140 2018-01-06T17:10:02.194490+00:00 node-115 kernel: [87885.157858] [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2] 2018-01-06T17:10:02.194492+00:00 node-115 kernel: [87885.157862] [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0 2018-01-06T17:10:02.194493+00:00 node-115 kernel: [87885.157865] [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60 2018-01-06T17:10:02.194495+00:00 node-115 kernel: [87885.157867] [<ffffffff8122e933>] ? __fdget+0x13/0x20 2018-01-06T17:10:02.194496+00:00 node-115 kernel: [87885.157868] [<ffffffff812622cf>] do_io_submit+0x25f/0x500 2018-01-06T17:10:02.194497+00:00 node-115 kernel: [87885.157871] [<ffffffff81262580>] SyS_io_submit+0x10/0x20 2018-01-06T17:10:02.194499+00:00 node-115 kernel: [87885.157873] [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71 2018-01-06T17:10:02.194500+00:00 node-115 kernel: [87885.157874] Code: 01 48 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 <8b> 07 84 c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f 2018-01-06T17:10:30.192979+00:00 node-115 kernel: [87913.154413] Modules linked in: vhost_net vhost macvtap macvlan ip6table_raw xt_mac xt_tcpudp xt_physdev br_netfilter veth ebtable_filter ebtables openvswitch ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs ip6table_filter ip6_tables xt_multiport xt_conntrack iptable_filter xt_comment xt_CT iptable_raw ip_tables x_tables xfs bridge 8021q garp mrp stp llc intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul ipmi_ssif crc32_pclmul ghash_clmulni_intel kvm_intel aesni_intel aes_x86_64 kvm lrw gf128mul glue_helper ablk_helper irqbypass cryptd hpilo 8250_fintek serio_raw ioatdma ipmi_si sb_edac edac_core ipmi_msghandler shpchp dca acpi_power_meter lpc_ich mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nf_conntrack_proto_gre nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor dm_round_robin ses enclosure scsi_transport_sas raid6_pq libcrc32c raid1 raid0 multipath linear uas usb_storage psmouse lpfc be2net vxlan ip6_udp_tunnel scsi_transport_fc udp_tunnel wmi fjes scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath 2018-01-06T17:10:30.192984+00:00 node-115 kernel: [87913.155150] CPU: 15 PID: 11936 Comm: qemu-system-x86 Tainted: G L 4.4.0-98-generic #121-Ubuntu 2018-01-06T17:10:30.192987+00:00 node-115 kernel: [87913.155151] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 02/17/2017 2018-01-06T17:10:30.192988+00:00 node-115 kernel: [87913.155153] task: ffff882036ff0000 ti: ffff881f80ca0000 task.ti: ffff881f80ca0000 2018-01-06T17:10:30.192990+00:00 node-115 kernel: [87913.155154] RIP: 0010:[<ffffffff810cb27e>] [<ffffffff810cb27e>] native_queued_spin_lock_slowpath+0x15e/0x170 2018-01-06T17:10:30.192992+00:00 node-115 kernel: [87913.155160] RSP: 0018:ffff88203f143c30 EFLAGS: 00000202 2018-01-06T17:10:30.192994+00:00 node-115 kernel: [87913.155161] RAX: 0000000000000101 RBX: ffff8820046c83f0 RCX: 0000000000000001 2018-01-06T17:10:30.192996+00:00 node-115 kernel: [87913.155162] RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff8820046c83ec 2018-01-06T17:10:30.193019+00:00 node-115 kernel: [87913.155163] RBP: ffff88203f143c30 R08: 0000000000000101 R09: ffffffff811924a7 2018-01-06T17:10:30.193023+00:00 node-115 kernel: [87913.155164] R10: ffffea0040d6d680 R11: 0000000000000800 R12: ffff8820046c83ec 2018-01-06T17:10:30.193024+00:00 node-115 kernel: [87913.155165] R13: 0000000000000800 R14: 000000004c63ee00 R15: 0000000000000800 2018-01-06T17:10:30.193026+00:00 node-115 kernel: [87913.155166] FS: 00007fbcbb7eec00(0000) GS:ffff88203f140000(0000) knlGS:0000000000000000 2018-01-06T17:10:30.193028+00:00 node-115 kernel: [87913.155167] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2018-01-06T17:10:30.193030+00:00 node-115 kernel: [87913.155168] CR2: 00007f54266a8000 CR3: 0000000fcc2f2000 CR4: 00000000001426e0 2018-01-06T17:10:30.193032+00:00 node-115 kernel: [87913.155169] Stack: 2018-01-06T17:10:30.193034+00:00 node-115 kernel: [87913.155170] ffff88203f143c40 ffffffff81844421 ffff88203f143c60 ffffffff81842535 2018-01-06T17:10:30.193036+00:00 node-115 kernel: [87913.155172] ffff881e88a9ca80 ffff8820046c84b0 ffff88203f143c70 ffffffff8184257b 2018-01-06T17:10:30.193037+00:00 node-115 kernel: [87913.155173] ffff88203f143ca0 ffffffffc074158d ffff881e5d3beb80 0000000000000800 2018-01-06T17:10:30.193039+00:00 node-115 kernel: [87913.155175] Call Trace: 2018-01-06T17:10:30.193040+00:00 node-115 kernel: [87913.155176] <IRQ> 2018-01-06T17:10:30.193042+00:00 node-115 kernel: [87913.155183] [<ffffffff81844421>] _raw_spin_lock+0x21/0x30 2018-01-06T17:10:30.193044+00:00 node-115 kernel: [87913.155186] [<ffffffff81842535>] __mutex_unlock_slowpath+0x25/0x50 2018-01-06T17:10:30.193046+00:00 node-115 kernel: [87913.155187] [<ffffffff8184257b>] mutex_unlock+0x1b/0x20 2018-01-06T17:10:30.193047+00:00 node-115 kernel: [87913.155224] [<ffffffffc074158d>] ocfs2_dio_end_io+0x6d/0x80 [ocfs2] 2018-01-06T17:10:30.193049+00:00 node-115 kernel: [87913.155228] [<ffffffff8124e57c>] dio_complete+0x11c/0x1c0 2018-01-06T17:10:30.193051+00:00 node-115 kernel: [87913.155230] [<ffffffff8124e693>] dio_bio_end_aio+0x73/0x100 2018-01-06T17:10:30.193053+00:00 node-115 kernel: [87913.155233] [<ffffffff813c3edf>] bio_endio+0x3f/0x60 2018-01-06T17:10:30.193055+00:00 node-115 kernel: [87913.155235] [<ffffffff813cb897>] blk_update_request+0x87/0x310 2018-01-06T17:10:30.193058+00:00 node-115 kernel: [87913.155239] [<ffffffff816bbd66>] end_clone_bio+0x46/0x70 2018-01-06T17:10:30.193060+00:00 node-115 kernel: [87913.155240] [<ffffffff813c3edf>] bio_endio+0x3f/0x60 2018-01-06T17:10:30.193062+00:00 node-115 kernel: [87913.155242] [<ffffffff813cb897>] blk_update_request+0x87/0x310 2018-01-06T17:10:30.193064+00:00 node-115 kernel: [87913.155245] [<ffffffff815c52f3>] scsi_end_request+0x33/0x1d0 2018-01-06T17:10:30.193066+00:00 node-115 kernel: [87913.155247] [<ffffffff815c8a26>] scsi_io_completion+0x1b6/0x690 2018-01-06T17:10:30.193068+00:00 node-115 kernel: [87913.155251] [<ffffffff810beb46>] ? rebalance_domains+0x166/0x2d0 2018-01-06T17:10:30.193069+00:00 node-115 kernel: [87913.155254] [<ffffffff815bf64f>] scsi_finish_command+0xcf/0x120 2018-01-06T17:10:30.193070+00:00 node-115 kernel: [87913.155256] [<ffffffff815c81b4>] scsi_softirq_done+0x124/0x150 2018-01-06T17:10:30.193071+00:00 node-115 kernel: [87913.155258] [<ffffffff813d3787>] blk_done_softirq+0x87/0xb0 2018-01-06T17:10:30.193087+00:00 node-115 kernel: [87913.155263] [<ffffffff81085dc1>] __do_softirq+0x101/0x290 2018-01-06T17:10:30.193090+00:00 node-115 kernel: [87913.155265] [<ffffffff810860c3>] irq_exit+0xa3/0xb0 2018-01-06T17:10:30.193092+00:00 node-115 kernel: [87913.155269] [<ffffffff81050e93>] smp_call_function_single_interrupt+0x33/0x40 2018-01-06T17:10:30.193094+00:00 node-115 kernel: [87913.155270] [<ffffffff81845ae2>] call_function_single_interrupt+0x82/0x90 2018-01-06T17:10:30.193095+00:00 node-115 kernel: [87913.155271] <EOI> 2018-01-06T17:10:30.193096+00:00 node-115 kernel: [87913.155273] [<ffffffff81844414>] ? _raw_spin_lock+0x14/0x30 2018-01-06T17:10:30.193098+00:00 node-115 kernel: [87913.155275] [<ffffffff81842422>] __mutex_lock_slowpath+0x72/0x130 2018-01-06T17:10:30.193099+00:00 node-115 kernel: [87913.155289] [<ffffffffc0758099>] ? ocfs2_inode_unlock+0x119/0x120 [ocfs2] 2018-01-06T17:10:30.193101+00:00 node-115 kernel: [87913.155291] [<ffffffff818424ff>] mutex_lock+0x1f/0x30 2018-01-06T17:10:30.193102+00:00 node-115 kernel: [87913.155303] [<ffffffffc076177a>] ocfs2_file_write_iter+0x95a/0xdf0 [ocfs2] 2018-01-06T17:10:30.193104+00:00 node-115 kernel: [87913.155306] [<ffffffff812252c0>] ? poll_select_copy_remaining+0x140/0x140 2018-01-06T17:10:30.193105+00:00 node-115 kernel: [87913.155317] [<ffffffffc0760e20>] ? ocfs2_check_range_for_refcount+0x150/0x150 [ocfs2] 2018-01-06T17:10:30.193106+00:00 node-115 kernel: [87913.155321] [<ffffffff812613ea>] aio_run_iocb+0x26a/0x2d0 2018-01-06T17:10:30.193107+00:00 node-115 kernel: [87913.155324] [<ffffffff8122e8e5>] ? __fget_light+0x25/0x60 2018-01-06T17:10:30.193108+00:00 node-115 kernel: [87913.155325] [<ffffffff8122e933>] ? __fdget+0x13/0x20 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155327] [<ffffffff812622cf>] do_io_submit+0x25f/0x500 2018-01-06T17:10:30.193109+00:00 node-115 kernel: [87913.155329] [<ffffffff81262580>] SyS_io_submit+0x10/0x20 2018-01-06T17:10:30.193110+00:00 node-115 kernel: [87913.155331] [<ffffffff818446b2>] entry_SYSCALL_64_fastpath+0x16/0x71 2018-01-06T17:10:30.193111+00:00 node-115 kernel: [87913.155332] Code: 8b 02 48 85 c0 75 0a f3 90 48 8b 02 48 85 c0 74 f6 c7 40 08 01 00 00 00 e9 63 ff ff ff 83 fa 01 75 07 e9 c4 fe ff ff f3 90 8b 07 <84> c0 75 f8 b8 01 00 00 00 66 89 07 5d c3 0f 1f 40 00 0f 1f 44
_______________________________________________ Ocfs2-users mailing list Ocfs2-users@oss.oracle.com https://oss.oracle.com/mailman/listinfo/ocfs2-users