Hi all, The version for TIPC which we use is TIPC version 2.0.0. The OS info is as follow:
Red Hat Enterprise Linux Server 7.2 (Maipo) Kernel 3.10.0-327.18.2.el7.x86_64 on an x86_64 We meet two soft lockup issue about TIPC, please help us to solve this issue. Thank you One issue is as follow: [85502.601198] BUG: soft lockup - CPU#0 stuck for 22s! [scm:2649] [85502.603585] Modules linked in: iptable_filter ip6table_mangle xt_limit iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [85502.618210] CPU: 0 PID: 2649 Comm: scm Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [85502.620482] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [85502.622361] task: ffff880405da5080 ti: ffff8800db68c000 task.ti: ffff8800db68c000 [85502.624400] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [85502.626558] RSP: 0018:ffff8800db68fa68 EFLAGS: 00000202 [85502.628412] RAX: 00000000000056f6 RBX: ffff880406600800 RCX: 000000000000712a [85502.630419] RDX: 000000000000712c RSI: 000000000000712c RDI: ffffffffa03f8548 [85502.632423] RBP: ffff8800db68fa70 R08: 00000000000002c0 R09: 0000000000000500 [85502.634421] R10: ffff88040f001500 R11: ffff8800db68fc58 R12: ffffffff81519be1 [85502.636413] R13: ffff8800db68fa08 R14: 00000000ffffffff R15: 0000000000000240 [85502.638415] FS: 0000000000000000(0000) GS:ffff88041fc00000(0063) knlGS:00000000d1495b40 [85502.640514] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 [85502.642399] CR2: 00000000d37c8208 CR3: 00000003cfff7000 CR4: 00000000000406f0 [85502.644420] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [85502.646416] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [85502.648393] Stack: [85502.649916] ffffffffa03f8548 ffff8800db68fa90 ffffffffa03e2afb ffff880036242c00 [85502.652019] ffff880406600800 ffff8800db68fac0 ffffffffa03e5c1b ffff880036242c00 [85502.654108] ffff880036b5a484 ffff88003698e800 ffff8800db68fb30 ffff8800db68fba8 [85502.656179] Call Trace: [85502.657742] [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [85502.659678] [<ffffffffa03e5c1b>] link_send_buf_fast+0x5b/0xb0 [tipc] [85502.661592] [<ffffffffa03e6026>] tipc_link_send_sections_fast+0xe6/0x630 [tipc] [85502.663594] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20 [85502.665473] [<ffffffffa03eb9b0>] ? tipc_nametbl_translate+0xc0/0x1f0 [tipc] [85502.667441] [<ffffffffa03ef5d5>] tipc_send2name+0x155/0x1d0 [tipc] [85502.669351] [<ffffffffa03f08cb>] send_msg+0x1eb/0x530 [tipc] [85502.671205] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0 [85502.673028] [<ffffffff810e31f3>] ? futex_wait+0x193/0x280 [85502.674859] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0 [85502.676676] [<ffffffff8164169d>] ? __do_page_fault+0x16d/0x450 [85502.678540] [<ffffffff812346d2>] ? poll_select_copy_remaining+0x62/0x130 [85502.680489] [<ffffffff815121ce>] SyS_sendto+0xe/0x10 [85502.682277] [<ffffffff8154be63>] compat_sys_socketcall+0x173/0x2a0 [85502.684193] [<ffffffff8164800c>] sysenter_dispatch+0x7/0x21 [85502.686046] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [85503.372189] BUG: soft lockup - CPU#2 stuck for 23s! [nodemgr:2560] [85503.374181] Modules linked in: iptable_filter ip6table_mangle xt_limit iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [85503.389354] CPU: 2 PID: 2560 Comm: nodemgr Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [85503.391715] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [85503.393668] task: ffff880405da6780 ti: ffff8803cfdd4000 task.ti: ffff8803cfdd4000 [85503.395749] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [85503.397949] RSP: 0018:ffff8803cfdd7a40 EFLAGS: 00000202 [85503.399809] RAX: 00000000000000f7 RBX: ffffffff8152c617 RCX: 000000000000712a [85503.401838] RDX: 000000000000712e RSI: 000000000000712e RDI: ffffffffa03f8548 [85503.403857] RBP: ffff8803cfdd7a48 R08: 000000000003f1e4 R09: 0000000000000500 [85503.405861] R10: ffff88040f001500 R11: ffff8803cfdd7c98 R12: ffff8803cfdd7a20 [85503.407861] R13: ffffffff81a70408 R14: ffff8803cfc44e00 R15: ffff88040900e000 [85503.409857] FS: 00007f2b09eaf740(0000) GS:ffff88041fd00000(0000) knlGS:0000000000000000 [85503.411951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [85503.413849] CR2: 00000000d2496120 CR3: 00000003cf432000 CR4: 00000000000406e0 [85503.415897] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [85503.417920] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [85503.419901] Stack: [85503.421402] ffffffffa03f8548 ffff8803cfdd7a68 ffffffffa03e2afb ffffffffa03f7ed8 [85503.423506] ffff8803b4968800 ffff8803cfdd7ab8 ffffffffa03e1270 ffffffff8151aaed [85503.425596] 0000000000000004 00000000cfdd7a98 ffffffffa03f76a0 ffff8803cfce5860 [85503.427703] Call Trace: [85503.429261] [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [85503.431225] [<ffffffffa03e1270>] tipc_bcbearer_send+0xb0/0x270 [tipc] [85503.433182] [<ffffffff8151aaed>] ? __alloc_skb+0x8d/0x2d0 [85503.435031] [<ffffffffa03e5943>] tipc_link_send_buf+0xc3/0x2a0 [tipc] [85503.436985] [<ffffffff81518c00>] ? skb_copy_bits+0x60/0x290 [85503.438857] [<ffffffff8151ab01>] ? __alloc_skb+0xa1/0x2d0 [85503.440698] [<ffffffffa03e1934>] tipc_bclink_send_msg+0x34/0x90 [tipc] [85503.442641] [<ffffffffa03ee9f2>] tipc_multicast+0x132/0x1f0 [tipc] [85503.444539] [<ffffffffa03f0872>] send_msg+0x192/0x530 [tipc] [85503.446377] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0 [85503.448187] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0 [85503.450014] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900 [85503.451831] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0 [85503.453646] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20 [85503.455470] [<ffffffff815121ce>] SyS_sendto+0xe/0x10 [85503.457239] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b [85503.459098] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [85505.146170] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:32669] [85505.150300] Modules linked in: iptable_filter ip6table_mangle xt_limit iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache[85505.160168] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] [85505.160168] Modules linked in: iptable_filter ip6table_mangle xt_limit iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [85505.160193] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [85505.160193] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [85505.160194] task: ffff88040957e780 ti: ffff8804095c8000 task.ti: ffff8804095c8000 [85505.160195] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [85505.160200] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000216 [85505.160200] RAX: 0000000000003a59 RBX: ffff88041fd83d30 RCX: 000000000000712a [85505.160201] RDX: 0000000000007130 RSI: 0000000000007130 RDI: ffffffffa03f8548 [85505.160202] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09: 0000000000000300 [85505.160202] R10: ffff88040f001600 R11: 0000000000000005 R12: ffff88041fd83ce8 [85505.160203] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15: ffffffffa03f8548 [85505.160204] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000) knlGS:0000000000000000 [85505.160204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [85505.160205] CR2: 000000000d2ecf58 CR3: 00000003cfff7000 CR4: 00000000000406e0 [85505.160208] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [85505.160209] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [85505.160209] Stack: [85505.160209] ffffffffa03f8548 ffff88041fd83da0 ffffffffa03e2afb ffff880405eea000 [85505.160211] ffff8803cf535000 ffff88041fd83de0 ffffffffa03e6cdc 0000000100000000 [85505.160212] ffff880405eea000 000000000000007d 0000000000000100 ffffffffa03e7330 [85505.160214] Call Trace: [85505.160215] <IRQ> [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [85505.160220] [<ffffffffa03e6cdc>] tipc_link_send_proto_msg+0x42c/0x500 [tipc] [85505.160222] [<ffffffffa03e7330>] ? link_start+0x40/0x40 [tipc] [85505.160224] [<ffffffffa03e7281>] link_state_event+0x4d1/0x540 [tipc] [85505.160227] [<ffffffffa03e739d>] link_timeout+0x6d/0x160 [tipc] [85505.160229] [<ffffffff8108b086>] call_timer_fn+0x36/0x110 [85505.160233] [<ffffffffa03e7330>] ? link_start+0x40/0x40 [tipc] [85505.160235] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340 [85505.160237] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [85505.160238] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [85505.160240] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [85505.160244] [<ffffffff81084e85>] irq_exit+0x115/0x120 [85505.160245] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60 [85505.160246] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80 [85505.160249] <EOI> [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [85505.160253] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [85505.160255] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [85505.160256] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [85505.160259] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 [85505.160261] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [85505.262147] jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [85505.267647] CPU: 1 PID: 32669 Comm: kworker/1:1 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [85505.269871] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [85505.271629] Workqueue: events linkwatch_event [85505.273230] task: ffff8800368ce780 ti: ffff8803b4894000 task.ti: ffff8803b4894000 [85505.275103] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [85505.277082] RSP: 0018:ffff8803b4897cd0 EFLAGS: 00000202 [85505.278757] RAX: 0000000000000542 RBX: ffffffff8151a6f6 RCX: 000000000000d0c6 [85505.280589] RDX: 000000000000d0c8 RSI: 000000000000d0c8 RDI: ffff880036b5a484 [85505.282400] RBP: ffff8803b4897cd8 R08: 0000000000017600 R09: ffff88041fc97600 [85505.284218] R10: ffffffff8151825d R11: ffffea000ed28f00 R12: ffffffff8151a5be [85505.286035] R13: ffff8803b4897c48 R14: ffff8803b4a3de00 R15: ffff8803b4a3de00 [85505.287857] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000) knlGS:0000000000000000 [85505.289789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [85505.291498] CR2: 00007fa0200011f8 CR3: 00000003cf432000 CR4: 00000000000406e0 [85505.293322] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [85505.295153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [85505.296956] Stack: [85505.298303] ffff880405eea000 ffff8803b4897d10 ffffffffa03e300a ffffffffa03f9ba0 [85505.300243] ffffffff81a277c0 0000000000000004 ffff880405c04000 0000000000000000 [85505.302181] ffff8803b4897d28 ffffffffa03f2e23 00000000ffffffed ffff8803b4897d60 [85505.304127] Call Trace: [85505.305557] [<ffffffffa03e300a>] tipc_block_bearer+0x9a/0x110 [tipc] [85505.307351] [<ffffffffa03f2e23>] recv_notification+0xb3/0xe0 [tipc] [85505.309139] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70 [85505.310889] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20 [85505.312648] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60 [85505.314406] [<ffffffff815284c3>] netdev_state_change+0x23/0x40 [85505.316133] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60 [85505.317840] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200 [85505.319592] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30 [85505.321301] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 [85505.323018] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400 [85505.324736] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 [85505.326465] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [85505.328097] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [85505.329904] [<ffffffff816460d8>] ret_from_fork+0x58/0x90 [85505.331600] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [85505.333399] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 The other issue is as follow: _acpi ata_piix libata floppy [86111.730893] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86111.733189] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86111.735086] task: ffffffff81951440 ti: ffffffff8193c000 task.ti: ffffffff8193c000 [86111.737137] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86111.739298] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202 [86111.741157] RAX: 000000000000418b RBX: ffff88041fc03ba8 RCX: 0000000000008bd0 [86111.743174] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI: ffff8800dba6e0c4 [86111.745181] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09: ffff880405ee0000 [86111.747183] R10: ffff88041fc102ac R11: ffffea0010142200 R12: ffff88041fc03b58 [86111.749177] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15: ffff8800dba6e0c4 [86111.751185] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [86111.753280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86111.755171] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4: 00000000000406f0 [86111.757188] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86111.759184] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86111.761169] Stack: [86111.762683] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1 0000000000000400 [86111.764791] 0000000000000400 0000000100000001 ffff8800dba6e0c4 ffffffffa03fa7a8 [86111.766898] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce 0000000000000400 [86111.768987] Call Trace: [86111.770561] <IRQ> [86111.770767] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc] [86111.774011] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc] [86111.775845] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0 [86111.777769] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30 [86111.779660] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60 [86111.781528] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0 [86111.783369] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0 [86111.785210] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730 [ixgbevf] [86111.787186] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80 [86111.789062] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf] [86111.790979] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240 [86111.792823] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86111.794661] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86111.796485] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86111.798282] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86111.800099] [<ffffffff81648638>] do_IRQ+0x58/0xf0 [86111.801864] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d [86111.803692] <EOI> [86111.803894] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86111.807013] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86111.808746] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86111.810458] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86111.812196] [<ffffffff81625647>] rest_init+0x77/0x80 [86111.813807] [<ffffffff81a8d057>] start_kernel+0x429/0x44a [86111.815414] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c [86111.817031] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120 [86111.818640] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c [86111.820239] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175 [86111.821787] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86112.505490] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561] [86112.507084] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86112.520240] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86112.522381] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86112.524120] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti: ffff8803d0760000 [86112.526036] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86112.528086] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212 [86112.529825] RAX: 0000000000004071 RBX: ffffffff8152c617 RCX: 000000000000483c [86112.531733] RDX: 0000000000004840 RSI: 0000000000004840 RDI: ffffffffa03fa548 [86112.533636] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09: 0000000000000040 [86112.535534] R10: ffff88040f001600 R11: ffff8803d0763c98 R12: ffff8803d0763a20 [86112.537419] R13: ffffffff81a70408 R14: ffff880402003e00 R15: ffff88040900e000 [86112.539324] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000) knlGS:0000000000000000 [86112.541341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86112.543132] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4: 00000000000406e0 [86112.545056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86112.547003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86112.548907] Stack: [86112.550350] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb ffffffffa03f9ed8 [86112.552380] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270 ffffffff812f2ad5 [86112.554411] 0000000000000004 00000000d0763a98 ffffffffa03f96a0 ffff8804063c7660 [86112.556446] Call Trace: [86112.557951] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86112.559835] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc] [86112.561731] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50 [86112.563565] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc] [86112.565462] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20 [86112.567330] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc] [86112.569251] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc] [86112.571147] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc] [86112.572984] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0 [86112.574781] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0 [86112.576607] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900 [86112.578420] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0 [86112.580231] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20 [86112.582056] [<ffffffff815121ce>] SyS_sendto+0xe/0x10 [86112.583832] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b [86112.585694] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86113.785497] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] [86113.789068] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86113.804040] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86113.806367] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86113.808283] task: ffff88040957e780 ti: ffff8804095c8000 task.ti: ffff8804095c8000 [86113.810337] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86113.812489] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202 [86113.814326] RAX: 000000000000093c RBX: ffff88041fd83d30 RCX: 000000000000483c [86113.816325] RDX: 000000000000483e RSI: 000000000000483e RDI: ffffffffa03fa548 [86113.818320] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09: 0000000000000300 [86113.820297] R10: ffff88040f001600 R11: 0000000000000005 R12: ffff88041fd83ce8 [86113.822272] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15: ffffffffa03fa548 [86113.824256] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000) knlGS:0000000000000000 [86113.826323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86113.828186] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4: 00000000000406e0 [86113.830187] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86113.832165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86113.834148] Stack: [86113.835627] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb ffff880407786c00 [86113.837689] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc 0000000100000000 [86113.839757] ffff880407786c00 000000000000007d 0000000000000100 ffffffffa03e9330 [86113.841826] Call Trace: [86113.843365] <IRQ> [86113.843575] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86113.846836] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500 [tipc] [86113.848821] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86113.850681] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc] [86113.852588] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc] [86113.854448] [<ffffffff8108b086>] call_timer_fn+0x36/0x110 [86113.856262] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86113.858126] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340 [86113.859978] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86113.861782] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86113.863568] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86113.865331] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86113.867101] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60 [86113.868999] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80 [86113.870846] <EOI> [86113.871048] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86113.874221] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86113.876019] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86113.877796] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86113.879604] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 [86113.881359] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86137.772653] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:22252] [86137.775109] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86137.789057] CPU: 1 PID: 22252 Comm: kworker/1:0 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86137.791303] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86137.793103] Workqueue: events linkwatch_event [86137.794785] task: ffff880405f58b80 ti: ffff8803c7a0c000 task.ti: ffff8803c7a0c000 [86137.796754] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86137.798845] RSP: 0018:ffff8803c7a0fcd0 EFLAGS: 00000202 [86137.800633] RAX: 0000000000002925 RBX: ffffffffa03fa548 RCX: 0000000000008bd0 [86137.802599] RDX: 0000000000008bd2 RSI: 0000000000008bd2 RDI: ffff8800dba6e0c4 [86137.804569] RBP: ffff8803c7a0fcd8 R08: 0000000000000001 R09: ffff8803d06d50f0 [86137.806516] R10: 0000000022304000 R11: ffff8800db881b30 R12: 0000000000000001 [86137.808465] R13: ffff880407785000 R14: ffff8804080c73c0 R15: 00000000a2bfb72e [86137.810431] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000) knlGS:0000000000000000 [86137.812490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86137.814333] CR2: 00000000f771f000 CR3: 00000004055c9000 CR4: 00000000000406e0 [86137.816303] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86137.818266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86137.820219] Stack: [86137.821678] ffffffffa03fa4d8 ffff8803c7a0fd10 ffffffffa03e500a ffffffffa03fbba0 [86137.823733] ffffffff81a277c0 0000000000000004 ffff8804062ec000 0000000000000000 [86137.825786] ffff8803c7a0fd28 ffffffffa03f4e23 00000000ffffffed ffff8803c7a0fd60 [86137.827837] Call Trace: [86137.829380] [<ffffffffa03e500a>] tipc_block_bearer+0x9a/0x110 [tipc] [86137.831308] [<ffffffffa03f4e23>] recv_notification+0xb3/0xe0 [tipc] [86137.833225] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70 [86137.835112] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20 [86137.837025] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60 [86137.838957] [<ffffffff815284c3>] netdev_state_change+0x23/0x40 [86137.840843] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60 [86137.842709] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200 [86137.844622] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30 [86137.846494] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 [86137.848384] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400 [86137.850266] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 [86137.852147] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [86137.853937] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [86137.855884] [<ffffffff816460d8>] ret_from_fork+0x58/0x90 [86137.857720] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [86137.859692] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86139.713656] BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] [86139.715810] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86139.730670] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86139.732968] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86139.734858] task: ffffffff81951440 ti: ffffffff8193c000 task.ti: ffffffff8193c000 [86139.736900] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86139.739071] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202 [86139.740936] RAX: 0000000000000c54 RBX: ffff88041fc03ba8 RCX: 0000000000008bd0 [86139.742961] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI: ffff8800dba6e0c4 [86139.744965] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09: ffff880405ee0000 [86139.746966] R10: ffff88041fc102ac R11: ffffea0010142200 R12: ffff88041fc03b58 [86139.748963] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15: ffff8800dba6e0c4 [86139.750967] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [86139.753068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86139.754964] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4: 00000000000406f0 [86139.756982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86139.758979] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86139.760967] Stack: [86139.762477] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1 0000000000000400 [86139.764579] 0000000000000400 0000000100000001 ffff8800dba6e0c4 ffffffffa03fa7a8 [86139.766700] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce 0000000000000400 [86139.768810] Call Trace: [86139.770384] <IRQ> [86139.770592] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc] [86139.773835] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc] [86139.775660] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0 [86139.777588] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30 [86139.779471] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60 [86139.781349] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0 [86139.783193] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0 [86139.785034] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730 [ixgbevf] [86139.787019] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80 [86139.788886] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf] [86139.790806] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240 [86139.792642] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86139.794474] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86139.796300] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86139.798089] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86139.799883] [<ffffffff81648638>] do_IRQ+0x58/0xf0 [86139.801629] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d [86139.803456] <EOI> [86139.803657] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86139.806770] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86139.808503] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86139.810226] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86139.811956] [<ffffffff81625647>] rest_init+0x77/0x80 [86139.813565] [<ffffffff81a8d057>] start_kernel+0x429/0x44a [86139.815182] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c [86139.816800] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120 [86139.818394] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c [86139.820005] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175 [86139.821545] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86140.505667] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561] [86140.507258] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86140.520420] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86140.522566] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86140.524309] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti: ffff8803d0760000 [86140.526226] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86140.528274] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212 [86140.530009] RAX: 0000000000004e1c RBX: ffffffff8152c617 RCX: 000000000000483c [86140.531914] RDX: 0000000000004840 RSI: 0000000000004840 RDI: ffffffffa03fa548 [86140.533814] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09: 0000000000000040 [86140.535724] R10: ffff88040f001600 R11: ffff8803d0763c98 R12: ffff8803d0763a20 [86140.537610] R13: ffffffff81a70408 R14: ffff880402003e00 R15: ffff88040900e000 [86140.539513] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000) knlGS:0000000000000000 [86140.541517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86140.543311] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4: 00000000000406e0 [86140.545238] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86140.547172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86140.549083] Stack: [86140.550528] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb ffffffffa03f9ed8 [86140.552565] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270 ffffffff812f2ad5 [86140.554585] 0000000000000004 00000000d0763a98 ffffffffa03f96a0 ffff8804063c7660 [86140.556603] Call Trace: [86140.558115] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86140.560007] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc] [86140.561904] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50 [86140.563734] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc] [86140.565618] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20 [86140.567491] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc] [86140.569409] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc] [86140.571304] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc] [86140.573144] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0 [86140.574929] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0 [86140.576747] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900 [86140.578549] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0 [86140.580352] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20 [86140.582165] [<ffffffff815121ce>] SyS_sendto+0xe/0x10 [86140.583927] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b [86140.585773] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86141.785677] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] [86141.788301] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86141.803307] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86141.805630] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86141.807547] task: ffff88040957e780 ti: ffff8804095c8000 task.ti: ffff8804095c8000 [86141.809601] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86141.811764] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202 [86141.813598] RAX: 0000000000000292 RBX: ffff88041fd83d30 RCX: 000000000000483c [86141.815599] RDX: 000000000000483e RSI: 000000000000483e RDI: ffffffffa03fa548 [86141.817586] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09: 0000000000000300 [86141.819566] R10: ffff88040f001600 R11: 0000000000000005 R12: ffff88041fd83ce8 [86141.821522] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15: ffffffffa03fa548 [86141.823486] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000) knlGS:0000000000000000 [86141.825554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86141.827411] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4: 00000000000406e0 [86141.829408] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86141.831379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86141.833338] Stack: [86141.834816] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb ffff880407786c00 [86141.836882] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc 0000000100000000 [86141.838951] ffff880407786c00 000000000000007d 0000000000000100 ffffffffa03e9330 [86141.841010] Call Trace: [86141.842549] <IRQ> [86141.842760] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86141.846017] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500 [tipc] [86141.847987] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86141.849838] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc] [86141.851743] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc] [86141.853606] [<ffffffff8108b086>] call_timer_fn+0x36/0x110 [86141.855418] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86141.857289] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340 [86141.859144] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86141.860955] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86141.862750] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86141.864511] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86141.866278] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60 [86141.868171] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80 [86141.870016] <EOI> [86141.870216] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86141.873404] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86141.875206] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86141.876984] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86141.878783] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 [86141.880538] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86165.772816] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:22252] [86165.776315] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86165.790273] CPU: 1 PID: 22252 Comm: kworker/1:0 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86165.792530] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86165.794334] Workqueue: events linkwatch_event [86165.796016] task: ffff880405f58b80 ti: ffff8803c7a0c000 task.ti: ffff8803c7a0c000 [86165.797984] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86165.800079] RSP: 0018:ffff8803c7a0fcd0 EFLAGS: 00000202 [86165.801875] RAX: 0000000000001b2d RBX: ffffffffa03fa548 RCX: 0000000000008bd0 [86165.803849] RDX: 0000000000008bd2 RSI: 0000000000008bd2 RDI: ffff8800dba6e0c4 [86165.805805] RBP: ffff8803c7a0fcd8 R08: 0000000000000001 R09: ffff8803d06d50f0 [86165.807753] R10: 0000000022304000 R11: ffff8800db881b30 R12: 0000000000000001 [86165.809711] R13: ffff880407785000 R14: ffff8804080c73c0 R15: 00000000a2bfb72e [86165.811671] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000) knlGS:0000000000000000 [86165.813729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86165.815573] CR2: 00000000f771f000 CR3: 00000004055c9000 CR4: 00000000000406e0 [86165.817547] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86165.819511] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86165.821468] Stack: [86165.822927] ffffffffa03fa4d8 ffff8803c7a0fd10 ffffffffa03e500a ffffffffa03fbba0 [86165.824986] ffffffff81a277c0 0000000000000004 ffff8804062ec000 0000000000000000 [86165.827041] ffff8803c7a0fd28 ffffffffa03f4e23 00000000ffffffed ffff8803c7a0fd60 [86165.829098] Call Trace: [86165.830641] [<ffffffffa03e500a>] tipc_block_bearer+0x9a/0x110 [tipc] [86165.832572] [<ffffffffa03f4e23>] recv_notification+0xb3/0xe0 [tipc] [86165.834492] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70 [86165.836382] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20 [86165.838292] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60 [86165.840227] [<ffffffff815284c3>] netdev_state_change+0x23/0x40 [86165.842112] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60 [86165.843983] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200 [86165.845913] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30 [86165.847778] [<ffffffff8109d5db>] process_one_work+0x17b/0x470 [86165.849676] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400 [86165.851556] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400 [86165.853440] [<ffffffff810a5acf>] kthread+0xcf/0xe0 [86165.855232] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [86165.857176] [<ffffffff816460d8>] ret_from_fork+0x58/0x90 [86165.859016] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140 [86165.860968] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86167.713832] BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] [86167.719975] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86167.735125] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86167.737431] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86167.739319] task: ffffffff81951440 ti: ffffffff8193c000 task.ti: ffffffff8193c000 [86167.741370] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86167.743532] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202 [86167.745390] RAX: 000000000000769d RBX: ffff88041fc03ba8 RCX: 0000000000008bd0 [86167.747408] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI: ffff8800dba6e0c4 [86167.749408] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09: ffff880405ee0000 [86167.751404] R10: ffff88041fc102ac R11: ffffea0010142200 R12: ffff88041fc03b58 [86167.753391] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15: ffff8800dba6e0c4 [86167.755388] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000) knlGS:0000000000000000 [86167.757484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86167.759364] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4: 00000000000406f0 [86167.761387] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86167.763382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86167.765367] Stack: [86167.766882] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1 0000000000000400 [86167.768980] 0000000000000400 0000000100000001 ffff8800dba6e0c4 ffffffffa03fa7a8 [86167.771081] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce 0000000000000400 [86167.773166] Call Trace: [86167.774731] <IRQ> [86167.774946] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc] [86167.778186] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc] [86167.780018] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0 [86167.781941] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30 [86167.783825] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60 [86167.785699] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0 [86167.787548] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0 [86167.789395] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730 [ixgbevf] [86167.791373] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80 [86167.793237] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf] [86167.795166] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240 [86167.797024] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86167.798848] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86167.800660] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86167.802464] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86167.804262] [<ffffffff81648638>] do_IRQ+0x58/0xf0 [86167.806024] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d [86167.807850] <EOI> [86167.808054] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86167.811173] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86167.812911] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86167.814615] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86167.816353] [<ffffffff81625647>] rest_init+0x77/0x80 [86167.817957] [<ffffffff81a8d057>] start_kernel+0x429/0x44a [86167.819562] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c [86167.821201] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120 [86167.822793] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c [86167.824411] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175 [86167.825959] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86168.505839] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561] [86168.507421] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86168.520623] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86168.522763] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86168.524499] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti: ffff8803d0760000 [86168.526406] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86168.528456] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212 [86168.530192] RAX: 0000000000006bda RBX: ffffffff8152c617 RCX: 000000000000483c [86168.532105] RDX: 0000000000004840 RSI: 0000000000004840 RDI: ffffffffa03fa548 [86168.534005] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09: 0000000000000040 [86168.535897] R10: ffff88040f001600 R11: ffff8803d0763c98 R12: ffff8803d0763a20 [86168.537785] R13: ffffffff81a70408 R14: ffff880402003e00 R15: ffff88040900e000 [86168.539684] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000) knlGS:0000000000000000 [86168.541689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86168.543499] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4: 00000000000406e0 [86168.545424] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86168.547351] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86168.549265] Stack: [86168.550712] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb ffffffffa03f9ed8 [86168.552751] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270 ffffffff812f2ad5 [86168.554771] 0000000000000004 00000000d0763a98 ffffffffa03f96a0 ffff8804063c7660 [86168.556786] Call Trace: [86168.558304] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86168.560188] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc] [86168.562081] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50 [86168.563916] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc] [86168.565813] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20 [86168.567681] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc] [86168.569606] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc] [86168.571492] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc] [86168.573335] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0 [86168.575127] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0 [86168.576948] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900 [86168.578750] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0 [86168.580567] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20 [86168.582381] [<ffffffff815121ce>] SyS_sendto+0xe/0x10 [86168.584171] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b [86168.586018] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 [86169.785848] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] [86169.788316] Modules linked in: iptable_filter xt_limit iptable_mangle ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring virtio ata_generic pata_acpi ata_piix libata floppy [86169.803317] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL ------------ 3.10.0-327.18.2.el7.x86_64 #1 [86169.805654] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [86169.807583] task: ffff88040957e780 ti: ffff8804095c8000 task.ti: ffff8804095c8000 [86169.809639] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>] _raw_spin_lock_bh+0x3d/0x50 [86169.811804] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202 [86169.813647] RAX: 000000000000447f RBX: ffff88041fd83d30 RCX: 000000000000483c [86169.815650] RDX: 000000000000483e RSI: 000000000000483e RDI: ffffffffa03fa548 [86169.817644] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09: 0000000000000300 [86169.819617] R10: ffff88040f001600 R11: 0000000000000005 R12: ffff88041fd83ce8 [86169.821582] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15: ffffffffa03fa548 [86169.823560] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000) knlGS:0000000000000000 [86169.825636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [86169.827499] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4: 00000000000406e0 [86169.829506] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [86169.831487] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [86169.833446] Stack: [86169.834927] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb ffff880407786c00 [86169.837002] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc 0000000100000000 [86169.839073] ffff880407786c00 000000000000007d 0000000000000100 ffffffffa03e9330 [86169.841144] Call Trace: [86169.842685] <IRQ> [86169.842894] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc] [86169.846165] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500 [tipc] [86169.848144] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86169.850004] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc] [86169.851933] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc] [86169.853797] [<ffffffff8108b086>] call_timer_fn+0x36/0x110 [86169.855612] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc] [86169.857486] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340 [86169.859336] [<ffffffff81084aef>] __do_softirq+0xef/0x280 [86169.861132] [<ffffffff81647a9c>] call_softirq+0x1c/0x30 [86169.862926] [<ffffffff81016fc5>] do_softirq+0x65/0xa0 [86169.864688] [<ffffffff81084e85>] irq_exit+0x115/0x120 [86169.866458] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60 [86169.868363] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80 [86169.870208] <EOI> [86169.870409] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10 [86169.873581] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0 [86169.875391] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30 [86169.877178] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290 [86169.878994] [<ffffffff810475fa>] start_secondary+0x1ba/0x230 [86169.880754] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66 ------------------------------------------------------------------------------ _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
