Hi Xiang,
I believe commit 512137eeff00f73a8a62e481a6575f1556cf962c ("tipc: remove
interface state mirroring in bearer") should be able to fix the issue.
However, it cannot be directly applied to 3.10 kernel as the commit was
accepted by upstream since v3.14-rc1~94^2~562. So you have two choices:
1. Just back port the commit to 3.10;
2. Another alternative method is to apply all TIPC patches which were
merged from 3.10 to 3.14.
Although the second approach seems a bit crazy, I still prefer to it.
This is because I don't think there are too many patches between 3.10
and 3.14, and these patches should be easily applied into 3.10 kernel
one by one. Moreover, if the attempt is successful, this means that it's
very easy to back port other TIPC patches from upstream.
Regards,
Ying
On 12/02/2016 09:37 AM, XIANG Haiming wrote:
> Hi Ying,
>
> Thank for your detailed analysis.
>
> We will wait for your patch.
>
>
> -----Original Message-----
> From: Ying Xue [mailto:ying....@windriver.com]
> Sent: 2016年12月1日 19:33
> To: XIANG Haiming; tipc-discussion@lists.sourceforge.net
> Subject: Re: [tipc-discussion] soft lockup for TIPC
>
> The root cause of the first soft lockup should like:
>
> link_timeout
> tipc_node_lock(l_ptr->owner) //hold node lock
> link_state_event
> tipc_link_send_proto_msg
> tipc_bearer_blocked
> spin_lock_bh(&b->lock); //hold bearer lock
>
> recv_notification
> read_lock_bh(&tipc_net_lock);
> tipc_block_bearer
> spin_lock_bh(&b_ptr->lock); //hold bearer lock
> spin_lock_bh(&n_ptr->lock); //hold node lock
>
> As the sequence of holding node lock and bearer lock in the two contexts
> above is completely reverse, as a result, deadlock happens.
>
> However, at 3.10 kernel version, tipc_net_lock still exists, which means
> TIPC code is significantly different with kernel upstream now.
>
> Let's think how to fix it on 3.10. The best way to fix the issue is to
> back port patches from versions which are a bit higher than 3.10.
>
> The second soft lockup is similar to the first one as CPU is still
> pending on bearer lock forever.
>
> Regards,
> Ying
>
> On 11/21/2016 10:34 AM, XIANG Haiming wrote:
>> Hi all,
>>
>> The version for TIPC which we use is TIPC version 2.0.0.
>> The OS info is as follow:
>>
>> Red Hat Enterprise Linux Server 7.2 (Maipo)
>> Kernel 3.10.0-327.18.2.el7.x86_64 on an x86_64
>>
>> We meet two soft lockup issue about TIPC, please help us to solve this
>> issue. Thank you
>>
>> One issue is as follow:
>>
>>
>> [85502.601198] BUG: soft lockup - CPU#0 stuck for 22s! [scm:2649]
>> [85502.603585] Modules linked in: iptable_filter ip6table_mangle xt_limit
>> iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [85502.618210] CPU: 0 PID: 2649 Comm: scm Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [85502.620482] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [85502.622361] task: ffff880405da5080 ti: ffff8800db68c000 task.ti:
>> ffff8800db68c000
>> [85502.624400] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [85502.626558] RSP: 0018:ffff8800db68fa68 EFLAGS: 00000202
>> [85502.628412] RAX: 00000000000056f6 RBX: ffff880406600800 RCX:
>> 000000000000712a
>> [85502.630419] RDX: 000000000000712c RSI: 000000000000712c RDI:
>> ffffffffa03f8548
>> [85502.632423] RBP: ffff8800db68fa70 R08: 00000000000002c0 R09:
>> 0000000000000500
>> [85502.634421] R10: ffff88040f001500 R11: ffff8800db68fc58 R12:
>> ffffffff81519be1
>> [85502.636413] R13: ffff8800db68fa08 R14: 00000000ffffffff R15:
>> 0000000000000240
>> [85502.638415] FS: 0000000000000000(0000) GS:ffff88041fc00000(0063)
>> knlGS:00000000d1495b40
>> [85502.640514] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033
>> [85502.642399] CR2: 00000000d37c8208 CR3: 00000003cfff7000 CR4:
>> 00000000000406f0
>> [85502.644420] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [85502.646416] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [85502.648393] Stack:
>> [85502.649916] ffffffffa03f8548 ffff8800db68fa90 ffffffffa03e2afb
>> ffff880036242c00
>> [85502.652019] ffff880406600800 ffff8800db68fac0 ffffffffa03e5c1b
>> ffff880036242c00
>> [85502.654108] ffff880036b5a484 ffff88003698e800 ffff8800db68fb30
>> ffff8800db68fba8
>> [85502.656179] Call Trace:
>> [85502.657742] [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [85502.659678] [<ffffffffa03e5c1b>] link_send_buf_fast+0x5b/0xb0 [tipc]
>> [85502.661592] [<ffffffffa03e6026>] tipc_link_send_sections_fast+0xe6/0x630
>> [tipc]
>> [85502.663594] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20
>> [85502.665473] [<ffffffffa03eb9b0>] ? tipc_nametbl_translate+0xc0/0x1f0
>> [tipc]
>> [85502.667441] [<ffffffffa03ef5d5>] tipc_send2name+0x155/0x1d0 [tipc]
>> [85502.669351] [<ffffffffa03f08cb>] send_msg+0x1eb/0x530 [tipc]
>> [85502.671205] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0
>> [85502.673028] [<ffffffff810e31f3>] ? futex_wait+0x193/0x280
>> [85502.674859] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0
>> [85502.676676] [<ffffffff8164169d>] ? __do_page_fault+0x16d/0x450
>> [85502.678540] [<ffffffff812346d2>] ? poll_select_copy_remaining+0x62/0x130
>> [85502.680489] [<ffffffff815121ce>] SyS_sendto+0xe/0x10
>> [85502.682277] [<ffffffff8154be63>] compat_sys_socketcall+0x173/0x2a0
>> [85502.684193] [<ffffffff8164800c>] sysenter_dispatch+0x7/0x21
>> [85502.686046] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [85503.372189] BUG: soft lockup - CPU#2 stuck for 23s! [nodemgr:2560]
>> [85503.374181] Modules linked in: iptable_filter ip6table_mangle xt_limit
>> iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [85503.389354] CPU: 2 PID: 2560 Comm: nodemgr Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [85503.391715] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [85503.393668] task: ffff880405da6780 ti: ffff8803cfdd4000 task.ti:
>> ffff8803cfdd4000
>> [85503.395749] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [85503.397949] RSP: 0018:ffff8803cfdd7a40 EFLAGS: 00000202
>> [85503.399809] RAX: 00000000000000f7 RBX: ffffffff8152c617 RCX:
>> 000000000000712a
>> [85503.401838] RDX: 000000000000712e RSI: 000000000000712e RDI:
>> ffffffffa03f8548
>> [85503.403857] RBP: ffff8803cfdd7a48 R08: 000000000003f1e4 R09:
>> 0000000000000500
>> [85503.405861] R10: ffff88040f001500 R11: ffff8803cfdd7c98 R12:
>> ffff8803cfdd7a20
>> [85503.407861] R13: ffffffff81a70408 R14: ffff8803cfc44e00 R15:
>> ffff88040900e000
>> [85503.409857] FS: 00007f2b09eaf740(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [85503.411951] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [85503.413849] CR2: 00000000d2496120 CR3: 00000003cf432000 CR4:
>> 00000000000406e0
>> [85503.415897] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [85503.417920] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [85503.419901] Stack:
>> [85503.421402] ffffffffa03f8548 ffff8803cfdd7a68 ffffffffa03e2afb
>> ffffffffa03f7ed8
>> [85503.423506] ffff8803b4968800 ffff8803cfdd7ab8 ffffffffa03e1270
>> ffffffff8151aaed
>> [85503.425596] 0000000000000004 00000000cfdd7a98 ffffffffa03f76a0
>> ffff8803cfce5860
>> [85503.427703] Call Trace:
>> [85503.429261] [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [85503.431225] [<ffffffffa03e1270>] tipc_bcbearer_send+0xb0/0x270 [tipc]
>> [85503.433182] [<ffffffff8151aaed>] ? __alloc_skb+0x8d/0x2d0
>> [85503.435031] [<ffffffffa03e5943>] tipc_link_send_buf+0xc3/0x2a0 [tipc]
>> [85503.436985] [<ffffffff81518c00>] ? skb_copy_bits+0x60/0x290
>> [85503.438857] [<ffffffff8151ab01>] ? __alloc_skb+0xa1/0x2d0
>> [85503.440698] [<ffffffffa03e1934>] tipc_bclink_send_msg+0x34/0x90 [tipc]
>> [85503.442641] [<ffffffffa03ee9f2>] tipc_multicast+0x132/0x1f0 [tipc]
>> [85503.444539] [<ffffffffa03f0872>] send_msg+0x192/0x530 [tipc]
>> [85503.446377] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0
>> [85503.448187] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
>> [85503.450014] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900
>> [85503.451831] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0
>> [85503.453646] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20
>> [85503.455470] [<ffffffff815121ce>] SyS_sendto+0xe/0x10
>> [85503.457239] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
>> [85503.459098] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [85505.146170] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:1:32669]
>> [85505.150300] Modules linked in: iptable_filter ip6table_mangle xt_limit
>> iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache[85505.160168] BUG: soft lockup - CPU#3 stuck for 22s!
>> [swapper/3:0]
>> [85505.160168] Modules linked in: iptable_filter ip6table_mangle xt_limit
>> iptable_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [85505.160193] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [85505.160193] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [85505.160194] task: ffff88040957e780 ti: ffff8804095c8000 task.ti:
>> ffff8804095c8000
>> [85505.160195] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [85505.160200] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000216
>> [85505.160200] RAX: 0000000000003a59 RBX: ffff88041fd83d30 RCX:
>> 000000000000712a
>> [85505.160201] RDX: 0000000000007130 RSI: 0000000000007130 RDI:
>> ffffffffa03f8548
>> [85505.160202] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09:
>> 0000000000000300
>> [85505.160202] R10: ffff88040f001600 R11: 0000000000000005 R12:
>> ffff88041fd83ce8
>> [85505.160203] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15:
>> ffffffffa03f8548
>> [85505.160204] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000)
>> knlGS:0000000000000000
>> [85505.160204] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [85505.160205] CR2: 000000000d2ecf58 CR3: 00000003cfff7000 CR4:
>> 00000000000406e0
>> [85505.160208] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [85505.160209] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [85505.160209] Stack:
>> [85505.160209] ffffffffa03f8548 ffff88041fd83da0 ffffffffa03e2afb
>> ffff880405eea000
>> [85505.160211] ffff8803cf535000 ffff88041fd83de0 ffffffffa03e6cdc
>> 0000000100000000
>> [85505.160212] ffff880405eea000 000000000000007d 0000000000000100
>> ffffffffa03e7330
>> [85505.160214] Call Trace:
>> [85505.160215] <IRQ> [<ffffffffa03e2afb>] tipc_bearer_blocked+0x1b/0x30
>> [tipc]
>> [85505.160220] [<ffffffffa03e6cdc>] tipc_link_send_proto_msg+0x42c/0x500
>> [tipc]
>> [85505.160222] [<ffffffffa03e7330>] ? link_start+0x40/0x40 [tipc]
>> [85505.160224] [<ffffffffa03e7281>] link_state_event+0x4d1/0x540 [tipc]
>> [85505.160227] [<ffffffffa03e739d>] link_timeout+0x6d/0x160 [tipc]
>> [85505.160229] [<ffffffff8108b086>] call_timer_fn+0x36/0x110
>> [85505.160233] [<ffffffffa03e7330>] ? link_start+0x40/0x40 [tipc]
>> [85505.160235] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340
>> [85505.160237] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [85505.160238] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [85505.160240] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [85505.160244] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [85505.160245] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60
>> [85505.160246] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80
>> [85505.160249] <EOI> [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [85505.160253] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [85505.160255] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [85505.160256] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [85505.160259] [<ffffffff810475fa>] start_secondary+0x1ba/0x230
>> [85505.160261] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>>
>> [85505.262147] jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [85505.267647] CPU: 1 PID: 32669 Comm: kworker/1:1 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [85505.269871] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [85505.271629] Workqueue: events linkwatch_event
>> [85505.273230] task: ffff8800368ce780 ti: ffff8803b4894000 task.ti:
>> ffff8803b4894000
>> [85505.275103] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [85505.277082] RSP: 0018:ffff8803b4897cd0 EFLAGS: 00000202
>> [85505.278757] RAX: 0000000000000542 RBX: ffffffff8151a6f6 RCX:
>> 000000000000d0c6
>> [85505.280589] RDX: 000000000000d0c8 RSI: 000000000000d0c8 RDI:
>> ffff880036b5a484
>> [85505.282400] RBP: ffff8803b4897cd8 R08: 0000000000017600 R09:
>> ffff88041fc97600
>> [85505.284218] R10: ffffffff8151825d R11: ffffea000ed28f00 R12:
>> ffffffff8151a5be
>> [85505.286035] R13: ffff8803b4897c48 R14: ffff8803b4a3de00 R15:
>> ffff8803b4a3de00
>> [85505.287857] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000)
>> knlGS:0000000000000000
>> [85505.289789] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [85505.291498] CR2: 00007fa0200011f8 CR3: 00000003cf432000 CR4:
>> 00000000000406e0
>> [85505.293322] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [85505.295153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [85505.296956] Stack:
>> [85505.298303] ffff880405eea000 ffff8803b4897d10 ffffffffa03e300a
>> ffffffffa03f9ba0
>> [85505.300243] ffffffff81a277c0 0000000000000004 ffff880405c04000
>> 0000000000000000
>> [85505.302181] ffff8803b4897d28 ffffffffa03f2e23 00000000ffffffed
>> ffff8803b4897d60
>> [85505.304127] Call Trace:
>> [85505.305557] [<ffffffffa03e300a>] tipc_block_bearer+0x9a/0x110 [tipc]
>> [85505.307351] [<ffffffffa03f2e23>] recv_notification+0xb3/0xe0 [tipc]
>> [85505.309139] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70
>> [85505.310889] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20
>> [85505.312648] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60
>> [85505.314406] [<ffffffff815284c3>] netdev_state_change+0x23/0x40
>> [85505.316133] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60
>> [85505.317840] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200
>> [85505.319592] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30
>> [85505.321301] [<ffffffff8109d5db>] process_one_work+0x17b/0x470
>> [85505.323018] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400
>> [85505.324736] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
>> [85505.326465] [<ffffffff810a5acf>] kthread+0xcf/0xe0
>> [85505.328097] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [85505.329904] [<ffffffff816460d8>] ret_from_fork+0x58/0x90
>> [85505.331600] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [85505.333399] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>>
>>
>> The other issue is as follow:
>>
>>
>>
>> _acpi ata_piix libata floppy
>> [86111.730893] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86111.733189] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86111.735086] task: ffffffff81951440 ti: ffffffff8193c000 task.ti:
>> ffffffff8193c000
>> [86111.737137] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86111.739298] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202
>> [86111.741157] RAX: 000000000000418b RBX: ffff88041fc03ba8 RCX:
>> 0000000000008bd0
>> [86111.743174] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI:
>> ffff8800dba6e0c4
>> [86111.745181] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09:
>> ffff880405ee0000
>> [86111.747183] R10: ffff88041fc102ac R11: ffffea0010142200 R12:
>> ffff88041fc03b58
>> [86111.749177] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15:
>> ffff8800dba6e0c4
>> [86111.751185] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000)
>> knlGS:0000000000000000
>> [86111.753280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86111.755171] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4:
>> 00000000000406f0
>> [86111.757188] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86111.759184] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86111.761169] Stack:
>> [86111.762683] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1
>> 0000000000000400
>> [86111.764791] 0000000000000400 0000000100000001 ffff8800dba6e0c4
>> ffffffffa03fa7a8
>> [86111.766898] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce
>> 0000000000000400
>> [86111.768987] Call Trace:
>> [86111.770561] <IRQ>
>> [86111.770767] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc]
>> [86111.774011] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc]
>> [86111.775845] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0
>> [86111.777769] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30
>> [86111.779660] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60
>> [86111.781528] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0
>> [86111.783369] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0
>> [86111.785210] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730
>> [ixgbevf]
>> [86111.787186] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80
>> [86111.789062] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf]
>> [86111.790979] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240
>> [86111.792823] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86111.794661] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86111.796485] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86111.798282] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86111.800099] [<ffffffff81648638>] do_IRQ+0x58/0xf0
>> [86111.801864] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d
>> [86111.803692] <EOI>
>> [86111.803894] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86111.807013] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86111.808746] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86111.810458] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86111.812196] [<ffffffff81625647>] rest_init+0x77/0x80
>> [86111.813807] [<ffffffff81a8d057>] start_kernel+0x429/0x44a
>> [86111.815414] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
>> [86111.817031] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
>> [86111.818640] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
>> [86111.820239] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
>> [86111.821787] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86112.505490] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561]
>> [86112.507084] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86112.520240] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86112.522381] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86112.524120] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti:
>> ffff8803d0760000
>> [86112.526036] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86112.528086] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212
>> [86112.529825] RAX: 0000000000004071 RBX: ffffffff8152c617 RCX:
>> 000000000000483c
>> [86112.531733] RDX: 0000000000004840 RSI: 0000000000004840 RDI:
>> ffffffffa03fa548
>> [86112.533636] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09:
>> 0000000000000040
>> [86112.535534] R10: ffff88040f001600 R11: ffff8803d0763c98 R12:
>> ffff8803d0763a20
>> [86112.537419] R13: ffffffff81a70408 R14: ffff880402003e00 R15:
>> ffff88040900e000
>> [86112.539324] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [86112.541341] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86112.543132] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4:
>> 00000000000406e0
>> [86112.545056] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86112.547003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86112.548907] Stack:
>> [86112.550350] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb
>> ffffffffa03f9ed8
>> [86112.552380] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270
>> ffffffff812f2ad5
>> [86112.554411] 0000000000000004 00000000d0763a98 ffffffffa03f96a0
>> ffff8804063c7660
>> [86112.556446] Call Trace:
>> [86112.557951] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86112.559835] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc]
>> [86112.561731] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50
>> [86112.563565] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc]
>> [86112.565462] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20
>> [86112.567330] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc]
>> [86112.569251] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc]
>> [86112.571147] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc]
>> [86112.572984] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0
>> [86112.574781] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
>> [86112.576607] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900
>> [86112.578420] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0
>> [86112.580231] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20
>> [86112.582056] [<ffffffff815121ce>] SyS_sendto+0xe/0x10
>> [86112.583832] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
>> [86112.585694] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86113.785497] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]
>> [86113.789068] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86113.804040] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86113.806367] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86113.808283] task: ffff88040957e780 ti: ffff8804095c8000 task.ti:
>> ffff8804095c8000
>> [86113.810337] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86113.812489] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202
>> [86113.814326] RAX: 000000000000093c RBX: ffff88041fd83d30 RCX:
>> 000000000000483c
>> [86113.816325] RDX: 000000000000483e RSI: 000000000000483e RDI:
>> ffffffffa03fa548
>> [86113.818320] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09:
>> 0000000000000300
>> [86113.820297] R10: ffff88040f001600 R11: 0000000000000005 R12:
>> ffff88041fd83ce8
>> [86113.822272] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15:
>> ffffffffa03fa548
>> [86113.824256] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000)
>> knlGS:0000000000000000
>> [86113.826323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86113.828186] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4:
>> 00000000000406e0
>> [86113.830187] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86113.832165] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86113.834148] Stack:
>> [86113.835627] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb
>> ffff880407786c00
>> [86113.837689] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc
>> 0000000100000000
>> [86113.839757] ffff880407786c00 000000000000007d 0000000000000100
>> ffffffffa03e9330
>> [86113.841826] Call Trace:
>> [86113.843365] <IRQ>
>> [86113.843575] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86113.846836] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500
>> [tipc]
>> [86113.848821] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86113.850681] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc]
>> [86113.852588] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc]
>> [86113.854448] [<ffffffff8108b086>] call_timer_fn+0x36/0x110
>> [86113.856262] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86113.858126] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340
>> [86113.859978] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86113.861782] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86113.863568] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86113.865331] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86113.867101] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60
>> [86113.868999] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80
>> [86113.870846] <EOI>
>> [86113.871048] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86113.874221] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86113.876019] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86113.877796] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86113.879604] [<ffffffff810475fa>] start_secondary+0x1ba/0x230
>> [86113.881359] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86137.772653] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:22252]
>> [86137.775109] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86137.789057] CPU: 1 PID: 22252 Comm: kworker/1:0 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86137.791303] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86137.793103] Workqueue: events linkwatch_event
>> [86137.794785] task: ffff880405f58b80 ti: ffff8803c7a0c000 task.ti:
>> ffff8803c7a0c000
>> [86137.796754] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86137.798845] RSP: 0018:ffff8803c7a0fcd0 EFLAGS: 00000202
>> [86137.800633] RAX: 0000000000002925 RBX: ffffffffa03fa548 RCX:
>> 0000000000008bd0
>> [86137.802599] RDX: 0000000000008bd2 RSI: 0000000000008bd2 RDI:
>> ffff8800dba6e0c4
>> [86137.804569] RBP: ffff8803c7a0fcd8 R08: 0000000000000001 R09:
>> ffff8803d06d50f0
>> [86137.806516] R10: 0000000022304000 R11: ffff8800db881b30 R12:
>> 0000000000000001
>> [86137.808465] R13: ffff880407785000 R14: ffff8804080c73c0 R15:
>> 00000000a2bfb72e
>> [86137.810431] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000)
>> knlGS:0000000000000000
>> [86137.812490] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86137.814333] CR2: 00000000f771f000 CR3: 00000004055c9000 CR4:
>> 00000000000406e0
>> [86137.816303] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86137.818266] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86137.820219] Stack:
>> [86137.821678] ffffffffa03fa4d8 ffff8803c7a0fd10 ffffffffa03e500a
>> ffffffffa03fbba0
>> [86137.823733] ffffffff81a277c0 0000000000000004 ffff8804062ec000
>> 0000000000000000
>> [86137.825786] ffff8803c7a0fd28 ffffffffa03f4e23 00000000ffffffed
>> ffff8803c7a0fd60
>> [86137.827837] Call Trace:
>> [86137.829380] [<ffffffffa03e500a>] tipc_block_bearer+0x9a/0x110 [tipc]
>> [86137.831308] [<ffffffffa03f4e23>] recv_notification+0xb3/0xe0 [tipc]
>> [86137.833225] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70
>> [86137.835112] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20
>> [86137.837025] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60
>> [86137.838957] [<ffffffff815284c3>] netdev_state_change+0x23/0x40
>> [86137.840843] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60
>> [86137.842709] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200
>> [86137.844622] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30
>> [86137.846494] [<ffffffff8109d5db>] process_one_work+0x17b/0x470
>> [86137.848384] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400
>> [86137.850266] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
>> [86137.852147] [<ffffffff810a5acf>] kthread+0xcf/0xe0
>> [86137.853937] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [86137.855884] [<ffffffff816460d8>] ret_from_fork+0x58/0x90
>> [86137.857720] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [86137.859692] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86139.713656] BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
>> [86139.715810] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86139.730670] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86139.732968] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86139.734858] task: ffffffff81951440 ti: ffffffff8193c000 task.ti:
>> ffffffff8193c000
>> [86139.736900] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86139.739071] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202
>> [86139.740936] RAX: 0000000000000c54 RBX: ffff88041fc03ba8 RCX:
>> 0000000000008bd0
>> [86139.742961] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI:
>> ffff8800dba6e0c4
>> [86139.744965] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09:
>> ffff880405ee0000
>> [86139.746966] R10: ffff88041fc102ac R11: ffffea0010142200 R12:
>> ffff88041fc03b58
>> [86139.748963] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15:
>> ffff8800dba6e0c4
>> [86139.750967] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000)
>> knlGS:0000000000000000
>> [86139.753068] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86139.754964] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4:
>> 00000000000406f0
>> [86139.756982] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86139.758979] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86139.760967] Stack:
>> [86139.762477] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1
>> 0000000000000400
>> [86139.764579] 0000000000000400 0000000100000001 ffff8800dba6e0c4
>> ffffffffa03fa7a8
>> [86139.766700] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce
>> 0000000000000400
>> [86139.768810] Call Trace:
>> [86139.770384] <IRQ>
>> [86139.770592] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc]
>> [86139.773835] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc]
>> [86139.775660] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0
>> [86139.777588] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30
>> [86139.779471] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60
>> [86139.781349] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0
>> [86139.783193] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0
>> [86139.785034] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730
>> [ixgbevf]
>> [86139.787019] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80
>> [86139.788886] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf]
>> [86139.790806] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240
>> [86139.792642] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86139.794474] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86139.796300] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86139.798089] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86139.799883] [<ffffffff81648638>] do_IRQ+0x58/0xf0
>> [86139.801629] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d
>> [86139.803456] <EOI>
>> [86139.803657] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86139.806770] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86139.808503] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86139.810226] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86139.811956] [<ffffffff81625647>] rest_init+0x77/0x80
>> [86139.813565] [<ffffffff81a8d057>] start_kernel+0x429/0x44a
>> [86139.815182] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
>> [86139.816800] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
>> [86139.818394] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
>> [86139.820005] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
>> [86139.821545] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86140.505667] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561]
>> [86140.507258] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86140.520420] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86140.522566] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86140.524309] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti:
>> ffff8803d0760000
>> [86140.526226] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86140.528274] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212
>> [86140.530009] RAX: 0000000000004e1c RBX: ffffffff8152c617 RCX:
>> 000000000000483c
>> [86140.531914] RDX: 0000000000004840 RSI: 0000000000004840 RDI:
>> ffffffffa03fa548
>> [86140.533814] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09:
>> 0000000000000040
>> [86140.535724] R10: ffff88040f001600 R11: ffff8803d0763c98 R12:
>> ffff8803d0763a20
>> [86140.537610] R13: ffffffff81a70408 R14: ffff880402003e00 R15:
>> ffff88040900e000
>> [86140.539513] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [86140.541517] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86140.543311] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4:
>> 00000000000406e0
>> [86140.545238] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86140.547172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86140.549083] Stack:
>> [86140.550528] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb
>> ffffffffa03f9ed8
>> [86140.552565] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270
>> ffffffff812f2ad5
>> [86140.554585] 0000000000000004 00000000d0763a98 ffffffffa03f96a0
>> ffff8804063c7660
>> [86140.556603] Call Trace:
>> [86140.558115] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86140.560007] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc]
>> [86140.561904] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50
>> [86140.563734] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc]
>> [86140.565618] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20
>> [86140.567491] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc]
>> [86140.569409] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc]
>> [86140.571304] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc]
>> [86140.573144] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0
>> [86140.574929] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
>> [86140.576747] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900
>> [86140.578549] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0
>> [86140.580352] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20
>> [86140.582165] [<ffffffff815121ce>] SyS_sendto+0xe/0x10
>> [86140.583927] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
>> [86140.585773] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86141.785677] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]
>> [86141.788301] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86141.803307] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86141.805630] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86141.807547] task: ffff88040957e780 ti: ffff8804095c8000 task.ti:
>> ffff8804095c8000
>> [86141.809601] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86141.811764] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202
>> [86141.813598] RAX: 0000000000000292 RBX: ffff88041fd83d30 RCX:
>> 000000000000483c
>> [86141.815599] RDX: 000000000000483e RSI: 000000000000483e RDI:
>> ffffffffa03fa548
>> [86141.817586] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09:
>> 0000000000000300
>> [86141.819566] R10: ffff88040f001600 R11: 0000000000000005 R12:
>> ffff88041fd83ce8
>> [86141.821522] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15:
>> ffffffffa03fa548
>> [86141.823486] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000)
>> knlGS:0000000000000000
>> [86141.825554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86141.827411] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4:
>> 00000000000406e0
>> [86141.829408] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86141.831379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86141.833338] Stack:
>> [86141.834816] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb
>> ffff880407786c00
>> [86141.836882] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc
>> 0000000100000000
>> [86141.838951] ffff880407786c00 000000000000007d 0000000000000100
>> ffffffffa03e9330
>> [86141.841010] Call Trace:
>> [86141.842549] <IRQ>
>> [86141.842760] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86141.846017] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500
>> [tipc]
>> [86141.847987] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86141.849838] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc]
>> [86141.851743] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc]
>> [86141.853606] [<ffffffff8108b086>] call_timer_fn+0x36/0x110
>> [86141.855418] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86141.857289] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340
>> [86141.859144] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86141.860955] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86141.862750] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86141.864511] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86141.866278] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60
>> [86141.868171] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80
>> [86141.870016] <EOI>
>> [86141.870216] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86141.873404] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86141.875206] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86141.876984] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86141.878783] [<ffffffff810475fa>] start_secondary+0x1ba/0x230
>> [86141.880538] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86165.772816] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:22252]
>> [86165.776315] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86165.790273] CPU: 1 PID: 22252 Comm: kworker/1:0 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86165.792530] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86165.794334] Workqueue: events linkwatch_event
>> [86165.796016] task: ffff880405f58b80 ti: ffff8803c7a0c000 task.ti:
>> ffff8803c7a0c000
>> [86165.797984] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86165.800079] RSP: 0018:ffff8803c7a0fcd0 EFLAGS: 00000202
>> [86165.801875] RAX: 0000000000001b2d RBX: ffffffffa03fa548 RCX:
>> 0000000000008bd0
>> [86165.803849] RDX: 0000000000008bd2 RSI: 0000000000008bd2 RDI:
>> ffff8800dba6e0c4
>> [86165.805805] RBP: ffff8803c7a0fcd8 R08: 0000000000000001 R09:
>> ffff8803d06d50f0
>> [86165.807753] R10: 0000000022304000 R11: ffff8800db881b30 R12:
>> 0000000000000001
>> [86165.809711] R13: ffff880407785000 R14: ffff8804080c73c0 R15:
>> 00000000a2bfb72e
>> [86165.811671] FS: 0000000000000000(0000) GS:ffff88041fc80000(0000)
>> knlGS:0000000000000000
>> [86165.813729] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86165.815573] CR2: 00000000f771f000 CR3: 00000004055c9000 CR4:
>> 00000000000406e0
>> [86165.817547] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86165.819511] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86165.821468] Stack:
>> [86165.822927] ffffffffa03fa4d8 ffff8803c7a0fd10 ffffffffa03e500a
>> ffffffffa03fbba0
>> [86165.824986] ffffffff81a277c0 0000000000000004 ffff8804062ec000
>> 0000000000000000
>> [86165.827041] ffff8803c7a0fd28 ffffffffa03f4e23 00000000ffffffed
>> ffff8803c7a0fd60
>> [86165.829098] Call Trace:
>> [86165.830641] [<ffffffffa03e500a>] tipc_block_bearer+0x9a/0x110 [tipc]
>> [86165.832572] [<ffffffffa03f4e23>] recv_notification+0xb3/0xe0 [tipc]
>> [86165.834492] [<ffffffff81641b5c>] notifier_call_chain+0x4c/0x70
>> [86165.836382] [<ffffffff810abb46>] raw_notifier_call_chain+0x16/0x20
>> [86165.838292] [<ffffffff815267ad>] call_netdevice_notifiers+0x2d/0x60
>> [86165.840227] [<ffffffff815284c3>] netdev_state_change+0x23/0x40
>> [86165.842112] [<ffffffff8153f180>] linkwatch_do_dev+0x40/0x60
>> [86165.843983] [<ffffffff8153f477>] __linkwatch_run_queue+0xf7/0x200
>> [86165.845913] [<ffffffff8153f5a5>] linkwatch_event+0x25/0x30
>> [86165.847778] [<ffffffff8109d5db>] process_one_work+0x17b/0x470
>> [86165.849676] [<ffffffff8109e3ab>] worker_thread+0x11b/0x400
>> [86165.851556] [<ffffffff8109e290>] ? rescuer_thread+0x400/0x400
>> [86165.853440] [<ffffffff810a5acf>] kthread+0xcf/0xe0
>> [86165.855232] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [86165.857176] [<ffffffff816460d8>] ret_from_fork+0x58/0x90
>> [86165.859016] [<ffffffff810a5a00>] ? kthread_create_on_node+0x140/0x140
>> [86165.860968] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86167.713832] BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0]
>> [86167.719975] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86167.735125] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86167.737431] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86167.739319] task: ffffffff81951440 ti: ffffffff8193c000 task.ti:
>> ffffffff8193c000
>> [86167.741370] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86167.743532] RSP: 0018:ffff88041fc03be0 EFLAGS: 00000202
>> [86167.745390] RAX: 000000000000769d RBX: ffff88041fc03ba8 RCX:
>> 0000000000008bd0
>> [86167.747408] RDX: 0000000000008bd4 RSI: 0000000000008bd4 RDI:
>> ffff8800dba6e0c4
>> [86167.749408] RBP: ffff88041fc03be8 R08: ffffffffa03fbc38 R09:
>> ffff880405ee0000
>> [86167.751404] R10: ffff88041fc102ac R11: ffffea0010142200 R12:
>> ffff88041fc03b58
>> [86167.753391] R13: ffffffff81646ddd R14: ffff88041fc03be8 R15:
>> ffff8800dba6e0c4
>> [86167.755388] FS: 0000000000000000(0000) GS:ffff88041fc00000(0000)
>> knlGS:0000000000000000
>> [86167.757484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86167.759364] CR2: 00007effe4c45000 CR3: 0000000402143000 CR4:
>> 00000000000406f0
>> [86167.761387] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86167.763382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86167.765367] Stack:
>> [86167.766882] ffff880402113500 ffff88041fc03c70 ffffffffa03e9ec1
>> 0000000000000400
>> [86167.768980] 0000000000000400 0000000100000001 ffff8800dba6e0c4
>> ffffffffa03fa7a8
>> [86167.771081] 0000000000000000 ffff880402113500 ffff8803c7a1b7ce
>> 0000000000000400
>> [86167.773166] Call Trace:
>> [86167.774731] <IRQ>
>> [86167.774946] [<ffffffffa03e9ec1>] tipc_recv_msg+0x1b1/0x8e0 [tipc]
>> [86167.778186] [<ffffffffa03f4c79>] recv_msg+0x49/0x50 [tipc]
>> [86167.780018] [<ffffffff8152add2>] __netif_receive_skb_core+0x582/0x7d0
>> [86167.781941] [<ffffffff81058aaf>] ? kvm_clock_get_cycles+0x1f/0x30
>> [86167.783825] [<ffffffff8152b038>] __netif_receive_skb+0x18/0x60
>> [86167.785699] [<ffffffff8152b0c0>] netif_receive_skb+0x40/0xc0
>> [86167.787548] [<ffffffff8152bc30>] napi_gro_receive+0x80/0xb0
>> [86167.789395] [<ffffffffa00806e3>] ixgbevf_clean_rx_irq+0x383/0x730
>> [ixgbevf]
>> [86167.791373] [<ffffffff810bd0b8>] ? __enqueue_entity+0x78/0x80
>> [86167.793237] [<ffffffffa00822f7>] ixgbevf_poll+0x4c7/0x770 [ixgbevf]
>> [86167.795166] [<ffffffff8152b4f2>] net_rx_action+0x152/0x240
>> [86167.797024] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86167.798848] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86167.800660] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86167.802464] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86167.804262] [<ffffffff81648638>] do_IRQ+0x58/0xf0
>> [86167.806024] [<ffffffff8163d8ed>] common_interrupt+0x6d/0x6d
>> [86167.807850] <EOI>
>> [86167.808054] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86167.811173] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86167.812911] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86167.814615] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86167.816353] [<ffffffff81625647>] rest_init+0x77/0x80
>> [86167.817957] [<ffffffff81a8d057>] start_kernel+0x429/0x44a
>> [86167.819562] [<ffffffff81a8ca37>] ? repair_env_string+0x5c/0x5c
>> [86167.821201] [<ffffffff81a8c120>] ? early_idt_handlers+0x120/0x120
>> [86167.822793] [<ffffffff81a8c5ee>] x86_64_start_reservations+0x2a/0x2c
>> [86167.824411] [<ffffffff81a8c742>] x86_64_start_kernel+0x152/0x175
>> [86167.825959] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86168.505839] BUG: soft lockup - CPU#2 stuck for 22s! [nodemgr:18561]
>> [86168.507421] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86168.520623] CPU: 2 PID: 18561 Comm: nodemgr Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86168.522763] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86168.524499] task: ffff8803d0607300 ti: ffff8803d0760000 task.ti:
>> ffff8803d0760000
>> [86168.526406] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86168.528456] RSP: 0018:ffff8803d0763a40 EFLAGS: 00000212
>> [86168.530192] RAX: 0000000000006bda RBX: ffffffff8152c617 RCX:
>> 000000000000483c
>> [86168.532105] RDX: 0000000000004840 RSI: 0000000000004840 RDI:
>> ffffffffa03fa548
>> [86168.534005] RBP: ffff8803d0763a48 R08: 000000000003f1cf R09:
>> 0000000000000040
>> [86168.535897] R10: ffff88040f001600 R11: ffff8803d0763c98 R12:
>> ffff8803d0763a20
>> [86168.537785] R13: ffffffff81a70408 R14: ffff880402003e00 R15:
>> ffff88040900e000
>> [86168.539684] FS: 00007ff5ecebc740(0000) GS:ffff88041fd00000(0000)
>> knlGS:0000000000000000
>> [86168.541689] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86168.543499] CR2: 00007effe1e43040 CR3: 00000003d061c000 CR4:
>> 00000000000406e0
>> [86168.545424] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86168.547351] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86168.549265] Stack:
>> [86168.550712] ffffffffa03fa548 ffff8803d0763a68 ffffffffa03e4afb
>> ffffffffa03f9ed8
>> [86168.552751] ffff8804063c7800 ffff8803d0763ab8 ffffffffa03e3270
>> ffffffff812f2ad5
>> [86168.554771] 0000000000000004 00000000d0763a98 ffffffffa03f96a0
>> ffff8804063c7660
>> [86168.556786] Call Trace:
>> [86168.558304] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86168.560188] [<ffffffffa03e3270>] tipc_bcbearer_send+0xb0/0x270 [tipc]
>> [86168.562081] [<ffffffff812f2ad5>] ? cpumask_next_and+0x35/0x50
>> [86168.563916] [<ffffffffa03e7943>] tipc_link_send_buf+0xc3/0x2a0 [tipc]
>> [86168.565813] [<ffffffff8163d216>] ? _raw_read_unlock_bh+0x16/0x20
>> [86168.567681] [<ffffffffa03e3934>] tipc_bclink_send_msg+0x34/0x90 [tipc]
>> [86168.569606] [<ffffffffa03f09f2>] tipc_multicast+0x132/0x1f0 [tipc]
>> [86168.571492] [<ffffffffa03f2872>] send_msg+0x192/0x530 [tipc]
>> [86168.573335] [<ffffffff815111d0>] sock_sendmsg+0xb0/0xf0
>> [86168.575127] [<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
>> [86168.576948] [<ffffffff8163aac8>] ? __schedule+0x2d8/0x900
>> [86168.578750] [<ffffffff81511741>] SYSC_sendto+0x121/0x1c0
>> [86168.580567] [<ffffffff810b88a0>] ? wake_up_state+0x20/0x20
>> [86168.582381] [<ffffffff815121ce>] SyS_sendto+0xe/0x10
>> [86168.584171] [<ffffffff81646189>] system_call_fastpath+0x16/0x1b
>> [86168.586018] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>> [86169.785848] BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0]
>> [86169.788316] Modules linked in: iptable_filter xt_limit iptable_mangle
>> ip6table_mangle ip6table_filter ip6_tables igb_uio(OE) uio tipc(OE) 8021q
>> garp stp mrp llc bonding dm_mirror dm_region_hash dm_log dm_mod ppdev
>> crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
>> ablk_helper cryptd pcspkr i6300esb virtio_balloon cirrus syscopyarea
>> sysfillrect sysimgblt ttm drm_kms_helper drm i2c_piix4 i2c_core parport_pc
>> parport binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
>> ext4 mbcache jbd2 virtio_blk virtio_console crct10dif_pclmul
>> crct10dif_common crc32c_intel serio_raw ixgbevf virtio_pci virtio_ring
>> virtio ata_generic pata_acpi ata_piix libata floppy
>> [86169.803317] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G OEL
>> ------------ 3.10.0-327.18.2.el7.x86_64 #1
>> [86169.805654] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
>> [86169.807583] task: ffff88040957e780 ti: ffff8804095c8000 task.ti:
>> ffff8804095c8000
>> [86169.809639] RIP: 0010:[<ffffffff8163d41d>] [<ffffffff8163d41d>]
>> _raw_spin_lock_bh+0x3d/0x50
>> [86169.811804] RSP: 0018:ffff88041fd83d78 EFLAGS: 00000202
>> [86169.813647] RAX: 000000000000447f RBX: ffff88041fd83d30 RCX:
>> 000000000000483c
>> [86169.815650] RDX: 000000000000483e RSI: 000000000000483e RDI:
>> ffffffffa03fa548
>> [86169.817644] RBP: ffff88041fd83d80 R08: 00000000000000c0 R09:
>> 0000000000000300
>> [86169.819617] R10: ffff88040f001600 R11: 0000000000000005 R12:
>> ffff88041fd83ce8
>> [86169.821582] R13: ffffffff81646ddd R14: ffff88041fd83d80 R15:
>> ffffffffa03fa548
>> [86169.823560] FS: 0000000000000000(0000) GS:ffff88041fd80000(0000)
>> knlGS:0000000000000000
>> [86169.825636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [86169.827499] CR2: 00007fb3cd1d0000 CR3: 0000000402143000 CR4:
>> 00000000000406e0
>> [86169.829506] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>> 0000000000000000
>> [86169.831487] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>> 0000000000000400
>> [86169.833446] Stack:
>> [86169.834927] ffffffffa03fa548 ffff88041fd83da0 ffffffffa03e4afb
>> ffff880407786c00
>> [86169.837002] ffff880405b01c00 ffff88041fd83de0 ffffffffa03e8cdc
>> 0000000100000000
>> [86169.839073] ffff880407786c00 000000000000007d 0000000000000100
>> ffffffffa03e9330
>> [86169.841144] Call Trace:
>> [86169.842685] <IRQ>
>> [86169.842894] [<ffffffffa03e4afb>] tipc_bearer_blocked+0x1b/0x30 [tipc]
>> [86169.846165] [<ffffffffa03e8cdc>] tipc_link_send_proto_msg+0x42c/0x500
>> [tipc]
>> [86169.848144] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86169.850004] [<ffffffffa03e9281>] link_state_event+0x4d1/0x540 [tipc]
>> [86169.851933] [<ffffffffa03e939d>] link_timeout+0x6d/0x160 [tipc]
>> [86169.853797] [<ffffffff8108b086>] call_timer_fn+0x36/0x110
>> [86169.855612] [<ffffffffa03e9330>] ? link_start+0x40/0x40 [tipc]
>> [86169.857486] [<ffffffff8108dd77>] run_timer_softirq+0x237/0x340
>> [86169.859336] [<ffffffff81084aef>] __do_softirq+0xef/0x280
>> [86169.861132] [<ffffffff81647a9c>] call_softirq+0x1c/0x30
>> [86169.862926] [<ffffffff81016fc5>] do_softirq+0x65/0xa0
>> [86169.864688] [<ffffffff81084e85>] irq_exit+0x115/0x120
>> [86169.866458] [<ffffffff81648715>] smp_apic_timer_interrupt+0x45/0x60
>> [86169.868363] [<ffffffff81646ddd>] apic_timer_interrupt+0x6d/0x80
>> [86169.870208] <EOI>
>> [86169.870409] [<ffffffff81058e96>] ? native_safe_halt+0x6/0x10
>> [86169.873581] [<ffffffff8101dbcf>] default_idle+0x1f/0xc0
>> [86169.875391] [<ffffffff8101e4d6>] arch_cpu_idle+0x26/0x30
>> [86169.877178] [<ffffffff810d6405>] cpu_startup_entry+0x245/0x290
>> [86169.878994] [<ffffffff810475fa>] start_secondary+0x1ba/0x230
>> [86169.880754] Code: b8 00 00 02 00 f0 0f c1 03 89 c2 c1 ea 10 66 39 c2 75
>> 03 5b 5d c3 83 e2 fe 0f b7 f2 b8 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90
>> <83> e8 01 75 f1 48 89 df 66 66 66 90 66 66 90 eb e0 66 90 66 66
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> tipc-discussion mailing list
>> tipc-discussion@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/tipc-discussion
>>
>
------------------------------------------------------------------------------
_______________________________________________
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion
