Hi Jon, First of all, slock lock is designed very specially and wisely. In process context, it's similar to a mutex. By contrast, in interrupt context, it likes a spin lock. Moreover, it can safely protect members of sock struct on both contexts. When we are in interrupt/softirq mode and the "owned" flag is set, we know that sk_receive_queue is being operated by a process. At the moment we have to put skb buffer into backlog queue. When release_lock() is called by that process who set "owner" flag, all skb buffers will be moved from backlog queue to sk_recevie_queue. Regarding the scenario of slock protection, there are three different race conditions: 1. process vs process contexts, that is, two processes running in different CPUs concurrently access a sock instance. In this case, slock can is used as mutex to protect sock struct. For example, one process is check "owned" flag is set, it will be slept until the flag is unset by another process. 2. process vs softirq contexts. In process, once "owned" flag is set, slock spin lock is released immediately. In softirq context, it first grabs slock, and then checks whether the flag is set or not. If it's set, it will try later on, or put skb into backlog queue and then exit. 3. softirq vs softirq contexts. Slock plays the role of spin lock semantics in this case. Moreover, it's not necessary to explicitly disable BH under this mode as we already know we are in BH mode, and BH is disabled when softirq's handler is called. This also means that all functions called in BH are nonreentarnt.
Regarding the principles above, firstly we cannot use lock_sock() tipc_sk_timeout() as it is called in BH mode. If we really want to do, deadlock will definitely happen as might_sleep() would be called in BH. Secondly, it's unnecessary to set "owned" flag in tipc_sk_timeout() as it's run in BH mode. In my opinion, the fundamental method to avoid the issue should be to ensure any functions called in BH cannot be iteratively invoked , otherwise, it's hard to definitely prevent the issue from happening again in other similar scenarios. In fact we ever proposed several possible solutions of how to well deal with the concurrent scenario in skb receive path. Especially, we need to stay BH mode to directly forward back a skb received in BH, which very easily causes the problem that a skb will be routed back and forth, leading deadlock occurs. Unfortunately, so far it seems that we are still unable to perfectly solve the issue. Therefore, maybe we have to radically change the current mechanism of receiving skb and forward skb on BH mode. Regards, Ying -----Original Message----- From: Jon Maloy [mailto:jon.ma...@ericsson.com] Sent: 2016年5月30日 5:32 To: GUNA; Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne; Xue, Ying; Xue Ying (ying.x...@gmail.com) Subject: RE: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the card on 4.4.0 Hi Guna, I am looking at it, but don't have much time to spend at it right now. A further study of your dump makes me believe this is a case of race between tipc_recv_stream(), which in user context is setting the "owned" flag but not grabbing slock, and tipc_sk_timeout(), which in interrupt mode is grabbing slock without checking the "owned" flag. This creates an obvious risk of a race, but I don't yet understand the exact implications or solution to it. It may be that it is sufficient to just replace the grabbing of slock with a call to lock_sock() in the timer function as I first suggested, but I rather believe we have to explicitly grab both slock and the flag and keep them for the duration of the timeout, before tipc_node_xmit_skb(). This will protect the timeout against interference from both user context (tipc_recv_stream() and other sw interrups (tipc_sk_rcv()) and vice versa. Or maybe not even this is enough? I will continue my analysis, but input from others would be appreciated. ///jon > -----Original Message----- > From: GUNA [mailto:gbala...@gmail.com] > Sent: Saturday, 28 May, 2016 06:00 > To: Jon Maloy; tipc-discussion@lists.sourceforge.net; Erik Hugne > Subject: Re: [tipc-discussion] tipc_sk_rcv: Kernel panic on one of the card > on 4.4.0 > > Any update on the issue? Any other thoughts or possible fix ? > > The issue was seen on slot12 (1.1.12) node only. The other slots were up. > > I got the full logs as listed here: > > > May 19 05:03:01 [SEQ 248049] dcsx5testslot13 /USR/SBIN/CROND[11359]: > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) > May 19 05:03:21 [SEQ 249182] dcsx5testslot12 kernel: [673637.606852] > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] > May 19 05:03:21 [SEQ 249183] dcsx5testslot12 kernel: [673637.607791] > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] > May 19 05:03:21 [SEQ 249184] dcsx5testslot12 kernel: [673637.607817] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:03:21 [SEQ 249185] dcsx5testslot12 kernel: [673637.607819] > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O 4.4.0 #17 > May 19 05:03:21 [SEQ 249186] dcsx5testslot12 kernel: [673637.607820] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:03:21 [SEQ 249187] dcsx5testslot12 kernel: [673637.607821] > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 > May 19 05:03:21 [SEQ 249188] dcsx5testslot12 kernel: [673637.607827] > RIP: 0010:[<ffffffff81361750>] [<ffffffff81361750>] > rhashtable_jhash2+0x0/0x > f0 > May 19 05:03:21 [SEQ 249189] dcsx5testslot12 kernel: [673637.607828] > RSP: 0018:ffff88035fc63a60 EFLAGS: 00000206 > May 19 05:03:21 [SEQ 249190] dcsx5testslot12 kernel: [673637.607829] > RAX: ffff880351613a80 RBX: ffff880347948000 RCX: ffff880347949060 > May 19 05:03:21 [SEQ 249191] dcsx5testslot12 kernel: [673637.607830] > RDX: 0000000003ff6972 RSI: 0000000000000001 RDI: ffff88035fc63a84 > May 19 05:03:21 [SEQ 249192] dcsx5testslot12 kernel: [673637.607830] > RBP: ffff88035fc63ab8 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:03:21 [SEQ 249193] dcsx5testslot12 kernel: [673637.607831] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 > May 19 05:03:21 [SEQ 249194] dcsx5testslot12 kernel: [673637.607832] > R13: ffff88009a4dc000 R14: ffff8803502f5640 R15: ffff88035fc63be4 > May 19 05:03:21 [SEQ 249195] dcsx5testslot12 kernel: [673637.607833] > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) > knlGS:0000000000000000 > May 19 05:03:21 [SEQ 249196] dcsx5testslot12 kernel: [673637.607834] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:03:21 [SEQ 249197] dcsx5testslot12 kernel: [673637.607835] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:03:21 [SEQ 249198] dcsx5testslot12 kernel: [673637.607835] Stack: > :6600 > May 19 05:04:45 [SEQ 252476] dcsx5testslot12 kernel: [673721.608818] > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0] > May 19 05:04:45 [SEQ 252477] dcsx5testslot12 kernel: [673721.608844] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:04:45 [SEQ 252478] dcsx5testslot12 kernel: [673721.608847] > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 > May 19 05:04:45 [SEQ 252479] dcsx5testslot12 kernel: [673721.608848] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:04:45 [SEQ 252480] dcsx5testslot12 kernel: [673721.608849] > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 > May 19 05:04:45 [SEQ 252481] dcsx5testslot12 kernel: [673721.608858] > RIP: 0010:[<ffffffffa01546a5>] [<ffffffffa01546a5>] > tipc_sk_rcv+0x95/0x490 [ > tipc] > May 19 05:04:45 [SEQ 252482] dcsx5testslot12 kernel: [673721.608859] > RSP: 0018:ffff88035fc63ac8 EFLAGS: 00000296 > May 19 05:04:45 [SEQ 252483] dcsx5testslot12 kernel: [673721.608860] > RAX: 0000000000000301 RBX: ffff88035fc63b70 RCX: 0000000000000001 > May 19 05:04:45 [SEQ 252484] dcsx5testslot12 kernel: [673721.608861] > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffff81ce6240 > May 19 05:04:45 [SEQ 252485] dcsx5testslot12 kernel: [673721.608862] > RBP: ffff88035fc63b38 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:04:45 [SEQ 252486] dcsx5testslot12 kernel: [673721.608862] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 > May 19 05:04:45 [SEQ 252487] dcsx5testslot12 kernel: [673721.608863] > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 > May 19 05:04:45 [SEQ 252488] dcsx5testslot12 kernel: [673721.608864] > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) > knlGS:0000000000000000 > May 19 05:04:45 [SEQ 252489] dcsx5testslot12 kernel: [673721.608865] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:04:45 [SEQ 252490] dcsx5testslot12 kernel: [673721.608866] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:04:45 [SEQ 252491] dcsx5testslot12 kernel: [673721.608867] Stack: > May 19 05:04:45 [SEQ 252492] dcsx5testslot12 kernel: [673721.608868] > ffff880350b37000 ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 > May 19 05:04:45 [SEQ 252493] dcsx5testslot12 kernel: [673721.608870] > ffff8803502f56c8 000000012821c002 ffff880300000000 02c14a9800000000 > May 19 05:04:45 [SEQ 252494] dcsx5testslot12 kernel: [673721.608871] > ffff88035fc63b48 ffff88035fc63b70 0000000000000000 000000000100100c > May 19 05:04:45 [SEQ 252495] dcsx5testslot12 kernel: [673721.608871] > Call Trace: > May 19 05:04:45 [SEQ 252496] dcsx5testslot12 kernel: [673721.608882] <IRQ> > May 19 05:04:45 [SEQ 252497] dcsx5testslot12 kernel: [673721.608882] > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] > May 19 05:04:45 [SEQ 252498] dcsx5testslot12 kernel: [673721.608887] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:04:45 [SEQ 252499] dcsx5testslot12 kernel: [673721.608890] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:04:45 [SEQ 252500] dcsx5testslot12 kernel: [673721.608894] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:04:45 [SEQ 252501] dcsx5testslot12 kernel: [673721.608900] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:04:45 [SEQ 252502] dcsx5testslot12 kernel: [673721.608903] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:04:45 [SEQ 252503] dcsx5testslot12 kernel: [673721.608907] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:04:45 [SEQ 252504] dcsx5testslot12 kernel: [673721.608910] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:04:45 [SEQ 252505] dcsx5testslot12 kernel: [673721.608913] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252506] dcsx5testslot12 kernel: [673721.608916] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:04:45 [SEQ 252507] dcsx5testslot12 kernel: [673721.608919] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:04:45 [SEQ 252508] dcsx5testslot12 kernel: [673721.608923] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252509] dcsx5testslot12 kernel: [673721.608925] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:04:45 [SEQ 252510] dcsx5testslot12 kernel: [673721.608926] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:04:45 [SEQ 252511] dcsx5testslot12 kernel: [673721.608929] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252512] dcsx5testslot12 kernel: [673721.608930] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:04:45 [SEQ 252513] dcsx5testslot12 kernel: [673721.608933] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:04:45 [SEQ 252514] dcsx5testslot12 kernel: [673721.608934] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:04:45 [SEQ 252515] dcsx5testslot12 kernel: [673721.608937] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > :6800 > May 19 05:05:13 [SEQ 254646] dcsx5testslot12 kernel: [673749.609200] > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffff81ce6240 > May 19 05:05:13 [SEQ 254647] dcsx5testslot12 kernel: [673749.609200] > RBP: ffff88035fc63b38 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:05:13 [SEQ 254648] dcsx5testslot12 kernel: [673749.609201] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 > May 19 05:05:13 [SEQ 254649] dcsx5testslot12 kernel: [673749.609202] > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 > May 19 05:05:13 [SEQ 254650] dcsx5testslot12 kernel: [673749.609203] > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) > knlGS:0000000000000000 > May 19 05:05:13 [SEQ 254651] dcsx5testslot12 kernel: [673749.609205] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:05:13 [SEQ 254652] dcsx5testslot12 kernel: [673749.609206] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:05:13 [SEQ 254653] dcsx5testslot12 kernel: [673749.609207] Stack: > May 19 05:05:13 [SEQ 254654] dcsx5testslot12 kernel: [673749.609209] > ffff880350b37000 ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 > May 19 05:05:13 [SEQ 254655] dcsx5testslot12 kernel: [673749.609211] > ffff8803502f56c8 000000012821c002 ffff880300000000 02c14a9800000000 > May 19 05:05:13 [SEQ 254656] dcsx5testslot12 kernel: [673749.609212] > ffff88035fc63b48 ffff88035fc63b70 0000000000000000 000000000100100c > May 19 05:05:13 [SEQ 254657] dcsx5testslot12 kernel: [673749.609213] > Call Trace: > May 19 05:05:13 [SEQ 254658] dcsx5testslot12 kernel: [673749.609224] <IRQ> > May 19 05:05:13 [SEQ 254659] dcsx5testslot12 kernel: [673749.609224] > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] > May 19 05:05:13 [SEQ 254660] dcsx5testslot12 kernel: [673749.609228] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:13 [SEQ 254661] dcsx5testslot12 kernel: [673749.609232] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:05:13 [SEQ 254662] dcsx5testslot12 kernel: [673749.609235] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:13 [SEQ 254663] dcsx5testslot12 kernel: [673749.609239] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:05:13 [SEQ 254664] dcsx5testslot12 kernel: [673749.609242] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:05:13 [SEQ 254665] dcsx5testslot12 kernel: [673749.609245] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:05:13 [SEQ 254666] dcsx5testslot12 kernel: [673749.609249] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:13 [SEQ 254667] dcsx5testslot12 kernel: [673749.609252] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254668] dcsx5testslot12 kernel: [673749.609255] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:13 [SEQ 254669] dcsx5testslot12 kernel: [673749.609259] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:05:13 [SEQ 254670] dcsx5testslot12 kernel: [673749.609262] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254671] dcsx5testslot12 kernel: [673749.609264] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:05:13 [SEQ 254672] dcsx5testslot12 kernel: [673749.609265] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:05:13 [SEQ 254673] dcsx5testslot12 kernel: [673749.609269] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254674] dcsx5testslot12 kernel: [673749.609270] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:05:13 [SEQ 254675] dcsx5testslot12 kernel: [673749.609272] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:05:13 [SEQ 254676] dcsx5testslot12 kernel: [673749.609274] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:05:13 [SEQ 254677] dcsx5testslot12 kernel: [673749.609277] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:05:13 [SEQ 254678] dcsx5testslot12 kernel: [673749.609279] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:05:13 [SEQ 254679] dcsx5testslot12 kernel: [673749.609282] <EOI> > May 19 05:05:13 [SEQ 254680] dcsx5testslot12 kernel: [673749.609282] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:05:13 [SEQ 254681] dcsx5testslot12 kernel: [673749.609284] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:05:13 [SEQ 254682] dcsx5testslot12 kernel: [673749.609285] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:05:13 [SEQ 254683] dcsx5testslot12 kernel: [673749.609287] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:05:13 [SEQ 254684] dcsx5testslot12 kernel: [673749.609291] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:05:13 [SEQ 254685] dcsx5testslot12 kernel: [673749.609292] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:05:13 [SEQ 254686] dcsx5testslot12 kernel: [673749.609295] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:05:13 [SEQ 254687] dcsx5testslot12 kernel: [673749.609309] > Code: 7d 02 00 00 84 c9 74 14 48 8b 90 d8 00 00 00 48 8b 00 8b 52 14 > 49 39 c4 > 0f ca 75 df 41 89 d5 4c 89 ff e8 ff 9a 58 e1 48 8b 7d a8 <44> 89 ee > e8 13 ec ff ff 48 85 c0 49 89 c6 0f 84 62 02 00 00 48 > May 19 05:05:13 [SEQ 254688] dcsx5testslot12 kernel: [673749.986573] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:05:13 [SEQ 254689] dcsx5testslot12 kernel: [673750.038404] > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 > May 19 05:05:13 [SEQ 254690] dcsx5testslot12 kernel: [673750.045635] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:05:13 [SEQ 254691] dcsx5testslot12 kernel: [673750.054809] > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 > May 19 05:05:13 [SEQ 254692] dcsx5testslot12 kernel: [673750.062396] > RIP: 0010:[<ffffffff81083fda>] [<ffffffff81083fda>] > __local_bh_enable_ip+0x3 > a/0x90 > May 19 05:05:13 [SEQ 254693] dcsx5testslot12 kernel: [673750.071327] > RSP: 0018:ffff88035fc43a98 EFLAGS: 00000203 > May 19 05:05:13 [SEQ 254694] dcsx5testslot12 kernel: [673750.076881] > RAX: 0000000000000301 RBX: 00000000fffffe01 RCX: 0000000000000001 > May 19 05:05:13 [SEQ 254695] dcsx5testslot12 kernel: [673750.084226] > RDX: 0000000000000003 RSI: 0000000000000200 RDI: ffffffffa01546cc > May 19 05:05:13 [SEQ 254696] dcsx5testslot12 kernel: [673750.091590] > RBP: ffff88035fc43aa8 R08: 00000000000002d4 R09: 0000000000000004 > May 19 05:05:13 [SEQ 254697] dcsx5testslot12 kernel: [673750.099014] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 > May 19 05:05:13 [SEQ 254698] dcsx5testslot12 kernel: [673750.106368] > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 > May 19 05:05:13 [SEQ 254699] dcsx5testslot12 kernel: [673750.113679] > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) > knlGS:0000000000000000 > May 19 05:05:13 [SEQ 254700] dcsx5testslot12 kernel: [673750.122083] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:05:13 [SEQ 254701] dcsx5testslot12 kernel: [673750.127989] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:05:13 [SEQ 254702] dcsx5testslot12 kernel: [673750.135328] Stack: > May 19 05:05:13 [SEQ 254703] dcsx5testslot12 kernel: [673750.137503] > ffff88035fc43bd0 ffff88035fc43b70 ffff88035fc43ab8 ffffffff816de340 > May 19 05:05:13 [SEQ 254704] dcsx5testslot12 kernel: [673750.145228] > ffff88035fc43b38 ffffffffa01546cc ffff88034a8fad10 0000000000000020 > May 19 05:05:13 [SEQ 254705] dcsx5testslot12 kernel: [673750.152927] > ffff88035fc43af8 ffffffff81ce6240 ffff880350124f48 000000012821c002 > May 19 05:05:13 [SEQ 254706] dcsx5testslot12 kernel: [673750.160645] > Call Trace: > May 19 05:05:13 [SEQ 254707] dcsx5testslot12 kernel: [673750.163228] <IRQ> > May 19 05:05:13 [SEQ 254708] dcsx5testslot12 kernel: [673750.165257] > [<ffffffff816de340>] _raw_spin_trylock_bh+0x40/0x50 > May 19 05:05:13 [SEQ 254709] dcsx5testslot12 kernel: [673750.171642] > [<ffffffffa01546cc>] tipc_sk_rcv+0xbc/0x490 [tipc] > May 19 05:05:13 [SEQ 254710] dcsx5testslot12 kernel: [673750.177847] > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 > May 19 05:05:13 [SEQ 254711] dcsx5testslot12 kernel: [673750.184297] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:13 [SEQ 254712] dcsx5testslot12 kernel: [673750.190730] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:05:13 [SEQ 254713] dcsx5testslot12 kernel: [673750.197159] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:13 [SEQ 254714] dcsx5testslot12 kernel: [673750.203846] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:05:13 [SEQ 254715] dcsx5testslot12 kernel: [673750.210235] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:05:13 [SEQ 254716] dcsx5testslot12 kernel: [673750.216391] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:05:13 [SEQ 254717] dcsx5testslot12 kernel: [673750.222523] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:13 [SEQ 254718] dcsx5testslot12 kernel: [673750.228950] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254719] dcsx5testslot12 kernel: [673750.235862] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:13 [SEQ 254720] dcsx5testslot12 kernel: [673750.242516] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:05:13 [SEQ 254721] dcsx5testslot12 kernel: [673750.249037] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254722] dcsx5testslot12 kernel: [673750.255863] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:05:13 [SEQ 254723] dcsx5testslot12 kernel: [673750.261477] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:05:13 [SEQ 254724] dcsx5testslot12 kernel: [673750.266684] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:13 [SEQ 254725] dcsx5testslot12 kernel: [673750.273508] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:05:13 [SEQ 254726] dcsx5testslot12 kernel: [673750.279572] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:05:13 [SEQ 254727] dcsx5testslot12 kernel: [673750.285152] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:05:13 [SEQ 254728] dcsx5testslot12 kernel: [673750.290309] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:05:13 [SEQ 254729] dcsx5testslot12 kernel: [673750.295338] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:05:13 [SEQ 254730] dcsx5testslot12 kernel: [673750.301132] <EOI> > May 19 05:05:13 [SEQ 254731] dcsx5testslot12 kernel: [673750.303207] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:05:13 [SEQ 254732] dcsx5testslot12 kernel: [673750.309734] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:05:13 [SEQ 254733] dcsx5testslot12 kernel: [673750.316040] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:05:13 [SEQ 254734] dcsx5testslot12 kernel: [673750.321603] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:05:13 [SEQ 254735] dcsx5testslot12 kernel: [673750.327128] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:05:13 [SEQ 254736] dcsx5testslot12 kernel: [673750.332915] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:05:13 [SEQ 254737] dcsx5testslot12 kernel: [673750.339033] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:05:13 [SEQ 254738] dcsx5testslot12 kernel: [673750.345007] > Code: 08 65 8b 05 21 6e f8 7e a9 00 00 0f 00 74 31 80 3d 49 80 c7 00 > 01 75 31 > 83 eb 01 f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e <a9> 00 ff > 1f 00 74 31 65 ff 0d f0 6d f8 7e 48 83 c4 08 5b 5d c3 > May 19 05:05:17 [SEQ 254974] dcsx5testslot2 kernel: [673706.438920] > TCP: request_sock_TCP: Possible SYN flooding on port 4322. Sending > cookies. Ch > eck SNMP counters. > May 19 05:05:26 [SEQ 255213] dcsx5testslot12 /USR/SBIN/CROND[9851]: > (root) CMD (/sbin/corecleanup) > May 19 05:05:41 [SEQ 255727] dcsx5testslot12 kernel: [673777.608487] > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [swapper/2:0] > May 19 05:05:41 [SEQ 255728] dcsx5testslot12 kernel: [673777.609460] > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [swapper/3:0] > May 19 05:05:41 [SEQ 255729] dcsx5testslot12 kernel: [673777.609482] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:05:41 [SEQ 255730] dcsx5testslot12 kernel: [673777.609485] > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 > May 19 05:05:41 [SEQ 255731] dcsx5testslot12 kernel: [673777.609485] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:05:41 [SEQ 255732] dcsx5testslot12 kernel: [673777.609486] > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 > May 19 05:05:41 [SEQ 255733] dcsx5testslot12 kernel: [673777.609492] > RIP: 0010:[<ffffffff81352b4d>] [<ffffffff81352b4d>] memcmp+0x1d/0x50 > May 19 05:05:41 [SEQ 255734] dcsx5testslot12 kernel: [673777.609493] > RSP: 0018:ffff88035fc63a58 EFLAGS: 00000297 > May 19 05:05:41 [SEQ 255735] dcsx5testslot12 kernel: [673777.609494] > RAX: 0000000000000000 RBX: ffff880347948000 RCX: 0000000000000002 > May 19 05:05:41 [SEQ 255736] dcsx5testslot12 kernel: [673777.609494] > RDX: 0000000000000003 RSI: ffff88035fc63a84 RDI: ffff8803502f5914 > May 19 05:05:41 [SEQ 255737] dcsx5testslot12 kernel: [673777.609495] > RBP: ffff88035fc63a58 R08: 00000000000000db R09: 0000000000000000 > May 19 05:05:41 [SEQ 255738] dcsx5testslot12 kernel: [673777.609496] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff8803502f5640 > May 19 05:05:41 [SEQ 255739] dcsx5testslot12 kernel: [673777.609496] > R13: ffff88009a4dc000 R14: ffff8803502f59b0 R15: fffffffffffffc90 > May 19 05:05:41 [SEQ 255740] dcsx5testslot12 kernel: [673777.609498] > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) > knlGS:0000000000000000 > May 19 05:05:41 [SEQ 255741] dcsx5testslot12 kernel: [673777.609498] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:05:41 [SEQ 255742] dcsx5testslot12 kernel: [673777.609499] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:05:41 [SEQ 255743] dcsx5testslot12 kernel: [673777.609500] Stack: > May 19 05:05:41 [SEQ 255744] dcsx5testslot12 kernel: [673777.609501] > ffff88035fc63ab8 ffffffffa0153372 00000000000002d4 0000000000000004 > May 19 05:05:41 [SEQ 255745] dcsx5testslot12 kernel: [673777.609503] > ffff880347949060 b2db99445fc63a90 0000000000000018 ffff88035fc63b70 > May 19 05:05:41 [SEQ 255746] dcsx5testslot12 kernel: [673777.609504] > ffff88035fc63bd0 00000000b2db9944 ffff8803502f5640 ffff88035fc63be4 > May 19 05:05:41 [SEQ 255747] dcsx5testslot12 kernel: [673777.609504] > Call Trace: > May 19 05:05:41 [SEQ 255748] dcsx5testslot12 kernel: [673777.609512] <IRQ> > May 19 05:05:41 [SEQ 255749] dcsx5testslot12 kernel: [673777.609512] > [<ffffffffa0153372>] tipc_sk_lookup+0xb2/0x110 [tipc] > May 19 05:05:41 [SEQ 255750] dcsx5testslot12 kernel: [673777.609520] > [<ffffffffa01546ad>] tipc_sk_rcv+0x9d/0x490 [tipc] > May 19 05:05:41 [SEQ 255751] dcsx5testslot12 kernel: [673777.609529] > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] > May 19 05:05:41 [SEQ 255752] dcsx5testslot12 kernel: [673777.609534] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:41 [SEQ 255753] dcsx5testslot12 kernel: [673777.609537] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:05:41 [SEQ 255754] dcsx5testslot12 kernel: [673777.609540] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:41 [SEQ 255755] dcsx5testslot12 kernel: [673777.609544] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:05:41 [SEQ 255756] dcsx5testslot12 kernel: [673777.609547] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:05:41 [SEQ 255757] dcsx5testslot12 kernel: [673777.609550] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:05:41 [SEQ 255758] dcsx5testslot12 kernel: [673777.609554] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:41 [SEQ 255759] dcsx5testslot12 kernel: [673777.609557] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255760] dcsx5testslot12 kernel: [673777.609561] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:41 [SEQ 255761] dcsx5testslot12 kernel: [673777.609564] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:05:41 [SEQ 255762] dcsx5testslot12 kernel: [673777.609567] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255763] dcsx5testslot12 kernel: [673777.609570] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:05:41 [SEQ 255764] dcsx5testslot12 kernel: [673777.609571] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:05:41 [SEQ 255765] dcsx5testslot12 kernel: [673777.609574] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255766] dcsx5testslot12 kernel: [673777.609575] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:05:41 [SEQ 255767] dcsx5testslot12 kernel: [673777.609578] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:05:41 [SEQ 255768] dcsx5testslot12 kernel: [673777.609582] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:05:41 [SEQ 255769] dcsx5testslot12 kernel: [673777.609584] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:05:41 [SEQ 255770] dcsx5testslot12 kernel: [673777.609586] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:05:41 [SEQ 255771] dcsx5testslot12 kernel: [673777.609589] <EOI> > May 19 05:05:41 [SEQ 255772] dcsx5testslot12 kernel: [673777.609589] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:05:41 [SEQ 255773] dcsx5testslot12 kernel: [673777.609591] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:05:41 [SEQ 255774] dcsx5testslot12 kernel: [673777.609592] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:05:41 [SEQ 255775] dcsx5testslot12 kernel: [673777.609594] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:05:41 [SEQ 255776] dcsx5testslot12 kernel: [673777.609595] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:05:41 [SEQ 255777] dcsx5testslot12 kernel: [673777.609596] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:05:41 [SEQ 255778] dcsx5testslot12 kernel: [673777.609599] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:05:41 [SEQ 255779] dcsx5testslot12 kernel: [673777.609613] > Code: c0 c6 06 00 5d c3 0f 1f 84 00 00 00 00 00 55 31 c0 48 85 d2 48 > 89 e5 74 > 2c 0f b6 07 0f b6 0e 29 c8 75 22 48 83 ea 01 31 c9 eb 15 <44> 0f b6 > 4c 0f 01 44 0f b6 44 0e 01 48 83 c1 01 45 29 c1 75 07 > May 19 05:05:41 [SEQ 255780] dcsx5testslot12 kernel: [673777.998251] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:05:41 [SEQ 255781] dcsx5testslot12 kernel: [673778.049966] > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 > May 19 05:05:41 [SEQ 255782] dcsx5testslot12 kernel: [673778.057328] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:05:41 [SEQ 255783] dcsx5testslot12 kernel: [673778.066505] > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 > May 19 05:05:41 [SEQ 255784] dcsx5testslot12 kernel: [673778.074176] > RIP: 0010:[<ffffffff81083fc7>] [<ffffffff81083fc7>] > __local_bh_enable_ip+0x2 > 7/0x90 > May 19 05:05:41 [SEQ 255785] dcsx5testslot12 kernel: [673778.083184] > RSP: 0018:ffff88035fc43a98 EFLAGS: 00000202 > May 19 05:05:41 [SEQ 255786] dcsx5testslot12 kernel: [673778.088616] > RAX: 0000000000000246 RBX: 0000000000000200 RCX: 0000000000000001 > May 19 05:05:41 [SEQ 255787] dcsx5testslot12 kernel: [673778.095909] > RDX: 00000000ab7500c6 RSI: 0000000000000200 RDI: ffffffffa01546a1 > May 19 05:05:41 [SEQ 255788] dcsx5testslot12 kernel: [673778.103187] > RBP: ffff88035fc43aa8 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:05:41 [SEQ 255789] dcsx5testslot12 kernel: [673778.110537] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 > May 19 05:05:41 [SEQ 255790] dcsx5testslot12 kernel: [673778.117816] > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 > May 19 05:05:41 [SEQ 255791] dcsx5testslot12 kernel: [673778.125094] > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) > knlGS:0000000000000000 > May 19 05:05:41 [SEQ 255792] dcsx5testslot12 kernel: [673778.133394] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:05:41 [SEQ 255793] dcsx5testslot12 kernel: [673778.139293] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:05:41 [SEQ 255794] dcsx5testslot12 kernel: [673778.146549] Stack: > May 19 05:05:41 [SEQ 255795] dcsx5testslot12 kernel: [673778.148733] > ffff88035fc43bd0 ffff88035fc43b70 ffff88035fc43ab8 ffffffff816de1ba > May 19 05:05:41 [SEQ 255796] dcsx5testslot12 kernel: [673778.156407] > ffff88035fc43b38 ffffffffa01546a1 ffff88034a8fad10 0000000000000020 > May 19 05:05:41 [SEQ 255797] dcsx5testslot12 kernel: [673778.164030] > ffff88035fc43af8 ffffffff81ce6240 ffff880350124f48 000000012821c002 > May 19 05:05:41 [SEQ 255798] dcsx5testslot12 kernel: [673778.171626] > Call Trace: > May 19 05:05:41 [SEQ 255799] dcsx5testslot12 kernel: [673778.174174] <IRQ> > May 19 05:05:41 [SEQ 255800] dcsx5testslot12 kernel: [673778.176224] > [<ffffffff816de1ba>] _raw_spin_unlock_bh+0x1a/0x20 > May 19 05:05:41 [SEQ 255801] dcsx5testslot12 kernel: [673778.182545] > [<ffffffffa01546a1>] tipc_sk_rcv+0x91/0x490 [tipc] > May 19 05:05:41 [SEQ 255802] dcsx5testslot12 kernel: [673778.188574] > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 > May 19 05:05:41 [SEQ 255803] dcsx5testslot12 kernel: [673778.195080] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:41 [SEQ 255804] dcsx5testslot12 kernel: [673778.201586] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:05:41 [SEQ 255805] dcsx5testslot12 kernel: [673778.207928] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:41 [SEQ 255806] dcsx5testslot12 kernel: [673778.214549] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:05:41 [SEQ 255807] dcsx5testslot12 kernel: [673778.220911] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:05:41 [SEQ 255808] dcsx5testslot12 kernel: [673778.227002] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:05:41 [SEQ 255809] dcsx5testslot12 kernel: [673778.233133] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:05:41 [SEQ 255810] dcsx5testslot12 kernel: [673778.239640] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255811] dcsx5testslot12 kernel: [673778.246432] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:05:41 [SEQ 255812] dcsx5testslot12 kernel: [673778.252997] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:05:41 [SEQ 255813] dcsx5testslot12 kernel: [673778.259399] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255814] dcsx5testslot12 kernel: [673778.266250] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:05:41 [SEQ 255815] dcsx5testslot12 kernel: [673778.271873] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:05:41 [SEQ 255816] dcsx5testslot12 kernel: [673778.277046] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:05:41 [SEQ 255817] dcsx5testslot12 kernel: [673778.283878] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:05:41 [SEQ 255818] dcsx5testslot12 kernel: [673778.289917] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:05:41 [SEQ 255819] dcsx5testslot12 kernel: [673778.295485] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:05:41 [SEQ 255820] dcsx5testslot12 kernel: [673778.300616] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:05:41 [SEQ 255821] dcsx5testslot12 kernel: [673778.305518] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:05:41 [SEQ 255822] dcsx5testslot12 kernel: [673778.311278] <EOI> > May 19 05:05:41 [SEQ 255823] dcsx5testslot12 kernel: [673778.313319] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:05:41 [SEQ 255824] dcsx5testslot12 kernel: [673778.319908] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:05:41 [SEQ 255825] dcsx5testslot12 kernel: [673778.326317] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:05:41 [SEQ 255826] dcsx5testslot12 kernel: [673778.332001] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:05:41 [SEQ 255827] dcsx5testslot12 kernel: [673778.337473] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:05:41 [SEQ 255828] dcsx5testslot12 kernel: [673778.343278] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:05:41 [SEQ 255829] dcsx5testslot12 kernel: [673778.349396] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:05:41 [SEQ 255830] dcsx5testslot12 kernel: [673778.355259] > Code: 1f 44 00 00 66 66 66 66 90 55 48 89 e5 53 89 f3 48 83 ec 08 65 > 8b 05 21 > 6e f8 7e a9 00 00 0f 00 74 31 80 3d 49 80 c7 00 01 75 31 <83> eb 01 > f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e a9 00 > May 19 05:06:01 [SEQ 256552] dcsx5testslot13 /USR/SBIN/CROND[30520]: > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) > May 19 05:06:01 [SEQ 256570] dcsx5testslot2 /USR/SBIN/CROND[17386]: > (root) CMD (/opt/cpu_ss7gw/current/scripts/mgmt_apache_watchdog) > May 19 05:06:09 [SEQ 256861] dcsx5testslot12 kernel: [673805.608813] > NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] > May 19 05:06:09 [SEQ 256862] dcsx5testslot12 kernel: [673805.609784] > NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [swapper/3:0] > May 19 05:06:09 [SEQ 256863] dcsx5testslot12 kernel: [673805.609807] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:06:09 [SEQ 256864] dcsx5testslot12 kernel: [673805.609809] > CPU: 3 PID: 0 Comm: swapper/3 Tainted: G O L 4.4.0 #17 > May 19 05:06:09 [SEQ 256865] dcsx5testslot12 kernel: [673805.609810] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:06:09 [SEQ 256866] dcsx5testslot12 kernel: [673805.609811] > task: ffff880351a88000 ti: ffff880351a84000 task.ti: ffff880351a84000 > May 19 05:06:09 [SEQ 256867] dcsx5testslot12 kernel: [673805.609817] > RIP: 0010:[<ffffffff81083ff0>] [<ffffffff81083ff0>] > __local_bh_enable_ip+0x5 > 0/0x90 > May 19 05:06:09 [SEQ 256868] dcsx5testslot12 kernel: [673805.609818] > RSP: 0018:ffff88035fc63a90 EFLAGS: 00000246 > May 19 05:06:09 [SEQ 256869] dcsx5testslot12 kernel: [673805.609819] > RAX: 0000000000000500 RBX: 0000000000000200 RCX: 0000000000000001 > May 19 05:06:09 [SEQ 256870] dcsx5testslot12 kernel: [673805.609819] > RDX: 00000000b2db9944 RSI: 0000000000000200 RDI: ffffffffa01546a1 > May 19 05:06:09 [SEQ 256871] dcsx5testslot12 kernel: [673805.609820] > RBP: ffff88035fc63aa8 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:06:09 [SEQ 256872] dcsx5testslot12 kernel: [673805.609821] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc63bd0 > May 19 05:06:09 [SEQ 256873] dcsx5testslot12 kernel: [673805.609821] > R13: 00000000b2db9944 R14: ffff8803502f5640 R15: ffff88035fc63be4 > May 19 05:06:09 [SEQ 256874] dcsx5testslot12 kernel: [673805.609823] > FS: 0000000000000000(0000) GS:ffff88035fc60000(0000) > knlGS:0000000000000000 > May 19 05:06:09 [SEQ 256875] dcsx5testslot12 kernel: [673805.609823] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:06:09 [SEQ 256876] dcsx5testslot12 kernel: [673805.609824] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:06:09 [SEQ 256877] dcsx5testslot12 kernel: [673805.609825] Stack: > May 19 05:06:09 [SEQ 256878] dcsx5testslot12 kernel: [673805.609826] > 0000000000000246 ffff88035fc63bd0 ffff88035fc63b70 ffff88035fc63ab8 > May 19 05:06:09 [SEQ 256879] dcsx5testslot12 kernel: [673805.609828] > ffffffff816de1ba ffff88035fc63b38 ffffffffa01546a1 ffff880350b37000 > May 19 05:06:09 [SEQ 256880] dcsx5testslot12 kernel: [673805.609829] > ffffffffa03017eb ffff88035fc63b7c ffffffff81ce6240 ffff8803502f56c8 > May 19 05:06:09 [SEQ 256881] dcsx5testslot12 kernel: [673805.609829] > Call Trace: > May 19 05:06:09 [SEQ 256882] dcsx5testslot12 kernel: [673805.609839] <IRQ> > May 19 05:06:09 [SEQ 256883] dcsx5testslot12 kernel: [673805.609839] > [<ffffffff816de1ba>] _raw_spin_unlock_bh+0x1a/0x20 > May 19 05:06:09 [SEQ 256884] dcsx5testslot12 kernel: [673805.609846] > [<ffffffffa01546a1>] tipc_sk_rcv+0x91/0x490 [tipc] > May 19 05:06:09 [SEQ 256885] dcsx5testslot12 kernel: [673805.609855] > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] > May 19 05:06:09 [SEQ 256886] dcsx5testslot12 kernel: [673805.609861] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:06:09 [SEQ 256887] dcsx5testslot12 kernel: [673805.609864] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:06:09 [SEQ 256888] dcsx5testslot12 kernel: [673805.609867] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:06:09 [SEQ 256889] dcsx5testslot12 kernel: [673805.609871] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:06:09 [SEQ 256890] dcsx5testslot12 kernel: [673805.609874] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:06:09 [SEQ 256891] dcsx5testslot12 kernel: [673805.609877] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:06:09 [SEQ 256892] dcsx5testslot12 kernel: [673805.609881] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:06:09 [SEQ 256893] dcsx5testslot12 kernel: [673805.609884] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:06:09 [SEQ 256894] dcsx5testslot12 kernel: [673805.609887] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:06:09 [SEQ 256895] dcsx5testslot12 kernel: [673805.609891] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:06:09 [SEQ 256896] dcsx5testslot12 kernel: [673805.609894] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:06:09 [SEQ 256897] dcsx5testslot12 kernel: [673805.609896] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:06:09 [SEQ 256898] dcsx5testslot12 kernel: [673805.609897] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:06:09 [SEQ 256899] dcsx5testslot12 kernel: [673805.609901] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:06:09 [SEQ 256900] dcsx5testslot12 kernel: [673805.609902] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:06:09 [SEQ 256901] dcsx5testslot12 kernel: [673805.609904] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:06:09 [SEQ 256902] dcsx5testslot12 kernel: [673805.609905] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:06:09 [SEQ 256903] dcsx5testslot12 kernel: [673805.609908] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:06:09 [SEQ 256904] dcsx5testslot12 kernel: [673805.609909] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:06:09 [SEQ 256905] dcsx5testslot12 kernel: [673805.609912] <EOI> > May 19 05:06:09 [SEQ 256906] dcsx5testslot12 kernel: [673805.609912] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:06:09 [SEQ 256907] dcsx5testslot12 kernel: [673805.609914] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:06:09 [SEQ 256908] dcsx5testslot12 kernel: [673805.609915] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:06:09 [SEQ 256909] dcsx5testslot12 kernel: [673805.609917] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:06:09 [SEQ 256910] dcsx5testslot12 kernel: [673805.609918] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:06:09 [SEQ 256911] dcsx5testslot12 kernel: [673805.609919] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:06:09 [SEQ 256912] dcsx5testslot12 kernel: [673805.609922] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:06:09 [SEQ 256913] dcsx5testslot12 kernel: [673805.609937] > Code: 75 31 83 eb 01 f7 db 65 01 1d 05 6e f8 7e 65 8b 05 fe 6d f8 7e > a9 00 ff > 1f 00 74 31 65 ff 0d f0 6d f8 7e 48 83 c4 08 5b 5d c3 9c <58> f6 c4 > 02 75 d1 eb c6 be 96 00 00 00 48 c7 c7 82 27 a3 81 e8 > May 19 05:06:09 [SEQ 256914] dcsx5testslot12 kernel: [673806.000394] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:06:09 [SEQ 256915] dcsx5testslot12 kernel: [673806.052443] > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 > May 19 05:06:09 [SEQ 256916] dcsx5testslot12 kernel: [673806.059943] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:06:09 [SEQ 256917] dcsx5testslot12 kernel: [673806.069246] > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 > May 19 05:06:09 [SEQ 256918] dcsx5testslot12 kernel: [673806.076929] > RIP: 0010:[<ffffffffa01546a5>] [<ffffffffa01546a5>] > tipc_sk_rcv+0x95/0x490 [ > tipc] > May 19 05:06:09 [SEQ 256919] dcsx5testslot12 kernel: [673806.085716] > RSP: 0018:ffff88035fc43ac8 EFLAGS: 00000296 > May 19 05:06:09 [SEQ 256920] dcsx5testslot12 kernel: [673806.091252] > RAX: 0000000000000301 RBX: ffff88035fc43b70 RCX: 0000000000000001 > May 19 05:06:09 [SEQ 256921] dcsx5testslot12 kernel: [673806.098623] > RDX: 00000000ab7500c6 RSI: 0000000000000200 RDI: ffffffff81ce6240 > May 19 05:06:09 [SEQ 256922] dcsx5testslot12 kernel: [673806.106021] > RBP: ffff88035fc43b38 R08: 0000000000000001 R09: 0000000000000004 > May 19 05:06:09 [SEQ 256923] dcsx5testslot12 kernel: [673806.113331] > R10: 0000000000000000 R11: 0000000000000000 R12: ffff88035fc43bd0 > May 19 05:06:09 [SEQ 256924] dcsx5testslot12 kernel: [673806.120634] > R13: 00000000ab7500c6 R14: ffff880350124ec0 R15: ffff88035fc43be4 > May 19 05:06:09 [SEQ 256925] dcsx5testslot12 kernel: [673806.127940] > FS: 0000000000000000(0000) GS:ffff88035fc40000(0000) > knlGS:0000000000000000 > May 19 05:06:09 [SEQ 256926] dcsx5testslot12 kernel: [673806.136201] > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 19 05:06:09 [SEQ 256927] dcsx5testslot12 kernel: [673806.142110] > CR2: 00007f6e9c244000 CR3: 0000000001c0a000 CR4: 00000000000006e0 > May 19 05:06:09 [SEQ 256928] dcsx5testslot12 kernel: [673806.149540] Stack: > May 19 05:06:09 [SEQ 256929] dcsx5testslot12 kernel: [673806.151715] > ffff88034a8fad10 0000000000000020 ffff88035fc43af8 ffffffff81ce6240 > May 19 05:06:09 [SEQ 256930] dcsx5testslot12 kernel: [673806.159346] > ffff880350124f48 000000012821c002 ffff88035fc43b58 ffffffff8162ecde > May 19 05:06:09 [SEQ 256931] dcsx5testslot12 kernel: [673806.167047] > ffff88035fc43b48 ffff88035fc43b70 0000000000000000 000000000100100c > May 19 05:06:09 [SEQ 256932] dcsx5testslot12 kernel: [673806.174661] > Call Trace: > May 19 05:06:09 [SEQ 256933] dcsx5testslot12 kernel: [673806.177199] <IRQ> > May 19 05:06:09 [SEQ 256934] dcsx5testslot12 kernel: [673806.179238] > [<ffffffff8162ecde>] ? tcp_rcv_established+0x40e/0x760 > May 19 05:06:09 [SEQ 256935] dcsx5testslot12 kernel: [673806.185854] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:06:09 [SEQ 256936] dcsx5testslot12 kernel: [673806.192362] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > :6400 > May 19 05:03:49 [SEQ 250358] dcsx5testslot12 kernel: [673665.608194] > [<ffffffff816de332>] ? _raw_spin_trylock_bh+0x32/0x50 > May 19 05:03:49 [SEQ 250359] dcsx5testslot12 kernel: [673665.608198] > [<ffffffffa01546ad>] tipc_sk_rcv+0x9d/0x490 [tipc] > May 19 05:03:49 [SEQ 250360] dcsx5testslot12 kernel: [673665.608206] > [<ffffffffa03017eb>] ? sctp_addto_chunk+0x8b/0xd0 [sctp] > May 19 05:03:49 [SEQ 250361] dcsx5testslot12 kernel: [673665.608210] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:03:49 [SEQ 250362] dcsx5testslot12 kernel: [673665.608213] > [<ffffffff810b5353>] ? find_busiest_group+0x153/0x980 > May 19 05:03:49 [SEQ 250363] dcsx5testslot12 kernel: [673665.608217] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:03:49 [SEQ 250364] dcsx5testslot12 kernel: [673665.608220] > [<ffffffffa0151cb9>] tipc_sk_respond+0x99/0xc0 [tipc] > May 19 05:03:49 [SEQ 250365] dcsx5testslot12 kernel: [673665.608223] > [<ffffffffa0152e6d>] filter_rcv+0x4cd/0x550 [tipc] > May 19 05:03:49 [SEQ 250366] dcsx5testslot12 kernel: [673665.608226] > [<ffffffffa01548ed>] tipc_sk_rcv+0x2dd/0x490 [tipc] > May 19 05:03:49 [SEQ 250367] dcsx5testslot12 kernel: [673665.608233] > [<ffffffffa014f58f>] tipc_node_xmit+0x11f/0x150 [tipc] > May 19 05:03:49 [SEQ 250368] dcsx5testslot12 kernel: [673665.608236] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:03:49 [SEQ 250369] dcsx5testslot12 kernel: [673665.608239] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:03:49 [SEQ 250370] dcsx5testslot12 kernel: [673665.608243] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:03:49 [SEQ 250371] dcsx5testslot12 kernel: [673665.608246] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:03:49 [SEQ 250372] dcsx5testslot12 kernel: [673665.608249] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:03:49 [SEQ 250373] dcsx5testslot12 kernel: [673665.608251] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:03:49 [SEQ 250374] dcsx5testslot12 kernel: [673665.608254] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:03:49 [SEQ 250375] dcsx5testslot12 kernel: [673665.608256] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:03:49 [SEQ 250376] dcsx5testslot12 kernel: [673665.608259] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:03:49 [SEQ 250377] dcsx5testslot12 kernel: [673665.608260] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:03:49 [SEQ 250378] dcsx5testslot12 kernel: [673665.608263] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:03:49 [SEQ 250379] dcsx5testslot12 kernel: [673665.608265] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:03:49 [SEQ 250380] dcsx5testslot12 kernel: [673665.608268] <EOI> > May 19 05:03:49 [SEQ 250381] dcsx5testslot12 kernel: [673665.608268] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:03:49 [SEQ 250382] dcsx5testslot12 kernel: [673665.608269] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:03:49 [SEQ 250383] dcsx5testslot12 kernel: [673665.608270] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:03:49 [SEQ 250384] dcsx5testslot12 kernel: [673665.608273] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:03:49 [SEQ 250385] dcsx5testslot12 kernel: [673665.608274] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:03:49 [SEQ 250386] dcsx5testslot12 kernel: [673665.608275] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:03:49 [SEQ 250387] dcsx5testslot12 kernel: [673665.608278] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:03:49 [SEQ 250388] dcsx5testslot12 kernel: [673665.608292] > Code: 41 31 c8 c1 c2 0e 41 29 d0 44 89 c2 44 31 c0 c1 c2 0b 29 d0 89 > c2 31 c1 > c1 c2 19 29 d1 89 ca 41 31 c8 c1 c2 10 41 29 d0 44 89 c2 <44> 31 c0 > c1 c2 04 29 d0 31 c1 c1 c0 0e 29 c1 41 31 c8 c1 c1 18 > May 19 05:03:49 [SEQ 250389] dcsx5testslot12 kernel: [673666.007702] > Modules linked in: nf_log_ipv4 nf_log_common xt_LOG sctp libcrc32c > e1000e tip > c udp_tunnel ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev > br_netfilter bridge stp llc nf_conntrack_ipv4 ipmiq_drv(O) sio_mmc(O) > nf_defrag_ipv4 ip6 > t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state > nf_conntrack event_drv(O) ip6table_filter ip6_tables ddi(O) lockd > pt_timer_info(O > ) grace ixgbe usb_storage igb pcspkr iTCO_vendor_support i2c_algo_bit > i2c_i801 ptp i2c_core ioatdma lpc_ich mfd_core tpm_tis dca pps_core > intel_ips > tpm mdio sunrpc [last unloaded: iTCO_wdt] > May 19 05:03:49 [SEQ 250390] dcsx5testslot12 kernel: [673666.060133] > CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.4.0 #17 > May 19 05:03:49 [SEQ 250391] dcsx5testslot12 kernel: [673666.067566] > Hardware name: PT AMC124/Base Board Product Name, BIOS > LGNAJFIP.PTI.0012.P15 > 01/15/2014 > May 19 05:03:49 [SEQ 250392] dcsx5testslot12 kernel: [673666.076908] > task: ffff880351a751c0 ti: ffff880351a80000 task.ti: ffff880351a80000 > May 19 05:03:49 [SEQ 250393] dcsx5testslot12 kernel: [673666.084770] > RIP: 0010:[<ffffffff816de29f>] [<ffffffff816de29f>] > _raw_spin_lock_bh+0x1f/0 > x40 > May 19 05:03:49 [SEQ 250394] dcsx5testslot12 kernel: [673666.093585] > RSP: 0018:ffff88035fc43ab8 EFLAGS: 00000246 > May 19 05:03:49 [SEQ 250395] dcsx5testslot12 kernel: [673666.099084] > RAX: 0000000000000000 RBX: ffff88035fc43b70 RCX: 0000000000000001 > May 19 05:03:49 [SEQ 250396] dcsx5testslot12 kernel: [673666.106567] > RDX: 0000000000000001 RSI: 0000000000000200 RDI: ffff88035fc43be4 > /1.1.12 > May 19 05:04:45 [SEQ 252557] dcsx5testslot12 kernel: [673722.241945] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252558] dcsx5testslot12 kernel: [673722.248734] > [<ffffffffa014f5f7>] tipc_node_xmit_skb+0x37/0x60 [tipc] > May 19 05:04:45 [SEQ 252559] dcsx5testslot12 kernel: [673722.255656] > [<ffffffffa0152770>] tipc_sk_timeout+0xe0/0x180 [tipc] > May 19 05:04:45 [SEQ 252560] dcsx5testslot12 kernel: [673722.262126] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252561] dcsx5testslot12 kernel: [673722.269038] > [<ffffffff810e1644>] call_timer_fn+0x44/0x110 > May 19 05:04:45 [SEQ 252562] dcsx5testslot12 kernel: [673722.274789] > [<ffffffff810e0e6a>] ? cascade+0x4a/0x80 > May 19 05:04:45 [SEQ 252563] dcsx5testslot12 kernel: [673722.280087] > [<ffffffffa0152690>] ? tipc_recv_stream+0x370/0x370 [tipc] > May 19 05:04:45 [SEQ 252564] dcsx5testslot12 kernel: [673722.286846] > [<ffffffff810e2c4c>] run_timer_softirq+0x22c/0x280 > May 19 05:04:45 [SEQ 252565] dcsx5testslot12 kernel: [673722.292978] > [<ffffffff81083d88>] __do_softirq+0xc8/0x260 > May 19 05:04:45 [SEQ 252566] dcsx5testslot12 kernel: [673722.298687] > [<ffffffff81084123>] irq_exit+0x83/0xb0 > May 19 05:04:45 [SEQ 252567] dcsx5testslot12 kernel: [673722.303788] > [<ffffffff816e0ca5>] do_IRQ+0x65/0xf0 > May 19 05:04:45 [SEQ 252568] dcsx5testslot12 kernel: [673722.308718] > [<ffffffff816df17f>] common_interrupt+0x7f/0x7f > May 19 05:04:45 [SEQ 252569] dcsx5testslot12 kernel: [673722.314666] <EOI> > May 19 05:04:45 [SEQ 252570] dcsx5testslot12 kernel: [673722.316729] > [<ffffffff81591c0d>] ? cpuidle_enter_state+0xad/0x200 > May 19 05:04:45 [SEQ 252571] dcsx5testslot12 kernel: [673722.323330] > [<ffffffff81591bf1>] ? cpuidle_enter_state+0x91/0x200 > May 19 05:04:45 [SEQ 252572] dcsx5testslot12 kernel: [673722.329738] > [<ffffffff81591d97>] cpuidle_enter+0x17/0x20 > May 19 05:04:45 [SEQ 252573] dcsx5testslot12 kernel: [673722.335346] > [<ffffffff810bcdc7>] call_cpuidle+0x37/0x60 > May 19 05:04:45 [SEQ 252574] dcsx5testslot12 kernel: [673722.340955] > [<ffffffff81591d73>] ? cpuidle_select+0x13/0x20 > May 19 05:04:45 [SEQ 252575] dcsx5testslot12 kernel: [673722.346777] > [<ffffffff810bd001>] cpu_startup_entry+0x211/0x2d0 > May 19 05:04:45 [SEQ 252576] dcsx5testslot12 kernel: [673722.352887] > [<ffffffff8103b213>] start_secondary+0x103/0x130 > May 19 05:04:45 [SEQ 252577] dcsx5testslot12 kernel: [673722.358765] > Code: 44 89 c0 5d c3 44 03 47 08 03 4f 04 89 ca 03 07 41 31 c8 c1 c2 > 0e 41 29 > d0 44 89 c2 44 31 c0 c1 c2 0b 29 d0 89 c2 31 c1 c1 c2 19 <29> d1 89 > ca 41 31 c8 c1 c2 10 41 29 d0 44 89 c2 44 31 c0 c1 c2 > May 19 05:04:57 [SEQ 253019] slot6_0 kernel: TIPC: Resetting link > <1.1.6:eth1-1.1.12:p19p2>, changeover initiated by peer > May 19 05:04:57 [SEQ 253020] slot7_0 kernel: TIPC: Resetting link > <1.1.7:eth1-1.1.12:p19p2>, changeover initiated by peer > May 19 05:04:57 [SEQ 253021] slot8_0 kernel: TIPC: Resetting link > <1.1.8:eth1-1.1.12:p19p2>, changeover initiated by peer > May 19 05:04:57 [SEQ 253022] slot9_0 kernel: TIPC: Resetting link > <1.1.9:eth1-1.1.12:p19p2>, changeover initiated by peer > May 19 05:04:57 [SEQ 253023] slot6_0 kernel: TIPC: Lost link > <1.1.6:eth1-1.1.12:p19p2> on network plane B > May 19 05:04:57 [SEQ 253024] slot7_0 kernel: TIPC: Lost link > <1.1.7:eth1-1.1.12:p19p2> on network plane B > May 19 05:04:57 [SEQ 253025] slot8_0 kernel: TIPC: Lost link > <1.1.8:eth1-1.1.12:p19p2> on network plane B > May 19 05:04:57 [SEQ 253026] slot9_0 kernel: TIPC: Lost link > <1.1.9:eth1-1.1.12:p19p2> on network plane B > May 19 05:05:01 [SEQ 253150] dcsx5testslot11 /USR/SBIN/CROND[4582]: > (root) CMD (/sbin/corecleanup) > May 19 05:05:01 [SEQ 253151] slot9_0 /USR/SBIN/CROND[23109]: (root) > CMD (/sbin/corecleanup) > May 19 05:05:01 [SEQ 253179] dcsx5testslot4 /USR/SBIN/CROND[10667]: > (root) CMD (/sbin/corecleanup) > May 19 05:05:01 [SEQ 253193] dcsx5testslot3 /USR/SBIN/CROND[10640]: > (root) CMD (/sbin/corecleanup) > May 19 05:05:01 [SEQ 253194] slot8_0 /USR/SBIN/CROND[19631]: (root) > CMD (/sbin/corecleanup) > > > > On Tue, May 24, 2016 at 7:29 PM, Jon Maloy <ma...@donjonn.com> wrote: > > > > > > On 05/24/2016 12:16 PM, GUNA wrote: > >> > >> I suspect there could be glitch on switch may cause lost the probe or > >> abort message. However, even if the messages are lost for what ever > >> reason, is not TIPC stack should handle the graceful shutdown of the > >> TIPC connection by releasing all the resources instead of panic or > >> dead itself ? > > > > I don't think this is about lost packets or shutdown at all, as far as I can > > see form the dump.This is a node local socket-to-socket call that bypasses > > any interfaces and switches. > >> > >> > >> Does lock_sock/release_sock use in tipc_sk_timeout() fix the issue ? > > > > > > No, it won't. At first I thought this obvious, as can be deferred from my > > first comment: > > tipc_sk_timeout() grabs sk.slock, sends a message to the receiver socket, > > which for some reason chooses to return it to sender. In tipc_sk_rcv we try > > to grab sk.slock again, and hence we end up in a deadlock. > > > > Unfortunately this theory is not correct. > > tipc_sk_timeout() actually releases sk.slock before it sends out the buffer, > > so there is no way a deadlock can happen this way. Besides, given that we > > don't actually spin on a spin_trylock_bh() call, I believe we are on the > > wrong track when suspecting a slock deadlock. > > > > A message produced by tipc_sk_timeout can only be a > CONN_MANAGER/CONN_PROBE > > message, but I don't see filter_rcv() calling tipc_sk_proto_rcv() anywhere. > > The explanation seems to be that tipc_sk_proto_rcv() is inlined. So, we can > > probably assume that the message causing the deadlock is a regular > > CONN_MANAGER/CONN_PROBE_REPLY. > > > > So, my theory right now is that the reply message for some reason causes > > tipc_sk_rv() to enter a loop where 'inputq' never is emptied. > > > > This is as far as I get in my analysis right now. Does anybody else have any > > good ideas? > > > > ///jon > > > > > > > >> > >> Thanks, > >> Guna > >> > >> > >> ------------------------------------------------------------------------------ > >> Mobile security can be enabling, not merely restricting. Employees who > >> bring their own devices (BYOD) to work are irked by the imposition of MDM > >> restrictions. Mobile Device Manager Plus allows you to control only the > >> apps on BYO-devices by containerizing them, leaving personal data > >> untouched! > >> https://ad.doubleclick.net/ddm/clk/304595813;131938128;j > >> _______________________________________________ > >> tipc-discussion mailing list > >> tipc-discussion@lists.sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > ------------------------------------------------------------------------------ > What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic > patterns at an interface-level. Reveals which users, apps, and protocols are > consuming the most bandwidth. Provides multi-vendor support for NetFlow, > J-Flow, sFlow and other flows. Make informed decisions using capacity > planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e > _______________________________________________ > tipc-discussion mailing list > tipc-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/tipc-discussion ------------------------------------------------------------------------------ What NetFlow Analyzer can do for you? Monitors network bandwidth and traffic patterns at an interface-level. Reveals which users, apps, and protocols are consuming the most bandwidth. Provides multi-vendor support for NetFlow, J-Flow, sFlow and other flows. Make informed decisions using capacity planning reports. https://ad.doubleclick.net/ddm/clk/305295220;132659582;e _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
