Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic

[Xue, Ying]

Thank you for the testing and report!

To be honest, we never met so many issues before. So I doubt all problems you 
encountered may be involved by the recent changes. 
If you have more detailed failure logs, please share them with us so that we 
can look into its root cause.

Thanks,
Ying

-Original Message-
From: GUNA [mailto:gbala...@gmail.com] 
Sent: 2016年5月4日 0:41
To: Xue, Ying
Cc: Erik Hugne; tipc-discussion@lists.sourceforge.net; 
parthasarathy.bhuvara...@ericsson.com; Richard Alpe
Subject: Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic

Thanks Ying.

As you suggested, I will revert the "tipc: avoid packets leaking on socket 
receive queue" patch. Since the issue is not reproducible on demand, I may need 
to wait the issue is seen again or not with the new driver.

We do experience in traffic throughput due to TIPC connections are failing (not 
with this change). If anyone aware of the failures please let me know.


Background 
System is originally based on Kernel 3.4.2 on Fedora 16 and stable.
Recently, I updated the system with kernel 4.4.0. All stock kernel drivers are 
being used and no customization except ported some latest TIPC patches to fix 
some TIPC issues.

All 6 routing CPUs went down over the course of the weekend. The noted output 
is from one CPU; others are unknown, but assumed to have gone down with the 
same cause. Also note that in addition, one of the routing cards (no output 
available) went down again on Monday.

The new kernel 4.4.0 is being used in the system Since April 1st, and seen the 
issue 3 times so far. All these times, mostly heartbeat type traffic, not heavy 
traffic.
=



On Tue, May 3, 2016 at 6:26 AM, Xue, Ying <ying@windriver.com> wrote:
> I agree with Erik too.
>
>
>
> The oops should be caused by socket was freed early. But
>
>
>
> GUNA, can you reproduce the issue? If so, please try to revert the 
> commit
> f4195d1eac954a67adf112dd53404560cc55b942 (“tipc: avoid packets leaking 
> on socket receive queue”), and verify whether the issue occurs or not.
>
>
>
> I suspect the commit bring some unknown side effect, leading to the panic.
>
>
>
> Thanks,
>
> Ying
>
> From: Erik Hugne [mailto:erik.hu...@gmail.com]
> Sent: 2016年5月3日 13:09
> To: GUNA
> Cc: tipc-discussion@lists.sourceforge.net;
> parthasarathy.bhuvara...@ericsson.com; Richard Alpe; Xue, Ying
> Subject: Re: [tipc-discussion] tipc: tipc_recv_stream with kernel 
> panic
>
>
>
> (On mobile)
>
> At first glance, it seems that the socket was freed, but there was a 
> pending wakeup signal for it. Which then causes the subsequent 
> spin_lock_bh() to deref freed mem.
>
> //E
>
> On May 3, 2016 02:43, "GUNA" <gbala...@gmail.com> wrote [...]
>>> [375832.498126] BUG: unable to handle kernel paging request at
>>> 01a400015ff4
>> [375832.505300] IP: []
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.512394] PGD 0
>> [375832.514657] Oops: 0002 [#1] SMP
>> [375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4 
>> nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel 
>> ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp 
>> llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O) 
>> ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state 
>> nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
>> pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801 
>> iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr 
>> sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
>> unloaded: iTCO_wdt]
>> [375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G   O
>>  4.4.0 #14
>> [375832.581385] Hardware name: PT AMC124/Base Board Product Name, 
>> BIOS
>> LGNAJFIP.PTI.0012.P15 01/15/2014
>> [375832.591028] task: 880351a89b40 ti: 880351a9 task.ti:
>> 880351a9
>> [375832.599026] RIP: 0010:[]  []
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.608964] RSP: 0018:88035fc83d58  EFLAGS: 00010002 
>> [375832.614825] RAX: 1447 RBX: 0292 RCX:
>> 88035fc95fc0
>> [375832.622743] RDX: 01a400015ff4 RSI: 0014 RDI:
>> 880351232f80
>> [375832.630567] RBP: 88035fc83d58 R08: 0101 R09:
>> 0004
>> [375832.638348] R10:  R11:  R12:
>> 01001002
>> [375832.645919] R13: 0001 R14:  R15:
>> 
>> [375832.653610] FS:  () GS:88035fc8()
>> knlGS:
>> [375832.662317] CS:  0010 DS:  

Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic

[GUNA]

Thanks Ying.

As you suggested, I will revert the "tipc: avoid packets leaking on
socket receive queue" patch. Since the issue is not reproducible on
demand, I may need to wait the issue is seen again or not with the new
driver.

We do experience in traffic throughput due to TIPC connections are
failing (not with this change). If anyone aware of the failures please
let me know.


Background 
System is originally based on Kernel 3.4.2 on Fedora 16 and stable.
Recently, I updated the system with kernel 4.4.0. All stock kernel
drivers are being used and no customization except ported some latest
TIPC patches to fix some TIPC issues.

All 6 routing CPUs went down over the course of the weekend. The noted
output is from one CPU; others are unknown, but assumed to have gone
down with the same cause. Also note that in addition, one of the
routing cards (no output available) went down again on Monday.

The new kernel 4.4.0 is being used in the system Since April 1st, and
seen the issue 3 times so far. All these times, mostly heartbeat type
traffic, not heavy traffic.
=



On Tue, May 3, 2016 at 6:26 AM, Xue, Ying <ying@windriver.com> wrote:
> I agree with Erik too.
>
>
>
> The oops should be caused by socket was freed early. But
>
>
>
> GUNA, can you reproduce the issue? If so, please try to revert the commit
> f4195d1eac954a67adf112dd53404560cc55b942 (“tipc: avoid packets leaking on
> socket receive queue”), and verify whether the issue occurs or not.
>
>
>
> I suspect the commit bring some unknown side effect, leading to the panic.
>
>
>
> Thanks,
>
> Ying
>
> From: Erik Hugne [mailto:erik.hu...@gmail.com]
> Sent: 2016年5月3日 13:09
> To: GUNA
> Cc: tipc-discussion@lists.sourceforge.net;
> parthasarathy.bhuvara...@ericsson.com; Richard Alpe; Xue, Ying
> Subject: Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic
>
>
>
> (On mobile)
>
> At first glance, it seems that the socket was freed, but there was a pending
> wakeup signal for it. Which then causes the subsequent spin_lock_bh() to
> deref freed mem.
>
> //E
>
> On May 3, 2016 02:43, "GUNA" <gbala...@gmail.com> wrote
> [...]
>>> [375832.498126] BUG: unable to handle kernel paging request at
>>> 01a400015ff4
>> [375832.505300] IP: []
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.512394] PGD 0
>> [375832.514657] Oops: 0002 [#1] SMP
>> [375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4
>> nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel
>> ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp
>> llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O)
>> ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
>> nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
>> pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801
>> iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr
>> sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
>> unloaded: iTCO_wdt]
>> [375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G   O
>>  4.4.0 #14
>> [375832.581385] Hardware name: PT AMC124/Base Board Product Name, BIOS
>> LGNAJFIP.PTI.0012.P15 01/15/2014
>> [375832.591028] task: 880351a89b40 ti: 880351a9 task.ti:
>> 880351a9
>> [375832.599026] RIP: 0010:[]  []
>> queued_spin_lock_slowpath+0xe6/0x160
>> [375832.608964] RSP: 0018:88035fc83d58  EFLAGS: 00010002
>> [375832.614825] RAX: 1447 RBX: 0292 RCX:
>> 88035fc95fc0
>> [375832.622743] RDX: 01a400015ff4 RSI: 0014 RDI:
>> 880351232f80
>> [375832.630567] RBP: 88035fc83d58 R08: 0101 R09:
>> 0004
>> [375832.638348] R10:  R11:  R12:
>> 01001002
>> [375832.645919] R13: 0001 R14:  R15:
>> 
>> [375832.653610] FS:  () GS:88035fc8()
>> knlGS:
>> [375832.662317] CS:  0010 DS:  ES:  CR0: 8005003b
>> [375832.668483] CR2: 01a400015ff4 CR3: 01c0a000 CR4:
>> 06e0
>> [375832.676133] Stack:
>> [375832.678344]  88035fc83d78 816de2c1 88034a8bba60
>> 880351232f80
>> [375832.686163]  88035fc83db8 810bc592 88035fc83dc8
>> 880351758000
>> [375832.694139]  01001002  b802f4bd
>> a024e6f0
>> [375832.702154] Call Trace:
>> [375832.704844]  
>> [375832.707018]  [] rqsav_raw_spin_lock_ie+0x31/0x40
>
>

Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic

[Xue, Ying]

I agree with Erik too.

The oops should be caused by socket was freed early. But

GUNA, can you reproduce the issue? If so, please try to revert the commit 
f4195d1eac954a67adf112dd53404560cc55b942 (“tipc: avoid packets leaking on 
socket receive queue”), and verify whether the issue occurs or not.

I suspect the commit bring some unknown side effect, leading to the panic.

Thanks,
Ying
From: Erik Hugne [mailto:erik.hu...@gmail.com]
Sent: 2016年5月3日 13:09
To: GUNA
Cc: tipc-discussion@lists.sourceforge.net; 
parthasarathy.bhuvara...@ericsson.com; Richard Alpe; Xue, Ying
Subject: Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic


(On mobile)

At first glance, it seems that the socket was freed, but there was a pending 
wakeup signal for it. Which then causes the subsequent spin_lock_bh() to deref 
freed mem.

//E

On May 3, 2016 02:43, "GUNA" <gbala...@gmail.com<mailto:gbala...@gmail.com>> 
wrote
[...]
>> [375832.498126] BUG: unable to handle kernel paging request at 
>> 01a400015ff4
> [375832.505300] IP: [] queued_spin_lock_slowpath+0xe6/0x160
> [375832.512394] PGD 0
> [375832.514657] Oops: 0002 [#1] SMP
> [375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4
> nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel
> ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp
> llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O)
> ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
> pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801
> iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr
> sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
> unloaded: iTCO_wdt]
> [375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G   O
>  4.4.0 #14
> [375832.581385] Hardware name: PT AMC124/Base Board Product Name, BIOS
> LGNAJFIP.PTI.0012.P15 01/15/2014
> [375832.591028] task: 880351a89b40 ti: 880351a9 task.ti:
> 880351a9
> [375832.599026] RIP: 0010:[]  []
> queued_spin_lock_slowpath+0xe6/0x160
> [375832.608964] RSP: 0018:88035fc83d58  EFLAGS: 00010002
> [375832.614825] RAX: 1447 RBX: 0292 RCX:
> 88035fc95fc0
> [375832.622743] RDX: 01a400015ff4 RSI: 0014 RDI:
> 880351232f80
> [375832.630567] RBP: 88035fc83d58 R08: 0101 R09:
> 0004
> [375832.638348] R10:  R11:  R12:
> 01001002
> [375832.645919] R13: 0001 R14:  R15:
> 
> [375832.653610] FS:  () GS:88035fc8()
> knlGS:
> [375832.662317] CS:  0010 DS:  ES:  CR0: 8005003b
> [375832.668483] CR2: 01a400015ff4 CR3: 01c0a000 CR4:
> 06e0
> [375832.676133] Stack:
> [375832.678344]  88035fc83d78 816de2c1 88034a8bba60
> 880351232f80
> [375832.686163]  88035fc83db8 810bc592 88035fc83dc8
> 880351758000
> [375832.694139]  01001002  b802f4bd
> a024e6f0
> [375832.702154] Call Trace:
> [375832.704844]  
> [375832.707018]  [] rqsav_raw_spin_lock_ie+0x31/0x40

rqsav_raw_spin_lock_ie??
Is this some proprietary spinlock implementation?

> [375832.713970]  [] __wake_up+0x32/0x70
> [375832.719444]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
> [375832.726589]  [] sock_def_wakeup+0x30/0x40
> [375832.732566]  [] tipc_sk_timeout+0x148/0x180 [tipc]
> [375832.739388]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
> [375832.746507]  [] call_timer_fn+0x44/0x110
> [375832.752378]  [] ? cascade+0x4a/0x80
> [375832.757848]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
> [375832.764871]  [] run_timer_softirq+0x22c/0x280
> [375832.771175]  [] __do_softirq+0xc8/0x260
> [375832.776958]  [] irq_exit+0x83/0xb0
> [375832.782369]  [] do_IRQ+0x65/0xf0
> [375832.787607]  [] common_interrupt+0x7f/0x7f
> [375832.793709]  
> [375832.795803]  [] ? cpuidle_enter_state+0xad/0x200
> [375832.802765]  [] ? cpuidle_enter_state+0x91/0x200
> [375832.809338]  [] cpuidle_enter+0x17/0x20
> [375832.815155]  [] call_cpuidle+0x37/0x60
> [375832.821184]  [] ? cpuidle_select+0x13/0x20
> [375832.827249]  [] cpu_startup_entry+0x211/0x2d0
> [375832.833535]  [] start_secondary+0x103/0x130
> [375832.839759] Code: 87 47 02 c1 e0 10 85 c0 74 38 48 89 c2 c1 e8 12
> 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 c0 5f 01 00 48 03 14 c5
> 00 b2 d1 81 <48> 89 0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 85 c0 74 f7
> eb 02
> [375832.861151] RIP  [] queued_spin_lock_slowpath+0xe6/0x160
> [375832.868607]  RSP 
> [375832.872371] CR2: 01a40

Re: [tipc-discussion] tipc: tipc_recv_stream with kernel panic

[Erik Hugne]

(On mobile)

At first glance, it seems that the socket was freed, but there was a
pending wakeup signal for it. Which then causes the subsequent
spin_lock_bh() to deref freed mem.

//E

On May 3, 2016 02:43, "GUNA"  wrote
[...]
>> [375832.498126] BUG: unable to handle kernel paging request at
01a400015ff4
> [375832.505300] IP: []
queued_spin_lock_slowpath+0xe6/0x160
> [375832.512394] PGD 0
> [375832.514657] Oops: 0002 [#1] SMP
> [375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4
> nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel
> ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp
> llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O)
> ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
> pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801
> iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr
> sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
> unloaded: iTCO_wdt]
> [375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G   O
>  4.4.0 #14
> [375832.581385] Hardware name: PT AMC124/Base Board Product Name, BIOS
> LGNAJFIP.PTI.0012.P15 01/15/2014
> [375832.591028] task: 880351a89b40 ti: 880351a9 task.ti:
> 880351a9
> [375832.599026] RIP: 0010:[]  []
> queued_spin_lock_slowpath+0xe6/0x160
> [375832.608964] RSP: 0018:88035fc83d58  EFLAGS: 00010002
> [375832.614825] RAX: 1447 RBX: 0292 RCX:
> 88035fc95fc0
> [375832.622743] RDX: 01a400015ff4 RSI: 0014 RDI:
> 880351232f80
> [375832.630567] RBP: 88035fc83d58 R08: 0101 R09:
> 0004
> [375832.638348] R10:  R11:  R12:
> 01001002
> [375832.645919] R13: 0001 R14:  R15:
> 
> [375832.653610] FS:  () GS:88035fc8()
> knlGS:
> [375832.662317] CS:  0010 DS:  ES:  CR0: 8005003b
> [375832.668483] CR2: 01a400015ff4 CR3: 01c0a000 CR4:
> 06e0
> [375832.676133] Stack:
> [375832.678344]  88035fc83d78 816de2c1 88034a8bba60
> 880351232f80
> [375832.686163]  88035fc83db8 810bc592 88035fc83dc8
> 880351758000
> [375832.694139]  01001002  b802f4bd
> a024e6f0
> [375832.702154] Call Trace:
> [375832.704844]  
> [375832.707018]  [] rqsav_raw_spin_lock_ie+0x31/0x40

rqsav_raw_spin_lock_ie??
Is this some proprietary spinlock implementation?

> [375832.713970]  [] __wake_up+0x32/0x70
> [375832.719444]  [] ? tipc_recv_stream+0x370/0x370
[tipc]
> [375832.726589]  [] sock_def_wakeup+0x30/0x40
> [375832.732566]  [] tipc_sk_timeout+0x148/0x180 [tipc]
> [375832.739388]  [] ? tipc_recv_stream+0x370/0x370
[tipc]
> [375832.746507]  [] call_timer_fn+0x44/0x110
> [375832.752378]  [] ? cascade+0x4a/0x80
> [375832.757848]  [] ? tipc_recv_stream+0x370/0x370
[tipc]
> [375832.764871]  [] run_timer_softirq+0x22c/0x280
> [375832.771175]  [] __do_softirq+0xc8/0x260
> [375832.776958]  [] irq_exit+0x83/0xb0
> [375832.782369]  [] do_IRQ+0x65/0xf0
> [375832.787607]  [] common_interrupt+0x7f/0x7f
> [375832.793709]  
> [375832.795803]  [] ? cpuidle_enter_state+0xad/0x200
> [375832.802765]  [] ? cpuidle_enter_state+0x91/0x200
> [375832.809338]  [] cpuidle_enter+0x17/0x20
> [375832.815155]  [] call_cpuidle+0x37/0x60
> [375832.821184]  [] ? cpuidle_select+0x13/0x20
> [375832.827249]  [] cpu_startup_entry+0x211/0x2d0
> [375832.833535]  [] start_secondary+0x103/0x130
> [375832.839759] Code: 87 47 02 c1 e0 10 85 c0 74 38 48 89 c2 c1 e8 12
> 48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 c0 5f 01 00 48 03 14 c5
> 00 b2 d1 81 <48> 89 0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 85 c0 74 f7
> eb 02
> [375832.861151] RIP  []
queued_spin_lock_slowpath+0xe6/0x160
> [375832.868607]  RSP 
> [375832.872371] CR2: 01a400015ff4
> [375832.876408] ---[ end trace f12e0074b180a165 ]---
> [375832.881433] Kernel panic - not syncing: Fatal exception in interrupt
> [375832.888391] Kernel Offset: disabled
> [375832.891968] ---[ end Kernel panic - not syncing: Fatal exception
> in interrupt
>
>
--
> Find and fix application performance issues faster with Applications
Manager
> Applications Manager provides deep performance insights into multiple
tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> ___
> tipc-discussion mailing list
> tipc-discussion@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tipc-discussion
--
Find and fix application performance issues faster 

[tipc-discussion] tipc: tipc_recv_stream with kernel panic

[GUNA]

The following TIPC traces were collected after cards were forced to
reboot to recover them.
Kernel: 4.4.0 is running and applied some latest TIPC patches.

[   65.954959] sm-msp-queue[1279]: unable to qualify my own domain
name (dcsx5testslot3) -- using short name
[  632.098785] perf interrupt took too long (2505 > 2500), lowering
kernel.perf_event_max_sample_rate to 5
[ 5880.428123] perf interrupt took too long (5585 > 5000), lowering
kernel.perf_event_max_sample_rate to 25000
[17934.014969] CE: hpet increased min_delta_ns to 20115 nsec
[38956.721789] CE: hpet4 increased min_delta_ns to 20115 nsec
[46927.872827] hrtimer: interrupt took 63361 ns
[101662.241093] CE: hpet2 increased min_delta_ns to 20115 nsec
[245973.044600] CE: hpet6 increased min_delta_ns to 20115 nsec
[368639.565040] show_signal_msg: 6 callbacks suppressed
[375832.498126] BUG: unable to handle kernel paging request at 01a400015ff4
[375832.505300] IP: [] queued_spin_lock_slowpath+0xe6/0x160
[375832.512394] PGD 0
[375832.514657] Oops: 0002 [#1] SMP
[375832.518306] Modules linked in: nf_log_ipv6 nf_log_ipv4
nf_log_common xt_LOG sctp libcrc32c e1000e tipc udp_tunnel
ip6_udp_tunnel 8021q garp iTCO_wdt xt_physdev br_netfilter bridge stp
llc nf_conntrack_ipv4 nf_defrag_ipv4 ipmiq_drv(O) sio_mmc(O)
ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
nf_conntrack lockd ip6table_filter event_drv(O) ip6_tables grace
pt_timer_info(O) ddi(O) usb_storage ixgbe igb i2c_i801
iTCO_vendor_support i2c_algo_bit ioatdma intel_ips i2c_core pcspkr
sunrpc ptp mdio dca pps_core lpc_ich tpm_tis mfd_core tpm [last
unloaded: iTCO_wdt]
[375832.573693] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G   O
 4.4.0 #14
[375832.581385] Hardware name: PT AMC124/Base Board Product Name, BIOS
LGNAJFIP.PTI.0012.P15 01/15/2014
[375832.591028] task: 880351a89b40 ti: 880351a9 task.ti:
880351a9
[375832.599026] RIP: 0010:[]  []
queued_spin_lock_slowpath+0xe6/0x160
[375832.608964] RSP: 0018:88035fc83d58  EFLAGS: 00010002
[375832.614825] RAX: 1447 RBX: 0292 RCX:
88035fc95fc0
[375832.622743] RDX: 01a400015ff4 RSI: 0014 RDI:
880351232f80
[375832.630567] RBP: 88035fc83d58 R08: 0101 R09:
0004
[375832.638348] R10:  R11:  R12:
01001002
[375832.645919] R13: 0001 R14:  R15:

[375832.653610] FS:  () GS:88035fc8()
knlGS:
[375832.662317] CS:  0010 DS:  ES:  CR0: 8005003b
[375832.668483] CR2: 01a400015ff4 CR3: 01c0a000 CR4:
06e0
[375832.676133] Stack:
[375832.678344]  88035fc83d78 816de2c1 88034a8bba60
880351232f80
[375832.686163]  88035fc83db8 810bc592 88035fc83dc8
880351758000
[375832.694139]  01001002  b802f4bd
a024e6f0
[375832.702154] Call Trace:
[375832.704844]  
[375832.707018]  [] rqsav_raw_spin_lock_ie+0x31/0x40
[375832.713970]  [] __wake_up+0x32/0x70
[375832.719444]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
[375832.726589]  [] sock_def_wakeup+0x30/0x40
[375832.732566]  [] tipc_sk_timeout+0x148/0x180 [tipc]
[375832.739388]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
[375832.746507]  [] call_timer_fn+0x44/0x110
[375832.752378]  [] ? cascade+0x4a/0x80
[375832.757848]  [] ? tipc_recv_stream+0x370/0x370 [tipc]
[375832.764871]  [] run_timer_softirq+0x22c/0x280
[375832.771175]  [] __do_softirq+0xc8/0x260
[375832.776958]  [] irq_exit+0x83/0xb0
[375832.782369]  [] do_IRQ+0x65/0xf0
[375832.787607]  [] common_interrupt+0x7f/0x7f
[375832.793709]  
[375832.795803]  [] ? cpuidle_enter_state+0xad/0x200
[375832.802765]  [] ? cpuidle_enter_state+0x91/0x200
[375832.809338]  [] cpuidle_enter+0x17/0x20
[375832.815155]  [] call_cpuidle+0x37/0x60
[375832.821184]  [] ? cpuidle_select+0x13/0x20
[375832.827249]  [] cpu_startup_entry+0x211/0x2d0
[375832.833535]  [] start_secondary+0x103/0x130
[375832.839759] Code: 87 47 02 c1 e0 10 85 c0 74 38 48 89 c2 c1 e8 12
48 c1 ea 0c 83 e8 01 83 e2 30 48 98 48 81 c2 c0 5f 01 00 48 03 14 c5
00 b2 d1 81 <48> 89 0a 8b 41 08 85 c0 75 0d f3 90 8b 41 08 85 c0 74 f7
eb 02
[375832.861151] RIP  [] queued_spin_lock_slowpath+0xe6/0x160
[375832.868607]  RSP 
[375832.872371] CR2: 01a400015ff4
[375832.876408] ---[ end trace f12e0074b180a165 ]---
[375832.881433] Kernel panic - not syncing: Fatal exception in interrupt
[375832.888391] Kernel Offset: disabled
[375832.891968] ---[ end Kernel panic - not syncing: Fatal exception
in interrupt

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!