Hi guys,
we're seeing unexpected crashes from kernel 4.15 to 4.18.17, using IPsec
VTI interfaces, on several vpn hosts, since upgrade from 4.4.
Attached, the offended oops against 4.18.
Output of decodedecode:
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00
00 8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe
<f6> 40 38 04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
All code
========
0: 8b 44 24 70 mov 0x70(%rsp),%eax
4: 0f c8 bswap %eax
6: 89 87 b4 00 00 00 mov %eax,0xb4(%rdi)
c: 48 8b 86 20 05 00 00 mov 0x520(%rsi),%rax
13: 8b 80 f8 14 00 00 mov 0x14f8(%rax),%eax
19: 85 c0 test %eax,%eax
1b: 75 05 jne 0x22
1d: 48 85 d2 test %rdx,%rdx
20: 74 0e je 0x30
22: 48 8b 43 58 mov 0x58(%rbx),%rax
26: 48 83 e0 fe and $0xfffffffffffffffe,%rax
2a:* f6 40 38 04 testb $0x4,0x38(%rax) <--
trapping instruction
2e: 74 7d je 0xad
30: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
37: 49 8b 44 24 20 mov 0x20(%r12),%rax
3c: 48 rex.W
3d: 39 .byte 0x39
3e: 86 20 xchg %ah,(%rax)
Code starting with the faulting instruction
===========================================
0: f6 40 38 04 testb $0x4,0x38(%rax)
4: 74 7d je 0x83
6: 44 89 b3 b4 00 00 00 mov %r14d,0xb4(%rbx)
d: 49 8b 44 24 20 mov 0x20(%r12),%rax
12: 48 rex.W
13: 39 .byte 0x39
14: 86 20 xchg %ah,(%rax)
if my understanding is correct, we fail here:
/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
0x0000000000000b19 <+185>: testb $0x4,0x38(%rax)
0x0000000000000b1d <+189>: je 0xb9c <vti_rcv_cb+316>
(gdb) list *0x0000000000000b19
0xb19 is in vti_rcv_cb
(/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/include/net/xfrm.h:1169).
1164 int ndir = dir | (reverse ? XFRM_POLICY_MASK + 1 : 0);
1165
1166 if (sk && sk->sk_policy[XFRM_POLICY_IN])
1167 return __xfrm_policy_check(sk, ndir, skb, family);
1168
1169 return (!net->xfrm.policy_count[dir] && !skb->sp) ||
1170 (skb_dst(skb)->flags & DST_NOPOLICY) ||
1171 __xfrm_policy_check(sk, ndir, skb, family);
1172 }
1173
I really have hard time to understand why skb seem to be freed twice.
I'm not able to repeat the bug in lab, but it happened regulary in prod,
seem to depend of the workload.
Any help will be appreciated.
Let me know if you need further informations.
Regards,
Jean-Philippe
[ 31.154360] BUG: unable to handle kernel NULL pointer dereference at
0000000000000038
[ 31.162233] PGD 0 P4D 0
[ 31.164786] Oops: 0000 [#1] SMP PTI
[ 31.168291] CPU: 5 PID: 42 Comm: ksoftirqd/5 Not tainted 4.18.0-11-generic
#12~18.04.1-Ubuntu
[ 31.176854] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS
1.0b 11/21/2016
[ 31.184980] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.189962] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00
8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38
04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.208916] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.214160] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.221328] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.228485] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.235643] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.242803] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.249964] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000)
knlGS:0000000000000000
[ 31.258077] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.263848] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.271004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.278163] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.285320] Call Trace:
[ 31.287789] xfrm4_rcv_cb+0x4a/0x70
[ 31.291297] xfrm_input+0x58f/0x8f0
[ 31.294807] vti_input+0xaa/0x110 [ip_vti]
[ 31.298926] vti_rcv+0x33/0x3c [ip_vti]
[ 31.302783] xfrm4_esp_rcv+0x39/0x50
[ 31.306375] ip_local_deliver_finish+0x62/0x200
[ 31.310923] ip_local_deliver+0xdf/0xf0
[ 31.314775] ? ip_rcv_finish+0x420/0x420
[ 31.318718] ip_rcv_finish+0x126/0x420
[ 31.322486] ip_rcv+0x28f/0x360
[ 31.325655] ? inet_del_offload+0x40/0x40
[ 31.329686] __netif_receive_skb_core+0x48c/0xb70
[ 31.334413] ? kmem_cache_alloc+0xb4/0x1d0
[ 31.338532] ? __build_skb+0x2b/0xf0
[ 31.342128] __netif_receive_skb+0x18/0x60
[ 31.346244] ? __netif_receive_skb+0x18/0x60
[ 31.350536] netif_receive_skb_internal+0x45/0xe0
[ 31.355263] napi_gro_receive+0xc5/0xf0
[ 31.359141] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 31.364476] ? skb_release_all+0x24/0x30
[ 31.368430] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 31.373432] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 31.378333] ? __switch_to_asm+0x34/0x70
[ 31.382270] ? __switch_to_asm+0x40/0x70
[ 31.386214] ? __switch_to_asm+0x34/0x70
[ 31.391056] ? __switch_to_asm+0x40/0x70
[ 31.395905] ? __switch_to_asm+0x34/0x70
[ 31.400743] net_rx_action+0x140/0x3a0
[ 31.405379] ? __switch_to+0xad/0x500
[ 31.409887] __do_softirq+0xe4/0x2bb
[ 31.414448] run_ksoftirqd+0x2b/0x40
[ 31.418862] smpboot_thread_fn+0xfc/0x170
[ 31.423700] kthread+0x121/0x140
[ 31.427701] ? sort_range+0x30/0x30
[ 31.432040] ? kthread_create_worker_on_cpu+0x70/0x70
[ 31.437816] ret_from_fork+0x35/0x40
[ 31.442219] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel
xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4
af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q garp mrp stp
llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp
xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4
nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle xt_CT
nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel
kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev mei_me
intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf ipmi_msghandler
acpi_pad mac_hid sch_fq_codel
[ 31.519488] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea
aes_x86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper
fb_sys_fops mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 31.578877] CR2: 0000000000000038
[ 31.583249] ---[ end trace c4bada38847a0075 ]---
[ 31.737166] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 31.737167] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00
8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38
04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 31.737209] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 31.737212] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 31.737213] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 31.737216] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 31.737219] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 31.737220] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 31.737222] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000)
knlGS:0000000000000000
[ 31.737224] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 31.737225] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 31.737227] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 31.737228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 31.737230] Kernel panic - not syncing: Fatal exception in interrupt
[ 31.737264] Kernel Offset: 0x3c00000 from 0xffffffff81000000 (relocation
range: 0xffffffff80000000-0xffffffffbfffffff)
[ 36.558279] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
[ 36.566769] ------------[ cut here ]------------
[ 36.572306] sched: Unexpected reschedule of offline CPU#3!
[ 36.578714] WARNING: CPU: 5 PID: 42 at
/build/linux-hwe-edge-yHKLQJ/linux-hwe-edge-4.18.0/arch/x86/kernel/smp.c:128
native_smp_send_resched
ule+0x3a/0x40
[ 36.594262] Modules linked in: esp6 authenc echainiv xfrm6_mode_tunnel
xfrm4_mode_tunnel xfrm_user xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp
esp4 ah4 af_key xfrm_algo ip_vti ip_tunnel ip6_vti ip6_tunnel tunnel6 8021q
garp mrp stp llc bonding ipt_REJECT nf_reject_ipv4 nfnetlink_log n
fnetlink xt_NFLOG xt_hl xt_limit xt_nat xt_TCPMSS xt_HL xt_comment xt_tcpudp
xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack
_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_connmark xt_mark iptable_mangle
xt_CT nf_conntrack xt_addrtype iptable_raw bpfilter ipmi_ssif gpio_
ich intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel
kvm irqbypass intel_cstate intel_rapl_perf input_leds joydev m
ei_me intel_pch_thermal ioatdma mei lpc_ich ipmi_si ipmi_devintf
ipmi_msghandler acpi_pad mac_hid sch_fq_codel
[ 36.673307] ib_iser rdma_cm iw_cm ib_cm iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compres
s raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid0 multipath linear mlx5_ib ib_uverbs ib
_core raid1 hid_generic usbhid hid crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel ast pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x
86_64 sysfillrect mxm_wmi crypto_simd sysimgblt cryptd glue_helper fb_sys_fops
mlx5_core ixgbe igb mpt3sas drm ahci tls libahci i2c_algo_bit m
lxfw raid_class dca devlink mdio scsi_transport_sas wmi
[ 36.733827] CPU: 5 PID: 42 Comm: ksoftirqd/5 Tainted: G D
4.18.0-11-generic #12~18.04.1-Ubuntu
[ 36.745027] Hardware name: Supermicro Super Server/X10SDV-4C-7TP4F, BIOS
1.0b 11/21/2016
[ 36.754392] RIP: 0010:native_smp_send_reschedule+0x3a/0x40
[ 36.761143] Code: c6 62 01 73 17 48 8b 05 24 d2 17 01 be fd 00 00 00 48 8b
40 30 e8 96 96 ba 00 5d c3 89 fe 48 c7 c7 90 2a cd 85 e8 26 4a 0
3 00 <0f> 0b 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 20 65 48
[ 36.782546] RSP: 0018:ffff9a35bfd43b68 EFLAGS: 00010082
[ 36.789019] RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000006
[ 36.797379] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff9a35bfd564b0
[ 36.805721] RBP: ffff9a35bfd43b68 R08: 00000000000004b7 R09: 0000000000cdcdcd
[ 36.814046] R10: 0000000000000324 R11: 00000000ffffffff R12: ffff9a35bfce2c40
[ 36.822350] R13: ffff9a3598e35c00 R14: 0000000000000008 R15: 0000000000000003
[ 36.830637] FS: 0000000000000000(0000) GS:ffff9a35bfd40000(0000)
knlGS:0000000000000000
[ 36.839869] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.846742] CR2: 0000000000000038 CR3: 000000041a40a003 CR4: 00000000003606e0
[ 36.855009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 36.863275] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 36.871510] Call Trace:
[ 36.875048] <IRQ>
[ 36.878136] resched_curr+0x5d/0xc0
[ 36.882689] check_preempt_wakeup+0x130/0x240
[ 36.888116] check_preempt_curr+0x2d/0x90
[ 36.893184] ttwu_do_wakeup+0x1e/0x140
[ 36.897984] ttwu_do_activate+0x77/0x80
[ 36.902857] try_to_wake_up+0x1d6/0x450
[ 36.907726] ? __netif_receive_skb_core+0x48c/0xb70
[ 36.913625] default_wake_function+0x12/0x20
[ 36.918914] __wake_up_common+0x73/0x130
[ 36.923851] __wake_up_locked+0x16/0x20
[ 36.928706] ep_poll_callback+0xcb/0x2b0
[ 36.933634] __wake_up_common+0x73/0x130
[ 36.938549] __wake_up_common_lock+0x80/0xc0
[ 36.943787] ? tick_sched_do_timer+0x60/0x60
[ 36.949011] __wake_up+0x13/0x20
[ 36.953190] wake_up_klogd_work_func+0x40/0x60
[ 36.958560] irq_work_run_list+0x52/0x80
[ 36.963388] irq_work_tick+0x3b/0x50
[ 36.967841] update_process_times+0x42/0x60
[ 36.972883] tick_sched_handle+0x25/0x70
[ 36.977644] tick_sched_timer+0x3c/0x80
[ 36.982285] __hrtimer_run_queues+0x10f/0x280
[ 36.987429] hrtimer_interrupt+0xe7/0x240
[ 36.992202] ? rcu_irq_exit+0x1d/0x20
[ 36.996602] smp_apic_timer_interrupt+0x6f/0x130
[ 37.001929] apic_timer_interrupt+0xf/0x20
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6
20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 0
0 00 <31> db e8 2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX:
ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.006712] </IRQ>
[ 37.009476] RIP: 0010:panic+0x1fe/0x244
[ 37.013951] Code: eb a6 83 3d 37 05 8f 01 00 74 05 e8 d0 73 02 00 48 c7 c6
20 f1 57 86 48 c7 c7 c8 cd cd 85 e8 f0 78 06 00 fb 66 0f 1f 44 00 00 <31> db e8
2f c6 0d 00 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 df 04 8f
[ 37.034119] RSP: 0018:ffffbc61832e3668 EFLAGS: 00000286 ORIG_RAX:
ffffffffffffff13
[ 37.042324] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[ 37.050086] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff9a35bfd564b0
[ 37.057835] RBP: ffffbc61832e36e0 R08: 00000000000004b5 R09: 0000000000cdcdcd
[ 37.065570] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000000
[ 37.073287] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000000
[ 37.080999] ? panic+0x1f7/0x244
[ 37.084812] oops_end+0xce/0xe0
[ 37.088531] no_context+0x17c/0x400
[ 37.092585] ? skcipher_walk_first+0x4c/0x110
[ 37.097500] __bad_area_nosemaphore+0x115/0x1d0
[ 37.102591] ? cbc_decrypt+0xb1/0xe0 [aesni_intel]
[ 37.107926] bad_area_nosemaphore+0x14/0x20
[ 37.112654] __do_page_fault+0xd4/0x4d0
[ 37.117030] do_page_fault+0x2d/0xf0
[ 37.121143] ? skb_copy_bits+0x61/0x260
[ 37.125511] page_fault+0x1e/0x30
[ 37.129362] RIP: 0010:vti_rcv_cb+0xb9/0x1a0 [ip_vti]
[ 37.134864] Code: 8b 44 24 70 0f c8 89 87 b4 00 00 00 48 8b 86 20 05 00 00
8b 80 f8 14 00 00 85 c0 75 05 48 85 d2 74 0e 48 8b 43 58 48 83 e0 fe <f6> 40 38
04 74 7d 44 89 b3 b4 00 00 00 49 8b 44 24 20 48 39 86 20
[ 37.154892] RSP: 0018:ffffbc61832e3920 EFLAGS: 00010246
[ 37.160711] RAX: 0000000000000000 RBX: ffff9a3504964a00 RCX: 0000000000000002
[ 37.168451] RDX: ffff9a351add4080 RSI: ffff9a351aa08000 RDI: ffff9a3504964a00
[ 37.176196] RBP: ffffbc61832e3940 R08: 0000000000000004 R09: ffffffffc0aa612b
[ 37.183950] R10: 0008f09b99881884 R11: 1884bd4e2d6b1fac R12: ffff9a3507b31900
[ 37.191716] R13: ffff9a3507b31000 R14: 0000000000000000 R15: ffff9a3504964a00
[ 37.199481] ? esp_input_done2+0x6b/0x320 [esp4]
[ 37.204740] xfrm4_rcv_cb+0x4a/0x70
[ 37.208873] xfrm_input+0x58f/0x8f0
[ 37.212996] vti_input+0xaa/0x110 [ip_vti]
[ 37.217735] vti_rcv+0x33/0x3c [ip_vti]
[ 37.222209] xfrm4_esp_rcv+0x39/0x50
[ 37.226424] ip_local_deliver_finish+0x62/0x200
[ 37.231599] ip_local_deliver+0xdf/0xf0
[ 37.236072] ? ip_rcv_finish+0x420/0x420
[ 37.240635] ip_rcv_finish+0x126/0x420
[ 37.245025] ip_rcv+0x28f/0x360
[ 37.248801] ? inet_del_offload+0x40/0x40
[ 37.253457] __netif_receive_skb_core+0x48c/0xb70
[ 37.258807] ? kmem_cache_alloc+0xb4/0x1d0
[ 37.263541] ? __build_skb+0x2b/0xf0
[ 37.267749] __netif_receive_skb+0x18/0x60
[ 37.272483] ? __netif_receive_skb+0x18/0x60
[ 37.277391] netif_receive_skb_internal+0x45/0xe0
[ 37.282732] napi_gro_receive+0xc5/0xf0
[ 37.287232] mlx5e_handle_rx_cqe+0x1b2/0x5d0 [mlx5_core]
[ 37.293170] ? skb_release_all+0x24/0x30
[ 37.297708] mlx5e_poll_rx_cq+0xd3/0x990 [mlx5_core]
[ 37.303281] mlx5e_napi_poll+0x9b/0xc60 [mlx5_core]
[ 37.308752] ? __switch_to_asm+0x34/0x70
[ 37.313266] ? __switch_to_asm+0x40/0x70
[ 37.317768] ? __switch_to_asm+0x34/0x70
[ 37.322259] ? __switch_to_asm+0x40/0x70
[ 37.326742] ? __switch_to_asm+0x34/0x70
[ 37.331214] net_rx_action+0x140/0x3a0
[ 37.335519] ? __switch_to+0xad/0x500
[ 37.339728] __do_softirq+0xe4/0x2bb
[ 37.343852] run_ksoftirqd+0x2b/0x40
[ 37.347971] smpboot_thread_fn+0xfc/0x170
[ 37.352522] kthread+0x121/0x140
[ 37.356282] ? sort_range+0x30/0x30
[ 37.360303] ? kthread_create_worker_on_cpu+0x70/0x70
[ 37.365895] ret_from_fork+0x35/0x40
[ 37.370010] ---[ end trace c4bada38847a0076 ]---
