We are debugging an issue with fib6_node_lookup_1().
We use a 4.16 based kernel, and we have back ported most upstream
patches in ip6_fib.{c.h}. The only major differences I can spot are
8b7f2731bd68d83940714ce92381d1a72596407c
c3506372277779fccbffee2475400fcd689d5738
I guess the issue is not related to these two fixes.
After staring at the call trace and disassembly code (attached below)
I guess this is a use-after-free issue in (or right after) the lookup
loop:
for (;;) {
struct fib6_node *next;
dir = addr_bit_set(args->addr, fn->fn_bit);
next = dir ? rcu_dereference(fn->right) :
rcu_dereference(fn->left);
if (next) {
fn = next;
continue;
}
break;
}
I guess this probably also happens to latest upstream. I haven't
tested this with upstream kernel (or net tree) yet, because we
can only trigger this about once a week on 100 servers.
Does this look familiar? Any comments and/or suggestions are highly
appreciated.
Thanks,
Song
Bug stack trace:
[354764.457916] BUG: unable to handle kernel
[354764.466125] paging request
[354764.471720] at 00000000f60fc318
[354764.478360] IP: fib6_node_lookup_1+0x29/0x130
[354764.487249] PGD 800000010f725067
[354764.494062] P4D 800000010f725067
[354764.500878] PUD 0
[354764.505087] Oops: 0000 [#1] SMP PTI
[354764.512245] Modules linked in:
[354764.518536] udp_diag
[354764.523266] act_gact
[354764.527997] cls_bpf
[354764.532557] tcp_diag
[354764.537291] inet_diag
[354764.542200] nfsv3
[354764.546409] nfs
[354764.550273] fscache
[354764.554834] ip6table_raw
[354764.560260] ip6table_filter
[354764.566208] xt_DSCP
[354764.570765] iptable_raw
[354764.576020] iptable_filter
[354764.581790] ip6table_mangle
[354764.587738] iptable_mangle
[354764.593505] sb_edac
[354764.598058] x86_pkg_temp_thermal
[354764.604872] intel_powerclamp
[354764.610992] coretemp
[354764.615723] kvm_intel
[354764.620628] kvm
[354764.624494] irqbypass
[354764.629399] iTCO_wdt
[354764.634132] iTCO_vendor_support
[354764.640772] i2c_i801
[354764.645507] lpc_ich
[354764.650064] efivars
[354764.654619] mfd_core
[354764.659353] ipmi_si
[354764.663911] ipmi_devintf
[354764.669341] ipmi_msghandler
[354764.675281] acpi_cpufreq
[354764.680711] button
[354764.685096] sch_fq_codel
[354764.690520] nfsd
[354764.694557] nfs_acl
[354764.699118] lockd
[354764.703330] auth_rpcgss
[354764.708588] oid_registry
[354764.714006] grace
[354764.718213] sunrpc
[354764.722590] fuse
[354764.726626] loop
[354764.730661] efivarfs
[354764.735395] autofs4
[354764.739957] CPU: 5 PID: 3460038 Comm: java Not tainted
4.16.0-14_fbk2_1455_g6bcb99c57db6 #14
[354764.756996] Hardware name: Wiwynn Leopard-Orv2/Leopard-DDR BW, BIOS LBM03
06/02/2016
[354764.773001] RIP: 0010:fib6_node_lookup_1+0x29/0x130
[354764.782929] RSP: 0018:ffffc9003f0bb730 EFLAGS: 00010206
[354764.793557] RAX: ffff883fc131a000 RBX: 00000000f60fc300 RCX:
00000000ffffffe4
[354764.807999] RDX: 0000000000000010 RSI: 0000000000000001 RDI:
ffffc9003f0bb8f0
[354764.822436] RBP: ffffc9003f0bb750 R08: 0000000000000002 R09:
0000000000000004
[354764.836877] R10: ffffc9003f0bb7a8 R11: ffff883ff7795780 R12:
ffffffff82305080
[354764.851317] R13: 0000000000000002 R14: 0000000000000000 R15:
0000000000000000
[354764.865765] FS: 00007f8defcfc700(0000) GS:ffff881fff940000(0000)
knlGS:0000000000000000
[354764.882119] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[354764.893800] CR2: 00000000f60fc318 CR3: 0000000f68cae006 CR4:
00000000003606e0
[354764.908235] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[354764.922671] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[354764.937109] Call Trace:
[354764.942195] fib6_node_lookup+0x67/0x90
[354764.950042] ? fib6_table_lookup+0x43/0x2f0
[354764.958587] fib6_table_lookup+0x43/0x2f0
[354764.966794] ip6_pol_route+0x43/0x360
[354764.974294] ? ip6_pol_route_input+0x20/0x20
[354764.983016] fib6_rule_lookup+0x85/0x140
[354764.991050] ? ip6t_do_table+0x331/0x6b0
[354764.999089] ? ip6_route_output_flags+0xa3/0xc0
[354765.008342] ip6_route_me_harder+0xab/0x280
[354765.016889] ip6table_mangle_hook+0xd4/0x110 [ip6table_mangle]
[354765.028754] ? nf_hook_slow+0x43/0xc0
[354765.036269] nf_hook_slow+0x43/0xc0
[354765.043445] nf_hook+0x6e/0xc0
[354765.049731] ? ac6_proc_exit+0x20/0x20
[354765.057412] ip6_xmit+0x28a/0x500
[354765.064225] ? ac6_proc_exit+0x20/0x20
[354765.071902] ? inet6_csk_route_socket+0x10f/0x1c0
[354765.081495] ? update_group_capacity+0x23/0x1e0
[354765.090749] inet6_csk_xmit+0x82/0xd0
[354765.098277] tcp_transmit_skb+0x51d/0x9d0
[354765.106495] tcp_write_xmit+0x1bd/0xf40
[354765.114359] ? _copy_from_iter_full+0x9c/0x240
[354765.123444] tcp_sendmsg_locked+0x2c2/0xdd0
[354765.131991] tcp_sendmsg+0x27/0x40
[354765.138991] sock_sendmsg+0x36/0x40
[354765.146167] sock_write_iter+0x84/0xd0
Disassemble of the fib6_node_lookup_1:
Dump of assembler code for function fib6_node_lookup_1:
0xffffffff818b3c70 <+0>: callq 0xffffffff81a01610 <__fentry__>
0xffffffff818b3c75 <+5>: mov (%rsi),%eax
0xffffffff818b3c77 <+7>: test %eax,%eax
0xffffffff818b3c79 <+9>: je 0xffffffff818b3d94
<fib6_node_lookup_1+292>
0xffffffff818b3c7f <+15>: push %r12
0xffffffff818b3c81 <+17>: push %rbp
0xffffffff818b3c82 <+18>: mov %rsi,%rbp
0xffffffff818b3c85 <+21>: push %rbx
0xffffffff818b3c86 <+22>: mov %rdi,%rbx
0xffffffff818b3c89 <+25>: mov 0x8(%rsi),%rdi
0xffffffff818b3c8d <+29>: mov $0x1,%esi
0xffffffff818b3c92 <+34>: movzwl 0x28(%rbx),%ecx
0xffffffff818b3c96 <+38>: mov %esi,%edx
0xffffffff818b3c98 <+40>: mov %ecx,%eax
0xffffffff818b3c9a <+42>: xor $0xffffffe7,%ecx
0xffffffff818b3c9d <+45>: sar $0x5,%eax
0xffffffff818b3ca0 <+48>: shl %cl,%edx
0xffffffff818b3ca2 <+50>: cltq
0xffffffff818b3ca4 <+52>: test %edx,(%rdi,%rax,4)
0xffffffff818b3ca7 <+55>: je 0xffffffff818b3cb7
<fib6_node_lookup_1+71>
0xffffffff818b3ca9 <+57>: mov 0x10(%rbx),%rax
0xffffffff818b3cad <+61>: test %rax,%rax
0xffffffff818b3cb0 <+64>: je 0xffffffff818b3cc0
<fib6_node_lookup_1+80>
0xffffffff818b3cb2 <+66>: mov %rax,%rbx
0xffffffff818b3cb5 <+69>: jmp 0xffffffff818b3c92
<fib6_node_lookup_1+34>
0xffffffff818b3cb7 <+71>: mov 0x8(%rbx),%rax
0xffffffff818b3cbb <+75>: test %rax,%rax
0xffffffff818b3cbe <+78>: jne 0xffffffff818b3cb2
<fib6_node_lookup_1+66>
0xffffffff818b3cc0 <+80>: test %rbx,%rbx
0xffffffff818b3cc3 <+83>: je 0xffffffff818b3d17
<fib6_node_lookup_1+167>
0xffffffff818b3cc5 <+85>: mov $0xffffffffffffffff,%r12
0xffffffff818b3ccc <+92>: jmp 0xffffffff818b3d02
<fib6_node_lookup_1+146>
0xffffffff818b3cce <+94>: mov 0x20(%rbx),%rax
0xffffffff818b3cd2 <+98>: test %rax,%rax
0xffffffff818b3cd5 <+101>: je 0xffffffff818b3cf2
<fib6_node_lookup_1+130>
0xffffffff818b3cd7 <+103>: movslq 0x0(%rbp),%rdx
0xffffffff818b3cdb <+107>: mov 0x8(%rbp),%rsi
0xffffffff818b3cdf <+111>: add %rdx,%rax
0xffffffff818b3ce2 <+114>: mov 0x10(%rax),%edx
0xffffffff818b3ce5 <+117>: cmp $0x3f,%edx
0xffffffff818b3ce8 <+120>: jbe 0xffffffff818b3d1e
<fib6_node_lookup_1+174>
0xffffffff818b3cea <+122>: mov (%rsi),%rcx
0xffffffff818b3ced <+125>: cmp %rcx,(%rax)
0xffffffff818b3cf0 <+128>: je 0xffffffff818b3d52
<fib6_node_lookup_1+226>
0xffffffff818b3cf2 <+130>: movzwl 0x2a(%rbx),%eax
0xffffffff818b3cf6 <+134>: test $0x2,%al
0xffffffff818b3cf8 <+136>: jne 0xffffffff818b3d17
<fib6_node_lookup_1+167>
0xffffffff818b3cfa <+138>: mov (%rbx),%rbx
