> On May 22, 2024, at 12:17 PM, Bjoern A. Zeeb <bzeeb-li...@lists.zabbadoz.net>
> wrote:
>
> Hi,
>
> sorry, I cannot dump; this is a diskless and netdump does not do IPv6;
> needless to say that would be funny in this case anyway; unfortunately
> I have also already re-compiled the kernel so I can only look things up
> approx.
>
> FreeBSD main from May 13 (f3eeeb959c9b00c89a2e1ff009c78162eb398656).
>
> I assume we lost the ifp from a destroy of a cloned interface in ip6_output()
> between lines 806 and 811?
>
>
> Kernel page fault with the following non-sleepable locks held:
> exclusive rw rawinp (rawinp) r = 0 (0xfffff80002a6e1a0) locked @
> /usr/src/sys/netinet6/raw_ip6.c:393
> stack backtrace:
> #0 0xffffffff80bb679c at witness_debugger+0x6c
> #1 0xffffffff80bb7979 at witness_warn+0x3e9
> #2 0xffffffff81061d10 at trap_pfault+0x80
> #3 0xffffffff81033878 at calltrap+0x8
> #4 0xffffffff80d99228 at rip6_send+0x5a8
> #5 0xffffffff80bf570e at sosend_generic+0x5ee
> #6 0xffffffff80bf5c49 at sousrsend+0x79
> #7 0xffffffff80bfbd5c at kern_sendit+0x1bc
> #8 0xffffffff80bfc073 at sendit+0x1b3
> #9 0xffffffff80bfc1ab at sys_sendmsg+0x5b
> #10 0xffffffff81062638 at amd64_syscall+0x158
> #11 0xffffffff8103418b at fast_syscall_common+0xf8
> Created wlan(4) interfaces: wlan
Note the creation of wlan, and a following ICMP6 (ping6) packet.
> Fatal trap 12: page fault while in kernel mode
> cpuid = 2; apic id = 02
> fault virtual address = 0x0
> 30.
> fault code = supervisor read data, page not present
> instruction pointer = 0x20:0xffffffff80d821bd
> stack pointer = 0x28:0xfffffe00468ba850
> frame pointer = 0x28:0xfffffe00468baa00
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 1073 (ping6)
> rdi: ffffffff81be2c00 rsi: fffffe00468ba884 rdx: 00000000030002ff
> rcx: 0000000000001c1c r8: 0000000000000028 r9: 0000000000000003
> rax: 0000000000000000 rbx: 0000000000000000 rbp: fffffe00468baa00
> r10: 000000000000aa01 r11: 00000000000000ff r12: fffff800017d3000
> r13: fffff8000286df20 r14: 0000000000000000 r15: fffff80001f1059c
> trap number = 12
> panic: page fault
> cpuid = 2
> time = 1716352160
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00468ba520
> vpanic() at vpanic+0x13f/frame 0xfffffe00468ba650
> panic() at panic+0x43/frame 0xfffffe00468ba6b0
> trap_fatal() at trap_fatal+0x40b/frame 0xfffffe00468ba710
> trap_pfault() at trap_pfault+0xa0/frame 0xfffffe00468ba780
> calltrap() at calltrap+0x8/frame 0xfffffe00468ba780
> --- trap 0xc, rip = 0xffffffff80d821bd, rsp = 0xfffffe00468ba850, rbp =
> 0xfffffe00468baa00 ---
> ip6_output() at ip6_output+0xb5d/frame 0xfffffe00468baa00
> rip6_send() at rip6_send+0x5a8/frame 0xfffffe00468babb0
> sosend_generic() at sosend_generic+0x5ee/frame 0xfffffe00468bac60
> sousrsend() at sousrsend+0x79/frame 0xfffffe00468bacc0
> kern_sendit() at kern_sendit+0x1bc/frame 0xfffffe00468bad50
> sendit() at sendit+0x1b3/frame 0xfffffe00468bada0
> sys_sendmsg() at sys_sendmsg+0x5b/frame 0xfffffe00468bae00
> amd64_syscall() at amd64_syscall+0x158/frame 0xfffffe00468baf30
> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00468baf30
> --- syscall (28, FreeBSD ELF64, sendmsg), rip = 0x2959dfacc85a, rsp =
> 0x2959dae92668, rbp = 0x2959dae926c0 ---
> KDB: enter: panic
> [ thread pid 1073 tid 100157 ]
> Stopped at kdb_enter+0x33: movq $0,0x10544e2(%rip)
>
>
> (lldb) image lookup -v --address rip6_send+0x5a8
> Address: kernel.full[0xffffffff80d99228] (kernel.full.PT_LOAD[1]..text +
> 10572328)
> Summary: kernel.full`rip6_send + 1448 at raw_ip6.c:533:10
> Module: file = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full",
> arch = "x86_64"
> CompileUnit: id = {0x000005ba}, file = "/usr/src/sys/netinet6/raw_ip6.c",
> language = "c99"
> Function: id = {0x023e5ddd}, name = "rip6_send", range =
> [0xffffffff80d98c80-0xffffffff80d9973d)
> FuncType: id = {0x023e5ddd}, byte-size = 0, decl = raw_ip6.c:338,
> compiler_type = "int (struct socket *, int, struct mbuf *, struct sockaddr *,
> struct mbuf *, struct thread *)"
> Blocks: id = {0x023e5ddd}, range =
> [0xffffffff80d98c80-0xffffffff80d9973d)
> LineEntry: [0xffffffff80d99211-0xffffffff80d9922a):
> /usr/src/sys/netinet6/raw_ip6.c:533:10
> Symbol: id = {0x0000840a}, range =
> [0xffffffff80d98c80-0xffffffff80d9973d), name="rip6_send"
> Variable: id = {0x023e5df7}, name = "so", type = "socket *", valid ranges
> = <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6
> RBP-168, decl = raw_ip6.c:338
> Variable: id = {0x023e5e07}, name = "flags", type = "int", valid ranges =
> <block>, location = <empty>, decl = raw_ip6.c:338
> Variable: id = {0x023e5e13}, name = "m", type = "mbuf *", valid ranges =
> <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_reg12
> R12, decl = raw_ip6.c:338
> Variable: id = {0x023e5e23}, name = "nam", type = "sockaddr *", valid
> ranges = <block>, location = [0xffffffff80d98e48, 0xffffffff80d99398) ->
> DW_OP_GNU_entry_value(DW_OP_reg2 RCX), DW_OP_stack_value, decl = raw_ip6.c:338
> Variable: id = {0x023e5e33}, name = "control", type = "mbuf *", valid
> ranges = <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) ->
> DW_OP_breg6 RBP-48, decl = raw_ip6.c:339
> Variable: id = {0x023e5e43}, name = "td", type = "thread *", valid ranges
> = <block>, location = <empty>, decl = raw_ip6.c:339
> Variable: id = {0x023e5e4f}, name = "et", type = "epoch_tracker", valid
> ranges = <block>, location = DW_OP_fbreg -200, decl = raw_ip6.c:341
> Variable: id = {0x023e5e5f}, name = "opt", type = "ip6_pktopts", valid
> ranges = <block>, location = DW_OP_fbreg -408, decl = raw_ip6.c:347
> Variable: id = {0x023e5e6f}, name = "oifp", type = "ifnet *", valid
> ranges = <block>, location = DW_OP_fbreg -56, decl = raw_ip6.c:348
> Variable: id = {0x023e5e7e}, name = "hlim", type = "int", valid ranges =
> <block>, location = DW_OP_fbreg -80, decl = raw_ip6.c:353
> Variable: id = {0x023e5e8e}, name = "in6a", type = "in6_addr", valid
> ranges = <block>, location = DW_OP_fbreg -120, decl = raw_ip6.c:354
> Variable: id = {0x023e5ebe}, name = "type", type = "int", valid ranges =
> <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6
> RBP-72, decl = raw_ip6.c:350
> Variable: id = {0x023e5ece}, name = "code", type = "int", valid ranges =
> <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) -> DW_OP_breg6
> RBP-76, decl = raw_ip6.c:350
> Variable: id = {0x023e5ede}, name = "scope_ambiguous", type = "int",
> valid ranges = <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398)
> -> DW_OP_breg6 RBP-92, decl = raw_ip6.c:351
> Variable: id = {0x023e5efe}, name = "inp", type = "inpcb *", valid ranges
> = <block>, location = [0xffffffff80d98cb4, 0xffffffff80d9973d) -> DW_OP_breg6
> RBP-64, decl = raw_ip6.c:342
> Variable: id = {0x023e5f0e}, name = "error", type = "int", valid ranges =
> <block>, location = [0xffffffff80d991a9, 0xffffffff80d9922a) -> DW_OP_consts
> +0, DW_OP_stack_value, decl = raw_ip6.c:349
> Variable: id = {0x023e5f2e}, name = "optp", type = "ip6_pktopts *", valid
> ranges = <block>, location = [0xffffffff80d99183, 0xffffffff80d9922a) ->
> DW_OP_reg3 RBX, decl = raw_ip6.c:347
> Variable: id = {0x023e5f3e}, name = "ip6", type = "ip6_hdr *", valid
> ranges = <block>, location = [0xffffffff80d990c0, 0xffffffff80d99398) ->
> DW_OP_breg6 RBP-160, decl = raw_ip6.c:345
>
> 532 NET_EPOCH_ENTER(et);
> 533 error = ip6_output(m, optp, NULL, 0, inp->in6p_moptions,
> &oifp, inp);
> 534 NET_EPOCH_EXIT(et);
>
>
> (lldb) image lookup -v --address ip6_output+0xb5d
> Address: kernel.full[0xffffffff80d821bd] (kernel.full.PT_LOAD[1]..text +
> 10478013)
> Summary: kernel.full`ip6_output + 2909 at ip6_output.c:811:3
> Module: file = "/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full",
> arch = "x86_64"
> CompileUnit: id = {0x000005b5}, file = "/usr/src/sys/netinet6/ip6_output.c",
> language = "c99"
> Function: id = {0x023ab5ce}, name = "ip6_output", range =
> [0xffffffff80d81660-0xffffffff80d838c4)
> FuncType: id = {0x023ab5ce}, byte-size = 0, decl = ip6_output.c:403,
> compiler_type = "int (struct mbuf *, struct ip6_pktopts *, struct route_in6
> *, int, struct ip6_moptions *, struct ifnet **, struct inpcb *)"
> Blocks: id = {0x023ab5ce}, range =
> [0xffffffff80d81660-0xffffffff80d838c4)
> LineEntry: [0xffffffff80d821b6-0xffffffff80d821c4):
> /usr/src/sys/netinet6/ip6_output.c:811:3
> Symbol: id = {0x000101b9}, range =
> [0xffffffff80d81660-0xffffffff80d838c4), name="ip6_output"
> Variable: id = {0x023ab5e8}, name = "m0", type = "mbuf *", valid ranges =
> <block>, location = [0xffffffff80d819c2, 0xffffffff80d82a26) ->
> DW_OP_GNU_entry_value(DW_OP_reg5 RDI), DW_OP_stack_value, decl =
> ip6_output.c:403
> Variable: id = {0x023ab5f8}, name = "opt", type = "ip6_pktopts *", valid
> ranges = <block>, location = [0xffffffff80d8168e, 0xffffffff80d838c4) ->
> DW_OP_breg6 RBP-184, decl = ip6_output.c:403
> Variable: id = {0x023ab608}, name = "ro", type = "route_in6 *", valid
> ranges = <block>, location = [0xffffffff80d81a3b, 0xffffffff80d824c0) ->
> DW_OP_reg3 RBX, decl = ip6_output.c:404
> Variable: id = {0x023ab618}, name = "flags", type = "int", valid ranges =
> <block>, location = [0xffffffff80d81684, 0xffffffff80d838c4) -> DW_OP_breg6
> RBP-152, decl = ip6_output.c:404
> Variable: id = {0x023ab628}, name = "im6o", type = "ip6_moptions *",
> valid ranges = <block>, location = [0xffffffff80d816b7, 0xffffffff80d83032)
> -> DW_OP_breg6 RBP-352, decl = ip6_output.c:404
> Variable: id = {0x023ab638}, name = "ifpp", type = "ifnet **", valid
> ranges = <block>, location = [0xffffffff80d8167b, 0xffffffff80d838c4) ->
> DW_OP_breg6 RBP-304, decl = ip6_output.c:405
> Variable: id = {0x023ab648}, name = "inp", type = "inpcb *", valid ranges
> = <block>, location = DW_OP_fbreg +16, decl = ip6_output.c:405
> Variable: id = {0x023ab657}, name = "m", type = "mbuf *", valid ranges =
> <block>, location = DW_OP_fbreg -48, decl = ip6_output.c:409
> Variable: id = {0x023ab666}, name = "sin6", type = "sockaddr_in6", valid
> ranges = <block>, location = DW_OP_fbreg -380, decl = ip6_output.c:413
> Variable: id = {0x023ab676}, name = "src_sa", type = "sockaddr_in6",
> valid ranges = <block>, location = DW_OP_fbreg -336, decl = ip6_output.c:413
> Variable: id = {0x023ab686}, name = "dst_sa", type = "sockaddr_in6",
> valid ranges = <block>, location = DW_OP_fbreg -128, decl = ip6_output.c:413
> Variable: id = {0x023ab696}, name = "odst", type = "in6_addr", valid
> ranges = <block>, location = DW_OP_fbreg -432, decl = ip6_output.c:414
> Variable: id = {0x023ab6a6}, name = "alwaysfrag", type = "int", valid
> ranges = <block>, location = DW_OP_fbreg -192, decl = ip6_output.c:421
> Variable: id = {0x023ab6b6}, name = "exthdrs", type = "ip6_exthdrs",
> valid ranges = <block>, location = DW_OP_fbreg -248, decl = ip6_output.c:423
> Variable: id = {0x023ab6c6}, name = "src0", type = "in6_addr", valid
> ranges = <block>, location = DW_OP_fbreg -416, decl = ip6_output.c:424
> Variable: id = {0x023ab6d6}, name = "dst0", type = "in6_addr", valid
> ranges = <block>, location = DW_OP_fbreg -400, decl = ip6_output.c:424
> Variable: id = {0x023ab6e6}, name = "zone", type = "u_int32_t", valid
> ranges = <block>, location = DW_OP_fbreg -188, decl = ip6_output.c:425
> Variable: id = {0x023ab726}, name = "error", type = "int", valid ranges =
> <block>, location = [0xffffffff80d8214f, 0xffffffff80d8251a) -> DW_OP_consts
> +0, DW_OP_stack_value, decl = ip6_output.c:417
> Variable: id = {0x023ab736}, name = "vlan_pcp", type = "int", valid
> ranges = <block>, location = [0xffffffff80d819c2, 0xffffffff80d82a18) ->
> DW_OP_breg6 RBP-96, decl = ip6_output.c:418
> Variable: id = {0x023ab746}, name = "ia", type = "in6_ifaddr *", valid
> ranges = <block>, location = [0xffffffff80d82187, 0xffffffff80d82a18) ->
> DW_OP_breg6 RBP-264, decl = ip6_output.c:419
> Variable: id = {0x023ab776}, name = "ip6", type = "ip6_hdr *", valid
> ranges = <block>, location = [0xffffffff80d81e29, 0xffffffff80d82665) ->
> DW_OP_breg6 RBP-168, decl = ip6_output.c:407
> Variable: id = {0x023ab7c6}, name = "nexthdrp", type = "u_char *", valid
> ranges = <block>, location = [0xffffffff80d81a4f, 0xffffffff80d82a18) ->
> DW_OP_breg6 RBP-136, decl = ip6_output.c:415
> Variable: id = {0x023ab7d6}, name = "ro_pmtu", type = "route_in6 *",
> valid ranges = <block>, location = [0xffffffff80d81ab4, 0xffffffff80d824c0)
> -> DW_OP_reg3 RBX, decl = ip6_output.c:411
> Variable: id = {0x023ab7e6}, name = "dst", type = "sockaddr_in6 *", valid
> ranges = <block>, location = [0xffffffff80d81b39, 0xffffffff80d82466) ->
> DW_OP_breg6 RBP-88, decl = ip6_output.c:413
> Variable: id = {0x023ab7f6}, name = "fibnum", type = "uint32_t", valid
> ranges = <block>, location = [0xffffffff80d81b26, 0xffffffff80d827e3) ->
> DW_OP_breg6 RBP-72, decl = ip6_output.c:429
> Variable: id = {0x023ab806}, name = "origifp", type = "ifnet *", valid
> ranges = <block>, location = [0xffffffff80d8218b, 0xffffffff80d824ae) ->
> DW_OP_reg12 R12, decl = ip6_output.c:408
> Variable: id = {0x023ab876}, name = "tlen", type = "int", valid ranges =
> <block>, location = <empty>, decl = ip6_output.c:416
> Variable: id = {0x023ab882}, name = "dontfrag", type = "int", valid
> ranges = <block>, location = <empty>, decl = ip6_output.c:421
>
>
> 806 KASSERT((ifp != NULL), ("output interface must not be NULL"));
> 807 KASSERT((origifp != NULL), ("output address interface must not
> be NULL"));
> 808
> 809 if ((flags & IPV6_FORWARDING) == 0) {
> 810 /* XXX: the FORWARDING flag can be set for mrouting. */
> 811 in6_ifstat_inc(ifp, ifs6_out_request);
I'm not quite sure, but it seems the `ifp` is not fully constructed. See
https://cgit.freebsd.org/src/tree/sys/net/if.c#n950
<https://cgit.freebsd.org/src/tree/sys/net/if.c#n950>
If I read the code correctly, the clone created interface is made visible via
`if_link_ifnet(ifp);` , and at that time the
`ifp->if_afdata[AF_INET6]` is NULL and is not initialized yet by
`if_attachdomain1()` which will call `in6_domifattach()`
to allocate the required data.
So I guess there is a race condition. I bet this can be repeated easily.
I have not tested this yet, and not sure if it is the right fix, but you can
give it a try.
diff --git a/sys/net/if.c b/sys/net/if.c
index c3c27fbf678f..16ee5667e7bb 100644
--- a/sys/net/if.c
+++ b/sys/net/if.c
@@ -947,11 +947,11 @@ if_attach_internal(struct ifnet *ifp, bool vmove)
}
#endif
- if_link_ifnet(ifp);
-
if (domain_init_status >= 2)
if_attachdomain1(ifp);
+ if_link_ifnet(ifp);
+
EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp);
if (IS_DEFAULT_VNET(curvnet))
devctl_notify("IFNET", ifp->if_xname, "ATTACH", NULL);
> 812 }
> 813
> 814 /* Setup data structures for scope ID checks. */
>
> --
> Bjoern A. Zeeb r15:7
>
Best regards,
Zhenlei
