Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On 2019-05-09 14:38, Gleb Smirnoff wrote: > On Thu, May 09, 2019 at 02:32:44PM -0400, Michael Butler wrote: > M> (kgdb) frame 8 > M> #8 0x80a15377 in ip_output (m=, opt= M> optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at > M> /usr/src/sys/netinet/ip_output.c:362 > M> 362 IFP_TO_IA(ifp, ia, _ifa_tracker); > M> (kgdb) print imo > M> $1 = (struct ip_moptions *) 0xfe0072b14780 > M> (kgdb) print ifp > M> $2 = (struct ifnet *) 0xf8000411 > M> (kgdb) print ia > M> $3 = > ... > > This all looks good. > > Can you please traverse the 'in_ifaddrhead' linked list? To close the loop on this - it's now fixed by SVN r347466, Thanks! imb ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On Thu, May 09, 2019 at 02:32:44PM -0400, Michael Butler wrote: M> (kgdb) frame 8 M> #8 0x80a15377 in ip_output (m=, opt= optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at M> /usr/src/sys/netinet/ip_output.c:362 M> 362 IFP_TO_IA(ifp, ia, _ifa_tracker); M> (kgdb) print imo M> $1 = (struct ip_moptions *) 0xfe0072b14780 M> (kgdb) print ifp M> $2 = (struct ifnet *) 0xf8000411 M> (kgdb) print ia M> $3 = ... This all looks good. Can you please traverse the 'in_ifaddrhead' linked list? -- Gleb Smirnoff ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On 2019-05-09 14:22, Gleb Smirnoff wrote:
> Michael,
>
> On Thu, May 09, 2019 at 10:25:37AM -0500, Kyle Evans wrote:
> K> > #0 doadump () at src/sys/amd64/include/pcpu.h:241
> K> > #1 0x808393c8 in kern_reboot (howto=260) at
> K> > /usr/src/sys/kern/kern_shutdown.c:470
> K> > #2 0x80839826 in vpanic (fmt=, ap= K> > optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896
> K> > #3 0x80839663 in panic (fmt=) at
> K> > /usr/src/sys/kern/kern_shutdown.c:823
> K> > #4 0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156)
> K> > at /usr/src/sys/amd64/amd64/trap.c:946
> K> > #5 0x80c31c59 in trap_pfault (frame=0xfe0072b14510,
> K> > usermode=0) at src/sys/amd64/include/cpufunc.h:423
> K> > #6 0x80c30fde in trap (frame=0xfe0072b14510) at
> K> > /usr/src/sys/amd64/amd64/trap.c:441
> K> > #7 0x80c0d4b5 in calltrap () at
> K> > /usr/src/sys/amd64/amd64/exception.S:232
> K> > #8 0x80a15377 in ip_output (m=, opt= K> > optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
> K> > /usr/src/sys/netinet/ip_output.c:362
> K> > #9 0x809ffea4 in igmp_intr (m=) at
> K> > /usr/src/sys/netinet/igmp.c:3455
> K> > #10 0x80975a0f in netisr_dispatch_src (proto=2, source= K> > optimized out>, m=) at
> /usr/src/sys/net/netisr.c:1122
> K> > #11 0x809fe07a in igmp_fasttimo () at
> K> > /usr/src/sys/netinet/igmp.c:496
> K> > #12 0x808c5854 in pffasttimo (arg=) at
> K> > /usr/src/sys/kern/uipc_domain.c:521
> K> > #13 0x80853df3 in softclock_call_cc (c=0x813f7f48,
> K> > cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731
> K> > #14 0x808542b9 in softclock (arg=0x814c9ac0) at
> K> > /usr/src/sys/kern/kern_timeout.c:869
> K> > #15 0x807fd0c4 in ithread_loop (arg=) at
> K> > /usr/src/sys/kern/kern_intr.c:1129
> K> > #16 0x807f9f33 in fork_exit (callout=0x807fcef0
> K> > , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at
> K> > /usr/src/sys/kern/kern_fork.c:1058
> K> > #17 0x80c0e4ae in fork_trampoline () at
> K> > /usr/src/sys/amd64/amd64/exception.S:995
> K> > #18 0x in ?? ()
> K> >
> K>
> K> Ah, I misread your backtrace (and forgot the proper tap detachment
> K> from my previous patch, so that's fixed/committed anyways). CC'ing
> K> Gleb for further triage as committer of r347375 that touched things in
> K> this path.
>
> Michael, can you please dump a core and look at it in kgdb? Line 362 in
> ip_output() really belongs to part that had minimal change with r347375.
> So I need more details. Can you please print out in kgdb the following
> variables: imo, ifp, ia?
>
This was a backtrace from kgdb. From frame 8, I see ..
(kgdb) frame 8
#8 0x80a15377 in ip_output (m=, opt=, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at
/usr/src/sys/netinet/ip_output.c:362
362 IFP_TO_IA(ifp, ia, _ifa_tracker);
(kgdb) print imo
$1 = (struct ip_moptions *) 0xfe0072b14780
(kgdb) print ifp
$2 = (struct ifnet *) 0xf8000411
(kgdb) print ia
$3 =
(kgdb) print *imo
$4 = {imo_multicast_ifp = 0xf8000411, imo_multicast_addr =
{s_addr = 1924220944}, imo_multicast_vif = 18446744073709551615,
imo_multicast_ttl = 1 '\001', imo_multicast_loop = 0 '\0',
imo_num_memberships = 15350, imo_max_memberships = 63489,
imo_membership = 0xf80002c10d00, imo_mfilters = 0xf80002c10d00,
imo_epoch_ctx = {data = 0xfe0072b147b0}}
(kgdb) print *ifp
$5 = {if_link = {cstqe_next = 0x0}, if_clones = {le_next = 0x0, le_prev
= 0xf800040f9728}, if_groups = {cstqh_first = 0xf80002878d60,
cstqh_last = 0xf80002878d38},
if_alloctype = 6 '\006', if_numa_domain = 255 '�', if_softc =
0xf80004197300, if_llsoftc = 0x0, if_l2com = 0xf800040fe800,
if_dname = 0x80e0bd14 "tap", if_dunit = 0,
if_index = 4, if_index_reserved = 0, if_xname = 0xf80004110058
"tap0", if_description = 0x0, if_flags = 34818, if_drv_flags = 0,
if_capabilities = 524288, if_capenable = 524288,
if_linkmib = 0x0, if_linkmiblen = 0, if_refcount = 1, if_type = 6
'\006', if_addrlen = 6 '\006', if_hdrlen = 14 '\016', if_link_state = 1
'\001', if_mtu = 1500, if_metric = 0,
if_baudrate = 1000, if_hwassist = 0, if_epoch = 10, if_lastchange
= {tv_sec = 1557408022, tv_usec = 929504}, if_snd = {ifq_head = 0x0,
ifq_tail = 0x0, ifq_len = 0, ifq_maxlen = 50,
ifq_mtx = {lock_object = {lo_name = 0xf80004110058 "tap0",
lo_flags = 16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0},
ifq_drv_head = 0x0, ifq_drv_tail = 0x0,
ifq_drv_len = 0, ifq_drv_maxlen = 0, altq_type = 0, altq_flags = 0,
altq_disc = 0x0, altq_ifp = 0xf8000411, altq_enqueue = 0,
altq_dequeue = 0, altq_request = 0,
altq_clfier = 0x0, altq_classify = 0, altq_tbr = 0x0, altq_cdnr =
0x0}, if_linktask = {ta_link = {stqe_next = 0x0}, ta_pending = 0,
ta_priority = 0,
ta_func = 0x809494e0 , ta_context
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
Michael, On Thu, May 09, 2019 at 10:25:37AM -0500, Kyle Evans wrote: K> > #0 doadump () at src/sys/amd64/include/pcpu.h:241 K> > #1 0x808393c8 in kern_reboot (howto=260) at K> > /usr/src/sys/kern/kern_shutdown.c:470 K> > #2 0x80839826 in vpanic (fmt=, ap= > optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896 K> > #3 0x80839663 in panic (fmt=) at K> > /usr/src/sys/kern/kern_shutdown.c:823 K> > #4 0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156) K> > at /usr/src/sys/amd64/amd64/trap.c:946 K> > #5 0x80c31c59 in trap_pfault (frame=0xfe0072b14510, K> > usermode=0) at src/sys/amd64/include/cpufunc.h:423 K> > #6 0x80c30fde in trap (frame=0xfe0072b14510) at K> > /usr/src/sys/amd64/amd64/trap.c:441 K> > #7 0x80c0d4b5 in calltrap () at K> > /usr/src/sys/amd64/amd64/exception.S:232 K> > #8 0x80a15377 in ip_output (m=, opt= > optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at K> > /usr/src/sys/netinet/ip_output.c:362 K> > #9 0x809ffea4 in igmp_intr (m=) at K> > /usr/src/sys/netinet/igmp.c:3455 K> > #10 0x80975a0f in netisr_dispatch_src (proto=2, source= > optimized out>, m=) at /usr/src/sys/net/netisr.c:1122 K> > #11 0x809fe07a in igmp_fasttimo () at K> > /usr/src/sys/netinet/igmp.c:496 K> > #12 0x808c5854 in pffasttimo (arg=) at K> > /usr/src/sys/kern/uipc_domain.c:521 K> > #13 0x80853df3 in softclock_call_cc (c=0x813f7f48, K> > cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731 K> > #14 0x808542b9 in softclock (arg=0x814c9ac0) at K> > /usr/src/sys/kern/kern_timeout.c:869 K> > #15 0x807fd0c4 in ithread_loop (arg=) at K> > /usr/src/sys/kern/kern_intr.c:1129 K> > #16 0x807f9f33 in fork_exit (callout=0x807fcef0 K> > , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at K> > /usr/src/sys/kern/kern_fork.c:1058 K> > #17 0x80c0e4ae in fork_trampoline () at K> > /usr/src/sys/amd64/amd64/exception.S:995 K> > #18 0x in ?? () K> > K> K> Ah, I misread your backtrace (and forgot the proper tap detachment K> from my previous patch, so that's fixed/committed anyways). CC'ing K> Gleb for further triage as committer of r347375 that touched things in K> this path. Michael, can you please dump a core and look at it in kgdb? Line 362 in ip_output() really belongs to part that had minimal change with r347375. So I need more details. Can you please print out in kgdb the following variables: imo, ifp, ia? -- Gleb Smirnoff ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On Thu, May 9, 2019 at 8:27 AM Michael Butler wrote: > > On 2019-05-09 09:07, Kyle Evans wrote: > > On Thu, May 9, 2019 at 7:24 AM Michael Butler > > wrote: > >> > >> Seems there's a lock or tuntap issue after recent changes. Restarting > >> openvpn (configured to use /dev/tap) yields a panic as follows: > >> > > > > Ah, I knew I was forgetting something. =( Give this a shot: > > https://people.freebsd.org/~kevans/etherdetach.diff > > Sorry - same result, > > (kgdb) bt > #0 doadump () at src/sys/amd64/include/pcpu.h:241 > #1 0x808393c8 in kern_reboot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:470 > #2 0x80839826 in vpanic (fmt=, ap= optimized out>) at /usr/src/sys/kern/kern_shutdown.c:896 > #3 0x80839663 in panic (fmt=) at > /usr/src/sys/kern/kern_shutdown.c:823 > #4 0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156) > at /usr/src/sys/amd64/amd64/trap.c:946 > #5 0x80c31c59 in trap_pfault (frame=0xfe0072b14510, > usermode=0) at src/sys/amd64/include/cpufunc.h:423 > #6 0x80c30fde in trap (frame=0xfe0072b14510) at > /usr/src/sys/amd64/amd64/trap.c:441 > #7 0x80c0d4b5 in calltrap () at > /usr/src/sys/amd64/amd64/exception.S:232 > #8 0x80a15377 in ip_output (m=, opt= optimized out>, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at > /usr/src/sys/netinet/ip_output.c:362 > #9 0x809ffea4 in igmp_intr (m=) at > /usr/src/sys/netinet/igmp.c:3455 > #10 0x80975a0f in netisr_dispatch_src (proto=2, source= optimized out>, m=) at /usr/src/sys/net/netisr.c:1122 > #11 0x809fe07a in igmp_fasttimo () at > /usr/src/sys/netinet/igmp.c:496 > #12 0x808c5854 in pffasttimo (arg=) at > /usr/src/sys/kern/uipc_domain.c:521 > #13 0x80853df3 in softclock_call_cc (c=0x813f7f48, > cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731 > #14 0x808542b9 in softclock (arg=0x814c9ac0) at > /usr/src/sys/kern/kern_timeout.c:869 > #15 0x807fd0c4 in ithread_loop (arg=) at > /usr/src/sys/kern/kern_intr.c:1129 > #16 0x807f9f33 in fork_exit (callout=0x807fcef0 > , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at > /usr/src/sys/kern/kern_fork.c:1058 > #17 0x80c0e4ae in fork_trampoline () at > /usr/src/sys/amd64/amd64/exception.S:995 > #18 0x in ?? () > Ah, I misread your backtrace (and forgot the proper tap detachment from my previous patch, so that's fixed/committed anyways). CC'ing Gleb for further triage as committer of r347375 that touched things in this path. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On 2019-05-09 09:07, Kyle Evans wrote: > On Thu, May 9, 2019 at 7:24 AM Michael Butler > wrote: >> >> Seems there's a lock or tuntap issue after recent changes. Restarting >> openvpn (configured to use /dev/tap) yields a panic as follows: >> > > Ah, I knew I was forgetting something. =( Give this a shot: > https://people.freebsd.org/~kevans/etherdetach.diff Sorry - same result, (kgdb) bt #0 doadump () at src/sys/amd64/include/pcpu.h:241 #1 0x808393c8 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:470 #2 0x80839826 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:896 #3 0x80839663 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:823 #4 0x80c318a6 in trap_fatal (frame=0xfe0072b14510, eva=156) at /usr/src/sys/amd64/amd64/trap.c:946 #5 0x80c31c59 in trap_pfault (frame=0xfe0072b14510, usermode=0) at src/sys/amd64/include/cpufunc.h:423 #6 0x80c30fde in trap (frame=0xfe0072b14510) at /usr/src/sys/amd64/amd64/trap.c:441 #7 0x80c0d4b5 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #8 0x80a15377 in ip_output (m=, opt=, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at /usr/src/sys/netinet/ip_output.c:362 #9 0x809ffea4 in igmp_intr (m=) at /usr/src/sys/netinet/igmp.c:3455 #10 0x80975a0f in netisr_dispatch_src (proto=2, source=, m=) at /usr/src/sys/net/netisr.c:1122 #11 0x809fe07a in igmp_fasttimo () at /usr/src/sys/netinet/igmp.c:496 #12 0x808c5854 in pffasttimo (arg=) at /usr/src/sys/kern/uipc_domain.c:521 #13 0x80853df3 in softclock_call_cc (c=0x813f7f48, cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731 #14 0x808542b9 in softclock (arg=0x814c9ac0) at /usr/src/sys/kern/kern_timeout.c:869 #15 0x807fd0c4 in ithread_loop (arg=) at /usr/src/sys/kern/kern_intr.c:1129 #16 0x807f9f33 in fork_exit (callout=0x807fcef0 , arg=0xf800025f10a0, frame=0xfe0072b14ac0) at /usr/src/sys/kern/kern_fork.c:1058 #17 0x80c0e4ae in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:995 #18 0x in ?? () imb ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: at SVN r347375, terminating/restarting openvpn on tap causes panic
On Thu, May 9, 2019 at 7:24 AM Michael Butler wrote: > > Seems there's a lock or tuntap issue after recent changes. Restarting > openvpn (configured to use /dev/tap) yields a panic as follows: > Ah, I knew I was forgetting something. =( Give this a shot: https://people.freebsd.org/~kevans/etherdetach.diff Thanks, Kyle Evans ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
at SVN r347375, terminating/restarting openvpn on tap causes panic
Seems there's a lock or tuntap issue after recent changes. Restarting openvpn (configured to use /dev/tap) yields a panic as follows: (kgdb) bt #0 doadump () at src/sys/amd64/include/pcpu.h:241 #1 0x808393c8 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:470 #2 0x80839826 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:896 #3 0x80839663 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:823 #4 0x80c31686 in trap_fatal (frame=0xfe0072b14510, eva=156) at /usr/src/sys/amd64/amd64/trap.c:946 #5 0x80c31a39 in trap_pfault (frame=0xfe0072b14510, usermode=0) at src/sys/amd64/include/cpufunc.h:423 #6 0x80c30dbe in trap (frame=0xfe0072b14510) at /usr/src/sys/amd64/amd64/trap.c:441 #7 0x80c0d295 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #8 0x80a15387 in ip_output (m=, opt=, ro=0x0, flags=0, imo=0xfe0072b14780, inp=0x0) at /usr/src/sys/netinet/ip_output.c:362 #9 0x809ffeb4 in igmp_intr (m=) at /usr/src/sys/netinet/igmp.c:3455 #10 0x80975a1f in netisr_dispatch_src (proto=2, source=, m=) at /usr/src/sys/net/netisr.c:1122 #11 0x809fe08a in igmp_fasttimo () at /usr/src/sys/netinet/igmp.c:496 #12 0x808c5854 in pffasttimo (arg=) at /usr/src/sys/kern/uipc_domain.c:521 #13 0x80853df3 in softclock_call_cc (c=0x813f7f48, cc=0x814c9ac0, direct=0) at /usr/src/sys/kern/kern_timeout.c:731 #14 0x808542b9 in softclock (arg=0x814c9ac0) at /usr/src/sys/kern/kern_timeout.c:869 #15 0x807fd0c4 in ithread_loop (arg=) at /usr/src/sys/kern/kern_intr.c:1129 #16 0x807f9f33 in fork_exit (callout=0x807fcef0 , arg=0xf80002620100, frame=0xfe0072b14ac0) at /usr/src/sys/kern/kern_fork.c:1058 #17 0x80c0e28e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:995 #18 0x in ?? () imb ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
