Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 04:39:07PM +0300, Konstantin Belousov wrote: On Tue, Jul 21, 2015 at 05:57:34AM -0700, David Wolfskill wrote: My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 successes out of 4 tries); here's a bit from the core.txt file: There must be kernel messages before the panic string. They are crusial to understand what is going on. ... Sorry I wasn't able to capture those before I needed to do Other Things. The machine had a (PCI-attached) serial console that was working for FreeBSD (thanks mostly to sbruno's help), but Somthing seems to Have Happened, and that's not presently working (even in stable/10, where I first got it working). I will try to get it working again, but I doubt I will have time to focus on that until about 9 hours from now. Peace, david -- David H. Wolfskill da...@catwhisker.org Those who murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpXNNyz_zCPR.pgp Description: PGP signature
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 09:19:27AM -0700, David Wolfskill wrote: On Tue, Jul 21, 2015 at 04:39:07PM +0300, Konstantin Belousov wrote: On Tue, Jul 21, 2015 at 05:57:34AM -0700, David Wolfskill wrote: My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 successes out of 4 tries); here's a bit from the core.txt file: There must be kernel messages before the panic string. They are crusial to understand what is going on. ... Sorry I wasn't able to capture those before I needed to do Other Things. The machine had a (PCI-attached) serial console that was working for FreeBSD (thanks mostly to sbruno's help), but Somthing seems to Have Happened, and that's not presently working (even in stable/10, where I first got it working). I will try to get it working again, but I doubt I will have time to focus on that until about 9 hours from now. It's possible to extract log messages leading up to the panic from the vmcore. From the kgdb prompt, running (kgdb) printf %s, msgbufp-msg_ptr should bring them up. And, I just noticed that you posted the core.txt, which contains this info near the end: http://www.catwhisker.org/~david/FreeBSD/head/core.txt.1 ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 07:17:43PM +, Mark Johnston wrote: On Tue, Jul 21, 2015 at 09:19:27AM -0700, David Wolfskill wrote: On Tue, Jul 21, 2015 at 04:39:07PM +0300, Konstantin Belousov wrote: On Tue, Jul 21, 2015 at 05:57:34AM -0700, David Wolfskill wrote: My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 successes out of 4 tries); here's a bit from the core.txt file: There must be kernel messages before the panic string. They are crusial to understand what is going on. ... Sorry I wasn't able to capture those before I needed to do Other Things. The machine had a (PCI-attached) serial console that was working for FreeBSD (thanks mostly to sbruno's help), but Somthing seems to Have Happened, and that's not presently working (even in stable/10, where I first got it working). I will try to get it working again, but I doubt I will have time to focus on that until about 9 hours from now. It's possible to extract log messages leading up to the panic from the vmcore. From the kgdb prompt, running (kgdb) printf %s, msgbufp-msg_ptr should bring them up. And, I just noticed that you posted the core.txt, which contains this info near the end: http://www.catwhisker.org/~david/FreeBSD/head/core.txt.1 Indeed, thank you. ithread_loop() at ithread_loop+0xa6/frame 0xfe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe083b9c0ab0 --- trap 0, rip = 0, rsp = 0xfe083b9c0b70, rbp = 0 --- suspending ithread with the following locks held: shared rw udpinp (udpinp) r = 3 (0xf80010c7d7b0) locked @ /usr/src/sys/netinet6/in6_pcb.c:1174 panic: witness_warn cpuid = 3 So it looks like net swi, leaking some udp6 lock. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 03:21:16PM -0500, Eric van Gyzen wrote: ... So it looks like net swi, leaking some udp6 lock. Curiouser and curiouser... While I'm not taking any special pains to avoid building IPv6, I'm not actively actually doing anything with it (IPv6), either (for both the failing machine and my laptop). Once I'm back home, I should be able to poke around in ddb after re-creating the panic, if that would be a useful thing for me to do (and given some hints as to what to poke). Naturally, I'm also happy to change bits of sources, rebuild, and smoke-test. A quick check from the SVN update output only shows r285710, r285711, and r285740 in the range from (r285685,r285741] -- as the kernel running r285685 had no known issues -- that touched sys/netinet6/*. It's a multicast destination. Maybe something is using mDNS? Randall, does the test on line 406 of udp6_usrreq.c need to be inverted? Eric DING! We have a winner! FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1789 r285741M/285741:1100077: Tue Jul 21 14:50:59 PDT 2015 r...@freebeast.catwhisker.org:/common/S3/obj/usr/src/sys/GENERIC amd64 freebeast(11.0-C)[3] cd /usr/src freebeast(11.0-C)[4] svn diff sys/netinet netinet/ netinet6/ freebeast(11.0-C)[4] svn diff sys/netinet* Index: sys/netinet6/udp6_usrreq.c === --- sys/netinet6/udp6_usrreq.c (revision 285741) +++ sys/netinet6/udp6_usrreq.c (working copy) @@ -403,7 +403,7 @@ INP_RLOCK(last); INP_INFO_RUNLOCK(pcbinfo); UDP_PROBE(receive, NULL, last, ip6, last, uh); - if (udp6_append(last, m, off, fromsa)) + if (! udp6_append(last, m, off, fromsa)) INP_RUNLOCK(last); inp_lost: return (IPPROTO_DONE); freebeast(11.0-C)[5] Thanks! :-) Peace, david -- David H. Wolfskill da...@catwhisker.org Those who murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpJj1IlX26b0.pgp Description: PGP signature
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 10:28:32PM +0300, Konstantin Belousov wrote: ... Indeed, thank you. ithread_loop() at ithread_loop+0xa6/frame 0xfe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe083b9c0ab0 --- trap 0, rip = 0, rsp = 0xfe083b9c0b70, rbp = 0 --- suspending ithread with the following locks held: shared rw udpinp (udpinp) r = 3 (0xf80010c7d7b0) locked @ /usr/src/sys/netinet6/in6_pcb.c:1174 panic: witness_warn cpuid = 3 So it looks like net swi, leaking some udp6 lock. Curiouser and curiouser... While I'm not taking any special pains to avoid building IPv6, I'm not actively actually doing anything with it (IPv6), either (for both the failing machine and my laptop). Once I'm back home, I should be able to poke around in ddb after re-creating the panic, if that would be a useful thing for me to do (and given some hints as to what to poke). Naturally, I'm also happy to change bits of sources, rebuild, and smoke-test. A quick check from the SVN update output only shows r285710, r285711, and r285740 in the range from (r285685,r285741] -- as the kernel running r285685 had no known issues -- that touched sys/netinet6/*. Peace, david -- David H. Wolfskill da...@catwhisker.org Those who murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgpQ_E2uiyznk.pgp Description: PGP signature
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On 07/21/2015 15:05, David Wolfskill wrote: On Tue, Jul 21, 2015 at 10:28:32PM +0300, Konstantin Belousov wrote: ... Indeed, thank you. ithread_loop() at ithread_loop+0xa6/frame 0xfe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe083b9c0ab0 --- trap 0, rip = 0, rsp = 0xfe083b9c0b70, rbp = 0 --- suspending ithread with the following locks held: shared rw udpinp (udpinp) r = 3 (0xf80010c7d7b0) locked @ /usr/src/sys/netinet6/in6_pcb.c:1174 panic: witness_warn cpuid = 3 So it looks like net swi, leaking some udp6 lock. Curiouser and curiouser... While I'm not taking any special pains to avoid building IPv6, I'm not actively actually doing anything with it (IPv6), either (for both the failing machine and my laptop). Once I'm back home, I should be able to poke around in ddb after re-creating the panic, if that would be a useful thing for me to do (and given some hints as to what to poke). Naturally, I'm also happy to change bits of sources, rebuild, and smoke-test. A quick check from the SVN update output only shows r285710, r285711, and r285740 in the range from (r285685,r285741] -- as the kernel running r285685 had no known issues -- that touched sys/netinet6/*. It's a multicast destination. Maybe something is using mDNS? Randall, does the test on line 406 of udp6_usrreq.c need to be inverted? Eric signature.asc Description: OpenPGP digital signature
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On 07/21/2015 15:21, Eric van Gyzen wrote: On 07/21/2015 15:05, David Wolfskill wrote: On Tue, Jul 21, 2015 at 10:28:32PM +0300, Konstantin Belousov wrote: ... Indeed, thank you. ithread_loop() at ithread_loop+0xa6/frame 0xfe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe083b9c0ab0 --- trap 0, rip = 0, rsp = 0xfe083b9c0b70, rbp = 0 --- suspending ithread with the following locks held: shared rw udpinp (udpinp) r = 3 (0xf80010c7d7b0) locked @ /usr/src/sys/netinet6/in6_pcb.c:1174 panic: witness_warn cpuid = 3 So it looks like net swi, leaking some udp6 lock. Curiouser and curiouser... While I'm not taking any special pains to avoid building IPv6, I'm not actively actually doing anything with it (IPv6), either (for both the failing machine and my laptop). Once I'm back home, I should be able to poke around in ddb after re-creating the panic, if that would be a useful thing for me to do (and given some hints as to what to poke). Naturally, I'm also happy to change bits of sources, rebuild, and smoke-test. A quick check from the SVN update output only shows r285710, r285711, and r285740 in the range from (r285685,r285741] -- as the kernel running r285685 had no known issues -- that touched sys/netinet6/*. It's a multicast destination. Maybe something is using mDNS? Blurf. I wonder if it's a multicast destination. (I need more chocolate.) Randall, does the test on line 406 of udp6_usrreq.c need to be inverted? Eric signature.asc Description: OpenPGP digital signature
Re: panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
On Tue, Jul 21, 2015 at 05:57:34AM -0700, David Wolfskill wrote: My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 successes out of 4 tries); here's a bit from the core.txt file: There must be kernel messages before the panic string. They are crusial to understand what is going on. freebeast.catwhisker.org dumped core - see /var/crash/vmcore.1 Tue Jul 21 05:36:11 PDT 2015 FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1787 r285741M/285741:1100077: Tue Jul 21 04:48:37 PDT 2015 r...@freebeast.catwhisker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 panic: witness_warn ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
panic: witness_warn head/amd64 @r285741 on 1 of 2 machines
My laptop had no problems, but the build machine has a panic that appears quite reproducible (4 successes out of 4 tries); here's a bit from the core.txt file: freebeast.catwhisker.org dumped core - see /var/crash/vmcore.1 Tue Jul 21 05:36:11 PDT 2015 FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1787 r285741M/285741:1100077: Tue Jul 21 04:48:37 PDT 2015 r...@freebeast.catwhisker.org:/common/S4/obj/usr/src/sys/GENERIC amd64 panic: witness_warn GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Unread portion of the kernel message buffer: panic: witness_warn cpuid = 3 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfe083b9c0860 vpanic() at vpanic+0x189/frame 0xfe083b9c08e0 kassert_panic() at kassert_panic+0x132/frame 0xfe083b9c0950 witness_warn() at witness_warn+0x498/frame 0xfe083b9c0a20 ithread_loop() at ithread_loop+0x165/frame 0xfe083b9c0a70 fork_exit() at fork_exit+0x84/frame 0xfe083b9c0ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfe083b9c0ab0 --- trap 0, rip = 0, rsp = 0xfe083b9c0b70, rbp = 0 --- ... (kgdb) #0 doadump (textdump=0) at pcpu.h:221 #1 0x80377dfe in db_dump (dummy=value optimized out, dummy2=false, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:533 #2 0x80377971 in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440 #3 0x80377604 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #4 0x8037a19b in db_trap (type=value optimized out, code=0) at /usr/src/sys/ddb/db_main.c:251 #5 0x80a56624 in kdb_trap (type=3, code=0, tf=value optimized out) at /usr/src/sys/kern/subr_kdb.c:654 #6 0x80e61bd1 in trap (frame=0xfe083b9c0790) at /usr/src/sys/amd64/amd64/trap.c:540 #7 0x80e41e02 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:235 #8 0x80a55cfe in kdb_enter (why=0x8136f098 panic, msg=0x80a5bee0 UH\211E5AWAVATSH\203ECPI\211F7A\211FEH\213\004%D0C7B8\201H\211ED8\201%xD7B8\201) at cpufunc.h:63 #9 0x80a19739 in vpanic (fmt=value optimized out, ap=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:737 #10 0x80a19582 in kassert_panic (fmt=value optimized out) at /usr/src/sys/kern/kern_shutdown.c:634 #11 0x80a74908 in witness_warn (flags=2, lock=value optimized out, fmt=0x81367827 suspending ithread) at /usr/src/sys/kern/subr_witness.c:1757 #12 0x809e2985 in ithread_loop (arg=0xf8000770c820) at /usr/src/sys/kern/kern_intr.c:1345 #13 0x809df874 in fork_exit ( callout=0x809e2820 ithread_loop, arg=0xf8000770c820, frame=0xfe083b9c0ac0) at /usr/src/sys/kern/kern_fork.c:1006 #14 0x80e4233e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:610 #15 0x in ?? () Current language: auto; currently minimal (kgdb) On boot, it dropped into the debugger; it was on the most recent instantiation that I manually issued a dump command from that environment, then rebooted under the previous kernel: FreeBSD freebeast.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #1786 r285715M/285715:1100077: Mon Jul 20 04:22:26 PDT 2015 r...@freebeast.catwhisker.org:/common/S3/obj/usr/src/sys/GENERIC amd64 (And yes, it runs an unmodified GENERIC kernel.) The machine has been deployed only for a couple of months or so, but has been building stable/10 and head daily during that time. Until a couple of weeks ago, it was doing this for both i386 and amd64; since then, I dropped i386 from my home infrastructure, so it's been only amd64. In the stable/10 environment, it also make use of a 3-spindle zraid for running poudriere (to build the ports for my production machines), and it's been doing that quite well, also. Only other thing that I think of that's noteworthy is that its boot drive is an SSD (where I have not yet enabled TRIM, as it's a Crucial M500, and I need to be sure we don't try to use the queued TRIM commands on it, as there are reports that queued TRIM commands on the M500 will corrupt data). OK; please see http://www.catwhisker.org/~david/FreeBSD/head/ for the dump(-related) files. (It's on a residential ADSL, so it's going to be slow. Sorry; I have a limited amount of bandwidth.) Peace, david -- David H. Wolfskill da...@catwhisker.org Those who murder in the name of God or prophet are blasphemous cowards. See http://www.catwhisker.org/~david/publickey.gpg for my public key. pgp4y7XWYoZPA.pgp Description: PGP signature
