Spinlock panic in FreeBSD 7
We've observed a panic in FreeBSD 7 (7.2-PRERELEASE FreeBSD) several times that we've not been able to track down. Upgrading to 8+ is not an option at this time. Does this look at all familiar to anyone? Here's an example stack trace after panic: spin lock 0x8086bdc0 (smp rendezvous) held by 0xff0006d1f000 (tid 100060) too long panic: spin lock held too long cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at 0x8019120a = db_trace_self_wrapper+0x2a panic() at 0x80308797 = panic+0x187 _mtx_lock_spin_failed() at 0x802fbda9 = _mtx_lock_spin_failed+0x39 _mtx_lock_spin() at 0x802fbe4e = _mtx_lock_spin+0x9e _mtx_lock_spin_flags() at 0x802fc354 = _mtx_lock_spin_flags+0x104 smp_rendezvous_cpus() at 0x80340cb3 = smp_rendezvous_cpus+0xd3 xcall() at 0x80ad755e = xcall+0x3e cyclic_remove_here() at 0x80ad7715 = cyclic_remove_here+0x1a5 cyclic_remove() at 0x80ad7a0f = cyclic_remove+0x5f profile_disable() at 0x80acf0e5 = profile_disable+0x15 dtrace_state_destroy() at 0x80adfabd = dtrace_state_destroy+0x35d dtrace_close() at 0x80adffed = dtrace_close+0x8d devfs_close() at 0x802a825d = devfs_close+0x2dd vn_close() at 0x8039cb06 = vn_close+0xb6 vn_closefile() at 0x8039cc00 = vn_closefile+0x80 devfs_close_f() at 0x802a5738 = devfs_close_f+0x28 fdrop() at 0x802d98bb = fdrop+0xdb closef() at 0x802db2f9 = closef+0x29 fdfree() at 0x802dc061 = fdfree+0x161 exit1() at 0x802e56b2 = exit1+0x2c2 sigexit() at 0x8030a86f = sigexit+0x8f postsig() at 0x8030b6ce = postsig+0x38e ast() at 0x803425f7 = ast+0x337 Xfast_syscall() at 0x80494efd = Xfast_syscall+0xdd --- syscall (32, FreeBSD ELF64, getsockname), rip = 0x800df4d5c, rsp = 0x7fffe398, rbp = 0x5 --- KDB: enter: panic The panic always shows up from a syscall, and almost always from syscall 32, getsockname, but we've also observed it with syscall 5. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: dtrace in FreeBSD 7.2
On 2011-09-19 10:38, Julian Elischer wrote: On 9/19/11 9:17 AM, Charlie Martin wrote: Dude, you say that like it was an option. It's understood about the realities of commercial decisions, however, if you really want to trace APPS that means user space dtrace and 7.2 just doesn't have the hooks to do non-kernel dtrace. I think in fact I want to add probes in kernel space; I was trying to play with it in user space just to avoid a lot of test kernels. As a side note, realize that Dtrace support in FreeBSD had a serious problem a couple of years ago when the lead developer suddenly dropped dead (Very literally, to out great sadness), and it took a while for alternative resources to fill the gap. I bet. I'd kind of gotten that idea but don't know the history; I was working in Solaris until recently. On 2011-09-16 18:46, Matthias Andree wrote: Am Freitag, den 16.09.2011, 14:30 -0600 schrieb Charlie Martin: I need to add some custom static dtrace probes in 7.2 apps; the online documentation refers only to 9 however. Can someone tell me how to replace what's done in bsd.dtrace.mk for 7.2? FreeBSD 7.2 is quite old and wasn't under extended support, hence it is no longer supported. Consider upgrading. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: dtrace in FreeBSD 7.2
Dude, you say that like it was an option. On 2011-09-16 18:46, Matthias Andree wrote: Am Freitag, den 16.09.2011, 14:30 -0600 schrieb Charlie Martin: I need to add some custom static dtrace probes in 7.2 apps; the online documentation refers only to 9 however. Can someone tell me how to replace what's done in bsd.dtrace.mk for 7.2? FreeBSD 7.2 is quite old and wasn't under extended support, hence it is no longer supported. Consider upgrading. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: dtrace in FreeBSD 7.2
Thanks On 2011-09-16 23:14, Julian Elischer wrote: On 9/16/11 1:30 PM, Charlie Martin wrote: I need to add some custom static dtrace probes in 7.2 apps; the online documentation refers only to 9 however. Can someone tell me how to replace what's done in bsd.dtrace.mk for 7.2? I' am not sure that user space dtrace suport was available in 7.2.. My memory is that it came in 8.x but I'm not certain. Thanks Charlie -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
dtrace in FreeBSD 7.2
I need to add some custom static dtrace probes in 7.2 apps; the online documentation refers only to 9 however. Can someone tell me how to replace what's done in bsd.dtrace.mk for 7.2? Thanks Charlie -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Understanding panic and exit in the kernel
On 2011-09-07 12:53, m...@freebsd.org wrote: For my immediate purposes, I'd be happy with any way in which I could > brutally kill the kernel and force it to the debugger, say by replacing the > panic call with a printf followed by "1/0;". But I'm a little confused by > the panic.c code -- it prints the arguments using a var_args, and then calls > "exit(1);' What file are you looking in? The kernel panic() is in sys/kern/kern_shutdown.c, not sys/boot/common/panic.c. It will optionally call kdb_enter_why() and then boot(). Bingo, that's got to help. This makes a lot more sense. Do you have the debug.debugger_on_panic sysctl set to 1? Yes -- and panic does so *except* in the version with those changes to queue.h. Thanks, matthew -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Understanding panic and exit in the kernel
I'm still pursuing a FreeBSD bug in "7.2-PRERELEASE FreeBSD" -- and yes,
we know this is wildly out of date, but it's not feasible to upgrade
right now -- and while trying to backport a fix suggested here
http://permalink.gmane.org/gmane.os.freebsd.current/134266 I got a
situation where the panic calls in one of these two macros from
sys/queue.h
#defineQMD_LIST_CHECK_NEXT(elm, field) do {\
if (LIST_NEXT((elm), field) != NULL &&\
LIST_NEXT((elm), field)->field.le_prev !=\
&((elm)->field.le_next))\
panic("Bad link elm %p next->prev != elm", (elm));\
} while (0)
#defineQMD_LIST_CHECK_PREV(elm, field) do {\
if (*(elm)->field.le_prev != (elm))\
panic("Bad link elm %p prev->next != elm", (elm));\
} while (0)
print the message, but *don't* then proceed to drop to the debugger --
instead the system hangs, with the CPU running but I had no luck getting
its attention to force it to the debugger.
I'm not clear just what could be causing the hangup.
For my immediate purposes, I'd be happy with any way in which I could
brutally kill the kernel and force it to the debugger, say by replacing
the panic call with a printf followed by "1/0;". But I'm a little
confused by the panic.c code -- it prints the arguments using a
var_args, and then calls "exit(1);'
So my questions:
(1) will my brutal method actually force what I want or am I
misunderstanding something
(2) *which* of the several implementations of "int exit(int)" or similar
is the one called in the FreeBSD kernel?
(3) and how then does exit work?
--
Charles R. (Charlie) Martin
Senior Software Engineer
SGI logo
1900 Pike Road
Longmont, CO 80501
Phone: 303-532-0209
E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com>
Website: www.sgi.com <http://www.sgi.com>
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
Sorry, I mean *our* build, and having just made it work by running our build and making sure which modules I was getting, I'm pretty confident. On 2011-09-02 14:48, Garrett Cooper wrote: On Fri, Sep 2, 2011 at 1:37 PM, Charlie Martin wrote: All right, now it works. There's some flaw in the build which causes the FreeBSD 8 to get the modules built for FreeBSD 7. That assumption seems questionable. Do you have more details you can provide (i.e. where each source tree is located, how you're building things, etc)? Thanks, -Garrett -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
All right, now it works. There's some flaw in the build which causes the FreeBSD 8 to get the modules built for FreeBSD 7. Thanks all. On 2011-09-02 13:57, Charlie Martin wrote: On 2011-09-02 13:35, Andriy Gapon wrote: on 02/09/2011 19:55 Charlie Martin said the following: Yeah, and the same overall Makefile is successfully building in dtrace on the other part of the product, which is actually using FreeBSD 7-something. I'm sure there's some tiny difference, but it would sure help to have some idea of what those dmesg messages are telling me. It's not that the .ko's are missing, I checked that first thing. The message was expressly telling you that your cyclic.ko and your kernel are out of sync - built from different sources. Thanks. Puzzling ... it *seems* to be the same sources. On 2011-09-01 19:04, Brandon Gooch wrote: I'm guessing you've read this: http://wiki.freebsd.org/DTrace Make certain you've configure your kernel correctly, and that you've rebuilt your kernel and modules... -Brandon -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
On 2011-09-02 13:35, Andriy Gapon wrote: on 02/09/2011 19:55 Charlie Martin said the following: Yeah, and the same overall Makefile is successfully building in dtrace on the other part of the product, which is actually using FreeBSD 7-something. I'm sure there's some tiny difference, but it would sure help to have some idea of what those dmesg messages are telling me. It's not that the .ko's are missing, I checked that first thing. The message was expressly telling you that your cyclic.ko and your kernel are out of sync - built from different sources. Thanks. Puzzling ... it *seems* to be the same sources. On 2011-09-01 19:04, Brandon Gooch wrote: I'm guessing you've read this: http://wiki.freebsd.org/DTrace Make certain you've configure your kernel correctly, and that you've rebuilt your kernel and modules... -Brandon -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
Yeah, and the same overall Makefile is successfully building in dtrace on the other part of the product, which is actually using FreeBSD 7-something. I'm sure there's some tiny difference, but it would sure help to have some idea of what those dmesg messages are telling me. It's not that the .ko's are missing, I checked that first thing. On 2011-09-01 19:04, Brandon Gooch wrote: I'm guessing you've read this: http://wiki.freebsd.org/DTrace Make certain you've configure your kernel correctly, and that you've rebuilt your kernel and modules... -Brandon -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
okay, dmesg begins to give a clue. Here's a question: if this FreeBSD 8 is being built on a FreeBSD 7 machine, would that account for it? On 2011-09-01 11:23, Garrett Cooper wrote: On Thu, Sep 1, 2011 at 9:56 AM, Charlie Martin wrote: Okay, I'll grant this is probably a horrid noob question, but then on the Free kernel I'm a horrid noob so I guess it makes sense. This is for FreeBSD FreeBSD psmdev1 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 per uname -a. We have a FreeBSD based product on the AMD64 architecture; I'm trying to enable DTrace. The file amd64/conf/GENERIC with which the kernel was compiled has the required lines options KDTRACE_FRAME# Ensure frames are compiled in options KDTRACE_HOOKS# Kernel DTrace hooks options DDB_CTF # for DTrace but when I try kldload dtraceall I get kldload: can't load dtraceall: Exec format error From Google I get that this probably means some mismatch in compiles, but I'm unclear what to look for. Also, another big part of the product, compiled from the same master Makefile, *does* have dtrace enabled successfully. Hints, suggestions, and pointers to documentation gleefully accepted. What does dmesg say? Thanks, -Garrett -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: kldload dtraceall exec format error
On 2011-09-01 11:23, Garrett Cooper wrote: On Thu, Sep 1, 2011 at 9:56 AM, Charlie Martin wrote: Okay, I'll grant this is probably a horrid noob question, but then on the Free kernel I'm a horrid noob so I guess it makes sense. This is for FreeBSD FreeBSD psmdev1 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 per uname -a. We have a FreeBSD based product on the AMD64 architecture; I'm trying to enable DTrace. The file amd64/conf/GENERIC with which the kernel was compiled has the required lines options KDTRACE_FRAME# Ensure frames are compiled in options KDTRACE_HOOKS# Kernel DTrace hooks options DDB_CTF # for DTrace but when I try kldload dtraceall I get kldload: can't load dtraceall: Exec format error From Google I get that this probably means some mismatch in compiles, but I'm unclear what to look for. Also, another big part of the product, compiled from the same master Makefile, *does* have dtrace enabled successfully. Hints, suggestions, and pointers to documentation gleefully accepted. What does dmesg say? Thanks, -Garrett link_elf_obj: symbol lapic_cyclic_clock_func undefined linker_load_file: Unsupported file type KLD profile.ko: depends on cyclic - not available or version mismatch linker_load_file: Unsupported file type KLD dtraceall.ko: depends on profile - not available or version mismatch linker_load_file: Unsupported file type link_elf_obj: symbol lapic_cyclic_clock_func undefined linker_load_file: Unsupported file type KLD profile.ko: depends on cyclic - not available or version mismatch linker_load_file: Unsupported file type KLD dtraceall.ko: depends on profile - not available or version mismatch linker_load_file: Unsupported file type Aha, dmesg. Thanks. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
kldload dtraceall exec format error
Okay, I'll grant this is probably a horrid noob question, but then on the Free kernel I'm a horrid noob so I guess it makes sense. This is for FreeBSD FreeBSD psmdev1 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 per uname -a. We have a FreeBSD based product on the AMD64 architecture; I'm trying to enable DTrace. The file amd64/conf/GENERIC with which the kernel was compiled has the required lines options KDTRACE_FRAME# Ensure frames are compiled in options KDTRACE_HOOKS# Kernel DTrace hooks options DDB_CTF # for DTrace but when I try kldload dtraceall I get kldload: can't load dtraceall: Exec format error From Google I get that this probably means some mismatch in compiles, but I'm unclear what to look for. Also, another big part of the product, compiled from the same master Makefile, *does* have dtrace enabled successfully. Hints, suggestions, and pointers to documentation gleefully accepted. -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Where to ask about a 7.2 bug, and debugging sys/queue.h errors
We're having a crash in some internal code running on FreeBSD 7.2 (specifically 7.2-PRERELEASE FreeBSD 7.2-PRERELEASE and yeah, I know it's quite a bit behind) in which after 18-30 hours of running load tests, the code panics with: panic: Bad link elm 0xff0044c09600 next->prev != elm cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at 0x8019119a = db_trace_self_wrapper+0x2a panic() at 0x80307c72 = panic+0x182 devfs_populate_loop() at 0x802a43a8 = devfs_populate_loop+0x548 First question: where's the most appropriate place to ask about this kind of bug on a back version. Second: does this remind anyone of any bugs? Googling came up with a few somewhat similar things but hasn't provided much insight so far. Third: I tried compiling with the sys/queue.h QUEUE_MACRO_DEBUG defined in order to get more useful information from the panic. The kernel build fails in pmap.c when this macro is defined, giving an error saying the CTASSERT macro is resolving to a negative array size. Is there any particular secret to using this macro (like, no one goes there any more?) Thanks -- Charles R. (Charlie) Martin Senior Software Engineer SGI logo 1900 Pike Road Longmont, CO 80501 Phone: 303-532-0209 E-Mail: crmar...@sgi.com <mailto:crmar...@sgi.com> Website: www.sgi.com <http://www.sgi.com> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
