Re: Solaris /usr/proc/bin/pstack functionality?
I've submitted PR33874 that implements a version of pstack for FreeBSD. (er, actually due to difficulties with our mail server, it's logged twice, the second being 33873: can someone delete one of them?) Jos, if its still of use to you, can you see if it works? Bernd Walter wrote: On Thu, Jan 03, 2002 at 02:02:09PM +0100, Oliver Fromme wrote: Jos Backus [EMAIL PROTECTED] wrote: - Forwarded message from Justin Erenkrantz [EMAIL PROTECTED] - +1. =) I've talked to the FreeBSD people and they just laugh maniacally when I ask for a truss that follows children. AIUI, NetBSD has this, so it is possible to port these changes over, but it requires an overhaul to procfs from what I've been told. FreeBSD has a long way to get the stellar debugging capabilities of Solaris. -- justin I think that the output from FreeBSD's truss is pretty pathetic and unusable. ktrace is somewhat better. Mmmm - I can't compare... On alpha: === Building for strace-4.4 cc -b alpha--freebsd5.0 -Wall -DHAVE_CONFIG_H -I. -Ifreebsd/alpha -I./freebsd/alpha -Ifreebsd -I./freebsd -D_GNU_SOURCE -O -pipe -mcpu=ev56 -c strace.c In file included from strace.c:34: defs.h:96: #error FreeBSD support is only for i386 arch right now. strace.c: In function `trace': strace.c:1461: warning: label `FOUND' defined but not used *** Error code 1 I asume this should be tagged i386 only in the port. On i386: ticso@cicely5# strace sleep 1 strace: PIOCSTATUS: Inappropriate ioctl for device trouble opening proc file Exit 1 PS: I tried strace under -stable only. I don't know if it works under -current as well. Seems not. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
On Thu, Jan 03, 2002 at 02:02:09PM +0100, Oliver Fromme wrote: Jos Backus [EMAIL PROTECTED] wrote: - Forwarded message from Justin Erenkrantz [EMAIL PROTECTED] - +1. =) I've talked to the FreeBSD people and they just laugh maniacally when I ask for a truss that follows children. AIUI, NetBSD has this, so it is possible to port these changes over, but it requires an overhaul to procfs from what I've been told. FreeBSD has a long way to get the stellar debugging capabilities of Solaris. -- justin I think that the output from FreeBSD's truss is pretty pathetic and unusable. ktrace is somewhat better. Mmmm - I can't compare... On alpha: === Building for strace-4.4 cc -b alpha--freebsd5.0 -Wall -DHAVE_CONFIG_H -I. -Ifreebsd/alpha -I./freebsd/alpha -Ifreebsd -I./freebsd -D_GNU_SOURCE -O -pipe -mcpu=ev56 -c strace.c In file included from strace.c:34: defs.h:96: #error FreeBSD support is only for i386 arch right now. strace.c: In function `trace': strace.c:1461: warning: label `FOUND' defined but not used *** Error code 1 I asume this should be tagged i386 only in the port. On i386: ticso@cicely5# strace sleep 1 strace: PIOCSTATUS: Inappropriate ioctl for device trouble opening proc file Exit 1 PS: I tried strace under -stable only. I don't know if it works under -current as well. Seems not. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
:Here's what pstack does: : : pstackPrint a hex+symbolic stack trace for each lwp in : each process. : :Solaris truss(1) has this: : : -lIncludes the id of the responsible lightweight : process (LWP) with each line of trace output. If : -f is also specified, both the process-id and the : LWP-id are included. : :Justin says: :Yup, we're all scratching our heads right now at some weirdness :going on with select()/poll(), but all we can see is the kernel :primitives. *sigh* Our job would be a lot easier if we had :pstack. =) : :So the question is, does ktrace in fact have this functionality? He's talking :about LWPs, which I am assuming (please correct me if I am wrong) equates to :libc_r on FreeBSD. : :Fwiw, I'm asking as an interested 3rd party. Thanks! : : -Matt : :-- :Jos Backus _/ _/_/_/Santa Clara, CA : _/ _/ _/ : _/ _/_/_/ :_/ _/ _/_/ :[EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer; Hmm. I'm looking at the code. We do have a utrace() system call which allows a userland process to create a tracing record. In regards to light weight processes, or threads as we call them, with libc_r they are of course longjmp/select-or-poll-based rather then rfork() based. ktrace effectively only logs the process id, found in ktr_header (see sys/ktrace.h), but it should be possible to create a userland structure that we log with utrace() that includes the thread id whenever libc_r does an internal context switch. kdump() could then detect this and supply the proper thread id. It also would not be too big a deal to change the ktrace header structure to include additional information. We actually have a garbage caddr_t stored in there that holds the kernel buffer address but is also written out to the file. That space could be reclaimed to store a thread id (in the tracing file) but there would still have to be some way for libc_r to register the thread id during a context switch. Using utrace() is kinda hokey. A new system call or something. I dunno about pstack. What does it do? Dump stack backtraces for the threads? libc_r does use a fairly well-defined thread stack arrangement. It should be possible to write a program (or a gdb script) to track the stacks down and dump their backtraces. Tracing userland procedure calls is more difficult. I do not believe that the i386 has a separate trace trap for subroutine calls, it just has a single-step trace trap, so a userland trace would have to be built into the program (e.g. like profiling is with -pg). -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
On Thu, Jan 03, 2002 at 12:29:56AM -0800, Matthew Dillon wrote: I dunno about pstack. What does it do? Dump stack backtraces for the threads? libc_r does use a fairly well-defined thread stack arrangement. It should be possible to write a program (or a gdb script) to track the stacks down and dump their backtraces. Tracing userland procedure calls is more difficult. I do not believe that the i386 has a separate trace trap for subroutine calls, it just has a single-step trace trap, so a userland trace would have to be built into the program (e.g. like profiling is with -pg). pstack on Solaris includes user and system calls for all threads (LWP or kernel) in a process (if built with -g). This is even on x86 machines. I don't have any more info than this though. However, even if we could get just the stack portion that libc_r maintains, it would be of great benefit. In Apache 2.0-land, we are currently trying to sift through potential libc_r problems. We hope to file bugs as we can isolate them - as we did for sendfile(2). We have currently identified three potential issues, but we are having trouble narrowing them down past that and generating reproducable test cases that will be of help to FreeBSD developers. -- justin To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
Jos Backus [EMAIL PROTECTED] wrote: - Forwarded message from Justin Erenkrantz [EMAIL PROTECTED] - +1. =) I've talked to the FreeBSD people and they just laugh maniacally when I ask for a truss that follows children. AIUI, NetBSD has this, so it is possible to port these changes over, but it requires an overhaul to procfs from what I've been told. FreeBSD has a long way to get the stellar debugging capabilities of Solaris. -- justin I think that the output from FreeBSD's truss is pretty pathetic and unusable. ktrace is somewhat better. Recently I discovered that there is a version of strace in our ports collection. I installed it and noticed that its output is a lot better. Since then I use strace exclusively for debugging and I'm very happy with it. strace follows children when you use the -f option (this is similar to ktrace's -i option). Regards Oliver PS: I tried strace under -stable only. I don't know if it works under -current as well. -- Oliver Fromme, secnetix GmbH Co KG, Oettingenstr. 2, 80538 München Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. All that we see or seem is just a dream within a dream (E. A. Poe) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Solaris /usr/proc/bin/pstack functionality?
See the post below. Is there a way to get at this information in FreeBSD? Thanks, Jos - Forwarded message from Justin Erenkrantz [EMAIL PROTECTED] - Date: Wed, 2 Jan 2002 14:02:24 -0800 From: Justin Erenkrantz [EMAIL PROTECTED] To: Aaron Bannert [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: 2.0.30-dev load spiking [was: upgrade to FreeBSD 4.5-PRERELEASE] Reply-To: [EMAIL PROTECTED] Mail-Followup-To: Justin Erenkrantz [EMAIL PROTECTED], Aaron Bannert [EMAIL PROTECTED], [EMAIL PROTECTED] On Wed, Jan 02, 2002 at 12:52:57PM -0800, Aaron Bannert wrote: On Wed, Jan 02, 2002 at 12:46:46PM -0800, Brian Pane wrote: Do you have a way to take a snapshot of each httpd process's stack backtrace? On Solaris, I'd do this by running /usr/proc/bin/pstack on each pid; I don't know if FreeBSD has a similar functionality. This would give us a picture of what all those runnable processes are doing. Ooh, if it does I'd love to find out how. Same goes for a truss that can follow children, and a ps command to tell me how many threads are in a process./wishlist +1. =) I've talked to the FreeBSD people and they just laugh maniacally when I ask for a truss that follows children. AIUI, NetBSD has this, so it is possible to port these changes over, but it requires an overhaul to procfs from what I've been told. FreeBSD has a long way to get the stellar debugging capabilities of Solaris. -- justin - End forwarded message - -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer; To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
: :See the post below. Is there a way to get at this information in FreeBSD? : :Thanks, :Jos Well, ktrace -i will certainly follow children. In fact, ktrace can attach to all current children (-d) of a process as well as attach to new children. Yahoo found a few bugs in ktrace by running 'ktrace -i -d -p 1'. Think about what that does :-) If ktrace can do it, I'm sure truss could be made to do it. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Solaris /usr/proc/bin/pstack functionality?
On Wed, Jan 02, 2002 at 11:34:59PM -0800, Matthew Dillon wrote: Well, ktrace -i will certainly follow children. In fact, ktrace can attach to all current children (-d) of a process as well as attach to new children. Yahoo found a few bugs in ktrace by running 'ktrace -i -d -p 1'. Think about what that does :-) If ktrace can do it, I'm sure truss could be made to do it. Here's what pstack does: pstackPrint a hex+symbolic stack trace for each lwp in each process. Solaris truss(1) has this: -lIncludes the id of the responsible lightweight process (LWP) with each line of trace output. If -f is also specified, both the process-id and the LWP-id are included. Justin says: Yup, we're all scratching our heads right now at some weirdness going on with select()/poll(), but all we can see is the kernel primitives. *sigh* Our job would be a lot easier if we had pstack. =) So the question is, does ktrace in fact have this functionality? He's talking about LWPs, which I am assuming (please correct me if I am wrong) equates to libc_r on FreeBSD. Fwiw, I'm asking as an interested 3rd party. Thanks! -Matt -- Jos Backus _/ _/_/_/Santa Clara, CA _/ _/ _/ _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer; To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
