cvs commit: apache-devsite debugging.html
dirkx 99/12/03 02:37:34 Modified:.debugging.html Log: Added setuid() / coreadm trick Revision ChangesPath 1.8 +19 -0 apache-devsite/debugging.html Index: debugging.html === RCS file: /x3/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.7 retrieving revision 1.8 diff -u -r1.7 -r1.8 --- debugging.html1998/10/09 23:09:17 1.7 +++ debugging.html1999/12/03 10:37:32 1.8 @@ -23,6 +23,7 @@ Using 'truss/trace/strace' to trace system calls and signals Getting the server to dump core +Solaris 2.7 and coredumps Getting and analyzing a TCP packet trace @@ -275,7 +276,25 @@ and then look at the backtrace as discussed above for gdb. +Solaris 2.7 and coredumps +On Solaris 2.7 use coreadm to make setuid() +processes actually dump core. By default an setuid() process does not +dump core. + +Jens-Uwe Mager wrote: + +For example I am using: + + # coreadm + global core file pattern: /var/core/core.%f.%p.u%u + init core file pattern: core + global core dumps: enabled +per-process core dumps: enabled + global setid core dumps: enabled + per-process setid core dumps: enabled + global core dump logging: disabled + Getting and analyzing a TCP packet trace
cvs commit: apache-devsite debugging.html
fielding98/10/09 16:09:18 Modified:.debugging.html Log: Add brief notes about getting and analyzing a TCP packet trace. This is the last of my remembered tips, so feel free to add stuff if you have any more. Revision ChangesPath 1.7 +21 -3 apache-devsite/debugging.html Index: debugging.html === RCS file: /export/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.6 retrieving revision 1.7 diff -u -r1.6 -r1.7 --- debugging.html1998/10/09 22:14:05 1.6 +++ debugging.html1998/10/09 23:09:17 1.7 @@ -14,9 +14,8 @@ Apache Debugging Guide -This document is a collection of semi-random and unorganized notes -regarding tools and techniques for debugging Apache, and Apache -modules. +This document is a collection of notes regarding tools and techniques +for debugging Apache and Apache modules. Using 'gdb' @@ -24,6 +23,7 @@ Using 'truss/trace/strace' to trace system calls and signals Getting the server to dump core +Getting and analyzing a TCP packet trace @@ -275,6 +275,24 @@ and then look at the backtrace as discussed above for gdb. + + + +Getting and analyzing a TCP packet trace + + +This is more difficult than I have time to describe at the moment. +Here are some pointers to useful discussions and tools: + +http://jarok.cs.ohiou.edu/software/tcptrace/pcap.html";>tools for +producing TCP dumps +http://jarok.cs.ohiou.edu/software/tcptrace/tcptrace.html";> +tcptrace is a TCP dump file analysis tool +http://HTTP.CS.Berkeley.EDU/~daw/mike/";>tcpshow is another + +There is also a simple ASCII viewer for TCP dump traces in the Apache +repository in the file +http://www.apache.org/websrc/cvsweb.cgi/apache-1.3/src/test/tcpdumpscii.txt";>src/test/tcpdumpscii.txt.
cvs commit: apache-devsite debugging.html
fielding98/10/09 15:14:05 Modified:.debugging.html Log: Add information on getting a core dump. Revision ChangesPath 1.6 +42 -6 apache-devsite/debugging.html Index: debugging.html === RCS file: /export/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.5 retrieving revision 1.6 diff -u -r1.5 -r1.6 --- debugging.html1998/10/09 21:22:27 1.5 +++ debugging.html1998/10/09 22:14:05 1.6 @@ -23,11 +23,12 @@ Getting a live backtrace Using 'truss/trace/strace' to trace system calls and signals +Getting the server to dump core -Using 'gdb' +Using 'gdb' If you use the gcc or egcs compilers, it is likely that the best debugger for your system is gdb. This is only a brief summary of how @@ -164,7 +165,7 @@ -Getting a live backtrace +Getting a live backtrace A backtrace will let you know the hierarchy of procedures that were called to get to a particular point in the process. On some platforms @@ -216,8 +217,8 @@ -Using 'truss/trace/strace' to -trace system calls and signals +Using 'truss/trace/strace' to +trace system calls and signals Most Unix-based systems have at least one command for displaying a trace of system calls and signals as they are accessed by a running @@ -242,8 +243,43 @@ -Got more tips? Send 'em to mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]. Thanks! +Getting the server to dump core + +Strangely enough, sometimes you actually want to force the server +to crash so that you can get a look at some nutty behavior. Normally +this can be done simply by using the gcore command. +However, for security reasons, most Unix systems do not allow a setuid +process to dump core, since the file contents might reveal something +that is supposed to be protected in memory. + +Here is one way to get a core file from a setuid Apache httpd process +on Solaris, without knowing which httpd child might be the one to die +[note: it is probably easier to use the MaxClients trick in the first +section above]. + +# for pid in `ps -eaf | fgrep httpd | cut -d' ' -f4` +do + truss -f -l -t\!all -S SIGSEGV -p $pid 2>&1 | egrep SIGSEGV & +done + + +The http://www.dejanews.com/=zzz_maf/getdoc.xp?AN=352257973";> +undocumented '-S' flag to truss will halt the process in place +upon receipt of a given signal (SIGSEGV in this case). +At this point you can use: + + +# gcore PID + + +and then look at the backtrace as discussed above for gdb. + + + + + +Got more tips? Send 'em to +mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]. Thanks!
cvs commit: apache-devsite debugging.html
fielding98/10/09 14:22:27 Modified:.debugging.html Log: Add information about using truss Revision ChangesPath 1.5 +31 -2 apache-devsite/debugging.html Index: debugging.html === RCS file: /export/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.4 retrieving revision 1.5 diff -u -r1.4 -r1.5 --- debugging.html1998/10/09 20:53:45 1.4 +++ debugging.html1998/10/09 21:22:27 1.5 @@ -21,11 +21,13 @@ Using 'gdb' Getting a live backtrace +Using 'truss/trace/strace' to +trace system calls and signals -Using 'gdb' +Using 'gdb' If you use the gcc or egcs compilers, it is likely that the best debugger for your system is gdb. This is only a brief summary of how @@ -162,7 +164,7 @@ -Getting a live backtrace +Getting a live backtrace A backtrace will let you know the hierarchy of procedures that were called to get to a particular point in the process. On some platforms @@ -210,6 +212,33 @@ #4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534 (gdb) + + + + +Using 'truss/trace/strace' to +trace system calls and signals + +Most Unix-based systems have at least one command for displaying +a trace of system calls and signals as they are accessed by a running +process. This command is called truss on most SVR4-based +systems and either trace or strace on many +other systems. + +A useful tip for using the truss command on Solaris is +the -f option; it tells truss to follow and continue tracing +any child processes forked by the main process. The easiest way to get +a full trace of a server is to do something like: + +% truss -f httpd -d /usr/local/apache >& outfile + +and let it run through a few requests before killing the parent. You can +then view the entire trace in outfile, or use something like + +% egrep '^10698:' outfile + +to view just the trace of the process id 10698. +
cvs commit: apache-devsite debugging.html
fielding98/10/09 13:53:45 Modified:.debugging.html Log: Add notes about getting a live backtrace. Revision ChangesPath 1.4 +54 -2 apache-devsite/debugging.html Index: debugging.html === RCS file: /export/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.3 retrieving revision 1.4 diff -u -r1.3 -r1.4 --- debugging.html1998/10/09 20:28:59 1.3 +++ debugging.html1998/10/09 20:53:45 1.4 @@ -20,6 +20,7 @@ Using 'gdb' +Getting a live backtrace @@ -152,12 +153,63 @@ If you are debugging a system crash and you have a core file from the crash, then do the following: -% gdb httpd -c core -(gdb) where +% gdb httpd -c core +(gdb) where and it will (hopefully) print a stack backtrace of where the core dump occurred during processing. + + + +Getting a live backtrace + +A backtrace will let you know the hierarchy of procedures that +were called to get to a particular point in the process. On some platforms +you can get a live backtrace of any process. + +For SVR4-based variants of Unix, the pstack command for proc can +be used to display a a live backtrace. For example, on Solaris it looks like + +% /usr/proc/bin/pstack 10623 +10623: httpd -d /usr/local/apache + ef5b68d8 poll (efffcd08, 0, 3e8) + ef5d21e0 select (0, ef612c28, 0, 0, 3e8, efffcd08) + 288 + 00042574 wait_or_timeout (0, 75000, 75000, 7c3e8, 60f40, 52c00) + 78 + 00044310 standalone_main (5fd68, 75800, 75c00, 75000, 2, 64) + 240 + 000449f4 main (3, efffeee4, efffeef4, 75fe4, 1, 0) + 374 + 000162fc _start (0, 0, 0, 0, 0, 0) + 5c + + +Another technique is to use gdb to attach to the running process +and then using "where" to print the backtrace, as in + +% gdb httpd 10623 +GDB is free software and you are welcome to distribute copies of it + under certain conditions; type "show copying" to see the conditions. +There is absolutely no warranty for GDB; type "show warranty" for details. +GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5), +Copyright 1996 Free Software Foundation, Inc... + +/usr/local/apache/src/10623: No such file or directory. +Attaching to program `/usr/local/apache/src/httpd', process 10623 +Reading symbols from /usr/lib/libsocket.so.1...done. +Reading symbols from /usr/lib/libnsl.so.1...done. +Reading symbols from /usr/lib/libc.so.1...done. +Reading symbols from /usr/lib/libdl.so.1...done. +Reading symbols from /usr/lib/libintl.so.1...done. +Reading symbols from /usr/lib/libmp.so.1...done. +Reading symbols from /usr/lib/libw.so.1...done. +Reading symbols from /usr/platform/SUNW,Ultra-1/lib/libc_psr.so.1...done. +0xef5b68d8 in () +(gdb) where +#0 0xef5b68d8 in () +#1 0xef5d21e8 in select () +#2 0x4257c in wait_or_timeout (status=0x0) at http_main.c:2357 +#3 0x44318 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4273 +#4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534 +(gdb) +
cvs commit: apache-devsite debugging.html
fielding98/10/09 13:29:00 Modified:.debugging.html Log: Add Dean's "MaxClients 1" suggestion. Revision ChangesPath 1.3 +5 -1 apache-devsite/debugging.html Index: debugging.html === RCS file: /export/home/cvs/apache-devsite/debugging.html,v retrieving revision 1.2 retrieving revision 1.3 diff -u -r1.2 -r1.3 --- debugging.html1998/10/09 19:35:51 1.2 +++ debugging.html1998/10/09 20:28:59 1.3 @@ -37,7 +37,11 @@ The only tricky part of running gdb on Apache is forcing the server into a single-process mode so that the parent process being debugged does the request-handling work instead of forking child processes. -We have provided the -X option for that purpose. +We have provided the -X option for that purpose, which will +work fine for most cases. However, some modules don't like starting +up with -X, but are happy if you force only one child to run +(using "MaxClients 1"); you can then use gdb's attach command +to debug the child server. The following example, with user input in green, shows the output of gdb run on a server executable (httpd) in the current
cvs commit: apache-devsite debugging.html
fielding98/10/09 12:35:52
Modified:.debugging.html
Log:
Describe how to use gdb so that we can point to this file when
users ask questions or report problems.
Revision ChangesPath
1.2 +135 -2apache-devsite/debugging.html
Index: debugging.html
===
RCS file: /export/home/cvs/apache-devsite/debugging.html,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -r1.1 -r1.2
--- debugging.html1998/04/20 05:50:13 1.1
+++ debugging.html1998/10/09 19:35:51 1.2
@@ -1,6 +1,16 @@
-
+
+
+
Apache Debugging Guide
-
+
+
+
Apache Debugging Guide
@@ -16,10 +26,133 @@
Using 'gdb'
+If you use the gcc or egcs compilers, it is likely that the best
+debugger for your system is gdb. This is only a brief summary of how
+to run gdb on Apache -- you should look at the info and man files for
+gdb to get more information on gdb commands and common debugging techniques.
+Before running gdb, be sure that the server is compiled with the
+-g option in EXTRA_CFLAGS to include the
+symbol information in the object files.
+
+The only tricky part of running gdb on Apache is forcing the server
+into a single-process mode so that the parent process being debugged
+does the request-handling work instead of forking child processes.
+We have provided the -X option for that purpose.
+
+The following example, with user input in green,
+shows the output of gdb run on a server executable (httpd) in the current
+working directory and using the server root of
/usr/local/apache:
+
+% gdb httpd
+GDB is free software and you are welcome to distribute copies of it
+ under certain conditions; type "show copying" to see the conditions.
+There is absolutely no warranty for GDB; type "show warranty" for
details.
+GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5),
+Copyright 1996 Free Software Foundation, Inc...
+(gdb) b ap_process_request
+Breakpoint 1 at 0x49fb4: file http_request.c, line 1164.
+(gdb) run -X -d /usr/local/apache
+Starting program: /usr/local/apache/src/httpd -X -d /usr/local/apache
+
+[at this point I make a request from another window]
+
+Breakpoint 1, ap_process_request (r=0x95250) at http_request.c:1164
+1164if (ap_extended_status)
+(gdb) s
+1165ap_time_process_request(r->connection->child_num,
START_PREQUEST);
+(gdb) n
+1167process_request_internal(r);
+(gdb) s
+process_request_internal (r=0x95250) at http_request.c:1028
+1028if (!r->proxyreq && r->parsed_uri.path) {
+(gdb) s
+1029access_status = ap_unescape_url(r->parsed_uri.path);
+(gdb) n
+1030if (access_status) {
+(gdb) s
+1036ap_getparents(r->uri); /* OK --- shrinking
transformations... */
+(gdb) n
+1038if ((access_status = location_walk(r))) {
+(gdb) n
+1043if ((access_status = ap_translate_name(r))) {
+(gdb) n
+1048if (!r->proxyreq) {
+(gdb) n
+1053if (r->method_number == M_TRACE) {
+(gdb) n
+1062if (r->proto_num > HTTP_VERSION(1,0) &&
ap_table_get(r->subprocess_env, "downgrade-1.0")) {
+(gdb) n
+1071if ((access_status = directory_walk(r))) {
+(gdb) s
+directory_walk (r=0x95250) at http_request.c:288
+288 core_server_config *sconf =
ap_get_module_config(r->server->module_config,
+(gdb) b ap_send_error_response
+Breakpoint 2 at 0x47dcc: file http_protocol.c, line 2090.
+(gdb) c
+Continuing.
+
+Breakpoint 2, ap_send_error_response (r=0x95250, recursive_error=0)
+at http_protocol.c:2090
+2090BUFF *fd = r->connection->client;
+(gdb) where
+#0 ap_send_error_response (r=0x95250, recursive_error=0)
+at http_protocol.c:2090
+#1 0x49b10 in ap_die (type=403, r=0x95250) at http_request.c:989
+#2 0x49b60 in decl_die (status=403, phase=0x62db8 "check access",
r=0x95250)
+at http_request.c:1000
+#3 0x49f68 in process_request_internal (r=0x95250) at
http_request.c:1141
+#4 0x49fe0 in ap_process_request (r=0x95250) at http_request.c:1167
+#5 0x439d8 in child_main (child_num_arg=550608) at http_main.c:3826
+#6 0x43b5c in make_child (s=0x7c3e8, slot=0, now=907958743)
+at http_main.c:3898
+#7 0x43ca8 in startup_children (number_to_start=6) at http_main.c:3972
+#8 0x44260 in standalone_main (argc=392552, argv=0x75800) at
http_main.c:4250
+#9 0x449fc in main (argc=4, argv=0xefffee8c) at http_main.c:4534
+(gdb) s
+2091int status = r->status;
+(gdb) p status
+$1 = 403
+(gdb)
+
+
+There are a f
cvs commit: apache-devsite debugging.html index.html
brian 98/04/19 22:50:14 Modified:.index.html Added: .debugging.html Log: A meek beginning, but better than sitting around complaining that it doesn't exist. Feel free to add to this! Revision ChangesPath 1.20 +5 -8 apache-devsite/index.html Index: index.html === RCS file: /export/home/cvs/apache-devsite/index.html,v retrieving revision 1.19 retrieving revision 1.20 diff -u -r1.19 -r1.20 --- index.html1998/04/18 07:37:33 1.19 +++ index.html1998/04/20 05:50:13 1.20 @@ -59,17 +59,14 @@ draft form. [Old Guidelines]. (last modified on ) - The Apache coding -style guide + The Apache coding style guide (last modified on ) - Some notes on -the API + Some notes on the API (last modified on ) + + Some notes on debugging +(last modified on ) 1.1 apache-devsite/debugging.html Index: debugging.html === Apache Debugging Guide Apache Debugging Guide This document is a collection of semi-random and unorganized notes regarding tools and techniques for debugging Apache, and Apache modules. Using 'gdb' Using 'gdb' A file in the src/ directory, .gdbinit, provides a useful macro for printing out the contents of a table structure, called dump_table. Got more tips? Send 'em to mailto:[EMAIL PROTECTED]">[EMAIL PROTECTED]. Thanks!
