Alan, We fought a tiny issue last week that effected both worker (threaded) and pre- fork (non-threaded) machines that used DBI for mysql connections.
Symptom were segmentation faults at seemingly random times over the past few months, especially in the early morning hours on some slow weekends, in certain applications. They ultimately produced proxy errors at the load balancer. http://rt.cpan.org/Public/Bug/Display.html?id=37027 So, our core dump was producing the following with gdb: (gdb) #0 0x00002b8c60fae5ee in mysql_ping () from /usr/lib/libmysqlclient_r.so.15 A simple fix was to just upgrade affected machine's DBI from CPAN. Anyway, recompiling perl to trap and dump (with meaningful info) per the documents you cite below was a big help - with core dumping in place, discovering the actual cause was kind of like tiger hunting with big gun; we set the trap, waited in the bushes... then BAM!. Bryan http://www.shiftboard.com Online Scheduling On Monday 13 July 2009 12:15:55 Alan McKay wrote: > Hey folks, > > I'm running CentOS 5.2 with a custom built apache not from Yum. > > I am following instructions here > http://perl.apache.org/docs/2.0/devel/debug/c.html#Getting_the_core_File_Du >mped > > and here : > http://perl.apache.org/docs/2.0/devel/debug/c.html#Analyzing_Dumped_Core_Fi >les > > I have mod_perl built with symbols and reproduce my problem, but I'm > not having much luck with gdb. I really have no experience with this > aspect of Linux so am shooting in the dark a bit here. > > I also have no idea whether I'm "threaded" or "unthreaded", so I tried > both of the "curinfo" macros with no luck. > > Here is the output of "bt full" which clearly shows a few "my_perl" > entries. But I'm just not clear how to find out what part of our > Perl code is causing the issue. > > (gdb) bt full > #0 0x0044e172 in Perl_cv_const_sv () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #1 0x00442a7c in Perl_yylex () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #2 0x0044b295 in Perl_yyparse () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #3 0x004b8200 in Perl_pp_range () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #4 0x004ba0b2 in Perl_pp_require () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #5 0x004ba7ad in Perl_pp_dofile () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #6 0x0048488f in Perl_runops_standard () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #7 0x00424ffe in Perl_magicname () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #8 0x00429806 in Perl_call_sv () from > /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so > No symbol table info available. > #9 0x00140a7f in modperl_callback (my_perl=0x9476458, > handler=0xaccc268, p=0xaf2a840, r=0xaf2a880, s=0xad0bad0, > args=0xbd473e8) at modperl_callback.c:101 > items = <value optimized out> > cv = <value optimized out> > sp = (SV **) 0xafead24 > status = 0 > #10 0x0014117a in modperl_callback_run_handlers (idx=6, type=4, > r=0xaf2a880, c=0x0, s=0xad0bad0, pconf=0x0, plog=0x0, ptemp=0x0, > run_mode=MP_HOOK_RUN_FIRST) > at modperl_callback.c:262 > my_perl = (PerlInterpreter *) 0x9476458 > interp = (modperl_interp_t *) 0xa965888 > scfg = (modperl_config_srv_t *) 0x8d437b8 > dcfg = (modperl_config_dir_t *) 0xaf2a2e0 > rcfg = (modperl_config_req_t *) 0xaf2b5f0 > handlers = (modperl_handler_t **) 0xaccc298 > p = (apr_pool_t *) 0xaf2a840 > av = (MpAV *) 0xaccc280 > avp = <value optimized out> > i = 0 > status = 0 > desc = 0x158cd3 "PerlResponseHandler" > av_args = (AV *) 0xbd473e8 > #11 0x0014180a in modperl_callback_per_dir (idx=6, r=0xaf2a880, > run_mode=MP_HOOK_RUN_FIRST) at modperl_callback.c:369 > No locals. > #12 0x0013a79f in modperl_response_handler_run (r=0xaf2a880, finish=0) > at mod_perl.c:1000 > retval = <value optimized out> > #13 0x0013a96b in modperl_response_handler_cgi (r=0xaf2a880) at > mod_perl.c:1100 dcfg = (modperl_config_dir_t *) 0xaf2a2e0 > h_stdin = (GV *) 0xc2d7a20 > h_stdout = (GV *) 0xb4ba144 > retval = -1 > rc = <value optimized out> > rcfg = (modperl_config_req_t *) 0xaf2b5f0 > my_perl = (PerlInterpreter *) 0x9476458 > interp = (modperl_interp_t *) 0xa965888 > #14 0x08076a79 in ap_run_handler (r=0xaf2a880) at config.c:157 > n = 6 > rv = 40 > #15 0x08079be7 in ap_invoke_handler (r=0xaf2a880) at config.c:372 > handler = 0x0 > result = <value optimized out> > old_handler = 0xaccbe38 "perl-script" > ignore = <value optimized out> > #16 0x08090798 in ap_process_request (r=0xaf2a880) at http_request.c:258 > ---Type <return> to continue, or q <return> to quit--- > access_status = 0 > #17 0x0808da3b in ap_process_http_connection (c=0xaf1a9d8) at > http_core.c:190 r = (request_rec *) 0xaf2a880 > csd = (apr_socket_t *) 0x0 > #18 0x0807dab9 in ap_run_process_connection (c=0xaf1a9d8) at > connection.c:43 n = 1 > rv = 40 > #19 0x080a3dc5 in child_main (child_num_arg=<value optimized out>) at > prefork.c:650 > numdesc = 0 > pdesc = (const apr_pollfd_t *) 0x0 > current_conn = (conn_rec *) 0xaf1a9d8 > csd = (void *) 0xaf1a840 > ptrans = (apr_pool_t *) 0xaf1a800 > allocator = (apr_allocator_t *) 0x9301738 > status = <value optimized out> > i = <value optimized out> > lr = <value optimized out> > pollset = (apr_pollset_t *) 0xaf18850 > sbh = (ap_sb_handle_t *) 0xaf18848 > bucket_alloc = (apr_bucket_alloc_t *) 0xaf1c808 > last_poll_idx = 0 > #20 0x080a4087 in make_child (s=0x8d2fd70, slot=28) at prefork.c:746 > pid = 0 > #21 0x080a49e0 in ap_mpm_run (_pconf=0x8d070a8, plog=0x8d451a0, > s=0x8d2fd70) at prefork.c:881 > status = 0 > pid = {pid = -1, in = 0x8d01250, out = 0x2, err = 0xd070a8} > child_slot = <value optimized out> > exitwhy = APR_PROC_EXIT > processed_status = <value optimized out> > index = <value optimized out> > remaining_children_to_start = 0 > rv = <value optimized out> > #22 0x08064645 in main (argc=147869856, argv=0xaf187f8) at main.c:740 > c = 0 '\0' > configtestonly = 0 > confname = 0x80ae357 "conf/httpd.conf" > def_server_root = 0x80ae367 "/usr/local/apache2" > temp_error_log = 0x0 > error = <value optimized out> > process = (process_rec *) 0x8d05130 > server_conf = <value optimized out> > pglobal = (apr_pool_t *) 0x8d050a0 > pconf = (apr_pool_t *) 0x8d070a8 > plog = (apr_pool_t *) 0x8d451a0 > ptemp = (apr_pool_t *) 0x8d491b0 > pcommands = (apr_pool_t *) 0x8d090b0 > opt = (apr_getopt_t *) 0x8d09150 > rv = 0 > optarg = 0x3b6ff4 "\234m;" > (gdb)
