Re: tracking a coredump problem
I've got some coredumps now, after starting apache mp with apachectl not the init script. gdb shows this : Core was generated by `/usr/local/apache/2.2.11/bin/httpd -f /etc/httpd/conf/httpd-2.2.11-perl.conf -k'. Program terminated with signal 11, Segmentation fault. #0 0x0030a039 in Perl_pp_rv2cv () from /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so bt full shows a load of info but nothing I can immediately identify as a failure. This is its output : #0 0x0030a039 in Perl_pp_rv2cv () from /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so No symbol table info available. #1 0x002dd88f in Perl_runops_standard () from /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so No symbol table info available. #2 0x0027dffe in Perl_magicname () from /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so No symbol table info available. #3 0x00282806 in Perl_call_sv () from /usr/lib/perl5/5.8.8/i386-linux-thread-multi/CORE/libperl.so No symbol table info available. #4 0x00604a7f in modperl_callback (my_perl=0x8eac5f8, handler=0x89b93b8, p=0x95efc50, r=0x95efc90, s=0x94d9ec8, args=0x9688e94) at modperl_callback.c:101 items = value optimized out cv = value optimized out sp = (SV **) 0x970ac98 status = 0 #5 0x0060517a in modperl_callback_run_handlers (idx=6, type=4, r=0x95efc90, c=0x0, s=0x94d9ec8, pconf=0x0, plog=0x0, ptemp=0x0, run_mode=MP_HOOK_RUN_FIRST) at modperl_callback.c:262 my_perl = (PerlInterpreter *) 0x8eac5f8 interp = (modperl_interp_t *) 0x931fe18 scfg = (modperl_config_srv_t *) 0x94dc5e0 dcfg = (modperl_config_dir_t *) 0x94dd4f0 rcfg = (modperl_config_req_t *) 0x95f0a28 handlers = (modperl_handler_t **) 0x89b93e8 p = (apr_pool_t *) 0x95efc50 av = (MpAV *) 0x89b93d0 avp = value optimized out i = 0 status = 0 desc = 0x61ccd3 PerlResponseHandler av_args = (AV *) 0x9688e94 #6 0x0060580a in modperl_callback_per_dir (idx=6, r=0x95efc90, run_mode=MP_HOOK_RUN_FIRST) at modperl_callback.c:369 No locals. #7 0x005fe79f in modperl_response_handler_run (r=0x95efc90, finish=0) at mod_perl.c:1000 retval = value optimized out #8 0x005fe96b in modperl_response_handler_cgi (r=0x95efc90) at mod_perl.c:1100 dcfg = (modperl_config_dir_t *) 0x94dd4f0 h_stdin = (GV *) 0x9688f18 h_stdout = (GV *) 0x9689134 retval = -1 rc = value optimized out rcfg = (modperl_config_req_t *) 0x95f0a28 my_perl = (PerlInterpreter *) 0x8eac5f8 interp = (modperl_interp_t *) 0x931fe18 #9 0x080821e9 in ap_run_handler () No symbol table info available. #10 0x08082933 in ap_invoke_handler () No symbol table info available. #11 0x080e0153 in ap_process_request () No symbol table info available. #12 0x080dcb8f in ap_process_http_connection () No symbol table info available. #13 0x0808a873 in ap_run_process_connection () No symbol table info available. #14 0x0808ac86 in ap_process_connection () No symbol table info available. #15 0x08102a73 in child_main () No symbol table info available. #16 0x08102c5e in make_child () No symbol table info available. #17 0x08102eaf in perform_idle_server_maintenance () No symbol table info available. #18 0x081033d9 in ap_mpm_run () No symbol table info available. #19 0x0806b61c in main () No symbol table info available. Is there anything there that helps find where the problem is?
Re: tracking a coredump problem
eric.b...@barclayscapital.com wrote:
Carl,
I may have missed it, but did you say at what point you were seeing the
segfault? I assume you mean at startup, but can you confirm?
Not at startup, it happens after 'a while'. It's very hard to track,
and I am stumped at trying to get a core file to see where it's really
breaking.
My init script (CentOS 5.2):
start() {
echo -n $Starting $prog:
ulimit -S -c unlimited /dev/null 21
LANG=$HTTPD_LANG daemon $httpd $OPTIONS -f $conf
RETVAL=$?
echo
ulimit -c
[ $RETVAL = 0 ] touch ${lockfile}
return $RETVAL
}
I must be missing something, I get segfaults but no core dumps and I've
finally got to do something about it.
RE: tracking a coredump problem
Carl, I may have missed it, but did you say at what point you were seeing the segfault? I assume you mean at startup, but can you confirm? E -Original Message- From: Carl Brewer [mailto:c...@bl.echidna.id.au] Sent: Wednesday, January 28, 2009 7:43 AM To: Philippe M. Chiasson Cc: modperl@perl.apache.org Subject: Re: tracking a coredump problem Carl Brewer wrote: Philippe M. Chiasson wrote: Selinux enabled ? Good question, I don't think so, but will double-check. Nope, selinux is disabled on /etc/selinux/config ___ This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing. Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP. This email may relate to or be sent from other members of the Barclays Group. ___
Re: tracking a coredump problem
In a message dated 2009-2-12 22:39:48, eric.b...@barclayscapital.com writes: Nope, selinux is disabled on /etc/selinux/config Or you may take a look at this article: _http://blog.modsecurity.org/2009/01/building-qa-test-cases-from-waf-data.html _ (http://blog.modsecurity.org/2009/01/building-qa-test-cases-from-waf-data.html) J. **Nothing says I love you like flowers! Find a florist near you now. (http://yellowpages.aol.com/search?query=floristncid=emlcntusyelp0001)
Re: tracking a coredump problem
Philippe M. Chiasson wrote: Selinux enabled ? Good question, I don't think so, but will double-check.
Re: tracking a coredump problem
Carl Brewer wrote: Philippe M. Chiasson wrote: Selinux enabled ? Good question, I don't think so, but will double-check. Nope, selinux is disabled on /etc/selinux/config
Re: tracking a coredump problem
Carl Brewer wrote:
And in the (RHEL/CentOS format) init.d script
start() {
echo -n $Starting $prog:
ulimit -S -c unlimited /dev/null 21
LANG=$HTTPD_LANG daemon $httpd $OPTIONS -f $conf
RETVAL=$?
echo
ulimit -c
[ $RETVAL = 0 ] touch ${lockfile}
return $RETVAL
}
Why are there two ulimit lines in here? Does the second one undo what
the first one did, causing cores to not get generated?
When I needed to get cores generated I did exactly what it said here:
http://httpd.apache.org/dev/debugging.html#crashes
namely, that I ran ulimit in the shell, then in that same shell started
apache with apachectl. I was able to generate cores that way.
Adam
Re: tracking a coredump problem
Adam Prime wrote:
Carl Brewer wrote:
And in the (RHEL/CentOS format) init.d script
start() {
echo -n $Starting $prog:
ulimit -S -c unlimited /dev/null 21
LANG=$HTTPD_LANG daemon $httpd $OPTIONS -f $conf
RETVAL=$?
echo
ulimit -c
[ $RETVAL = 0 ] touch ${lockfile}
return $RETVAL
}
Why are there two ulimit lines in here? Does the second one undo what
the first one did, causing cores to not get generated?
The second one is a debugging thing, it just prints to STDOUT what the
ulimit has been set to.
When I needed to get cores generated I did exactly what it said here:
http://httpd.apache.org/dev/debugging.html#crashes
namely, that I ran ulimit in the shell, then in that same shell started
apache with apachectl. I was able to generate cores that way.
Sure, but I'm not sure why doing that in a startup script doesn't do the
same thing?
Adam
Re: tracking a coredump problem
Following up on this, I'm still stuck, but maybe someone here can help
with a debugging step?
I'm trying to get httpd to dump cores, as we're seeing this a lot :
[Sun Jan 25 18:14:17 2009] [notice] child pid 20822 exit signal
Segmentation fault (11)
and I figure a core might just help. So, in httpd-perl's config :
CoreDumpDirectory /var/cores
and /var/cores is 777
And in the (RHEL/CentOS format) init.d script
start() {
echo -n $Starting $prog:
ulimit -S -c unlimited /dev/null 21
LANG=$HTTPD_LANG daemon $httpd $OPTIONS -f $conf
RETVAL=$?
echo
ulimit -c
[ $RETVAL = 0 ] touch ${lockfile}
return $RETVAL
}
I think I got just about everything needed to get a coredump, but
nothing shows up in /var/cores, despite many segfaults in the logs.
Anything I've missed?
Thankyou
Carl
Re: tracking a coredump problem
On 27/1/09 15:55, Carl Brewer wrote:
Following up on this, I'm still stuck, but maybe someone here can help
with a debugging step?
I'm trying to get httpd to dump cores, as we're seeing this a lot :
[Sun Jan 25 18:14:17 2009] [notice] child pid 20822 exit signal
Segmentation fault (11)
and I figure a core might just help. So, in httpd-perl's config :
CoreDumpDirectory /var/cores
and /var/cores is 777
And in the (RHEL/CentOS format) init.d script
start() {
echo -n $Starting $prog:
ulimit -S -c unlimited /dev/null 21
LANG=$HTTPD_LANG daemon $httpd $OPTIONS -f $conf
RETVAL=$?
echo
ulimit -c
[ $RETVAL = 0 ] touch ${lockfile}
return $RETVAL
}
I think I got just about everything needed to get a coredump, but
nothing shows up in /var/cores, despite many segfaults in the logs.
Anything I've missed?
Selinux enabled ?
--
Philippe M. Chiasson GPG: F9BFE0C2480E7680 1AE53631CB32A107 88C3A5A5
http://gozer.ectoplasm.org/ m/gozer\@(apache|cpan|ectoplasm)\.org/
signature.asc
Description: OpenPGP digital signature
