Hi,
I work on a high-traffic site that uses apache/mod_perl, and we're
seeing some occasional segmentation faults and bus errors in our
apache error logs. These errors sometimes result in the entire apache
process group going down, though it seems to me that the problems
originate within one of apache's child processes (maybe shared memory
is getting corrupted somehow?).
I've searched through the archive of this list for similar situations,
and I found a lot of questions about seg faults, but none quite
matching our problem.
We installed some signal handlers in our perl code that trap SIGSEGV
and SIGBUS and then dump a perl stack trace to a log file (see below).
Using this stack information, we can track the point of failure to a
call to perl's "fork()" inside the IPC::Open3 standard module. Since
it seems very unlikely that fork() is broken, we're speculating that
there's some funny business going on prior to the fork that's putting
the process into an unstable state which prevents it from forking
successfully.
Due to a lot of sloppy, pre-existing Perl code, we're using PerlRun
(not Registry) with "PerlRunOnce On" (children die after servicing one
hit).
Does anyone have any suggestions about what might be going on here?
Thanks!
Justin Caballero
The following are: a backtrace from a core dump, the stack trace from
the perl signal handler, and the version information for our
environment.
---------
apache 1.3.12
mod_perl 1.24
---------
(gdb) where
#0 0xe765c in Perl_sv_free ()
#1 0xd89f4 in Perl_hv_free_ent ()
#2 0xd8bd8 in Perl_hv_clear ()
#3 0xd8b3c in Perl_hv_clear ()
#4 0x10e760 in Perl_pp_fork ()
#5 0x11b1d0 in Perl_runops_standard ()
#6 0xa49e8 in perl_call_sv ()
#7 0xa4490 in perl_call_method ()
#8 0x2aea8 in perl_call_handler ()
#9 0x2a6e0 in perl_run_stacked_handlers ()
#10 0x28da0 in perl_handler ()
#11 0x6e0f8 in ap_invoke_handler ()
#12 0x8a8e8 in ap_some_auth_required ()
#13 0x8a96c in ap_process_request ()
#14 0x7e3e4 in ap_child_terminate ()
#15 0x7e770 in ap_child_terminate ()
#16 0x7ece8 in ap_child_terminate ()
#17 0x7f54c in ap_child_terminate ()
#18 0x7fe80 in main ()
---------
SIGSEGV caught at:
IPC::Open3, /opt/perl-5.005_03/lib/5.00503/IPC/Open3.pm, 102,
main::cgi_stack_dump
IPC::Open3, /opt/perl-5.005_03/lib/5.00503/IPC/Open3.pm, 150,
IPC::Open3::xfork
IPC::Open2, /opt/perl-5.005_03/lib/5.00503/IPC/Open2.pm, 91,
IPC::Open3::_open3
Cyxsub, /prod/APP/vobs/ssp/cgi-bin/Cyxsub.pm, 69, IPC::Open2::open2
Cyxsub, /prod/APP/vobs/ssp/cgi-bin/Cyxsub.pm, 152, Cyxsub::sd_connect
main,
/prod/ssp_2.8_mp_prod_sv.001212/vobs/ssp_perl/cgi-bin/hy_inquiry_zc.pl,
43, Cyxsub::xs_ods_main
main,
/prod/ssp_2.8_mp_prod_sv.001212/vobs/ssp_perl/cgi-bin/cp_pers-io_zc.pl,
39, main::obtain_data
main, /prod/APP/vobs/ssp/cgi-bin/cp_pers_ub.pl, 285,
main::cp_pers_io_zc_get_data_from_host
main, /prod/APP/vobs/ssp/cgi-bin/cp_pers_ub.pl, 208,
main::cp_pers_ub_online_update
main, /prod/APP/vobs/ssp/cgi-bin/cp_pers_ub.pl, 70,
main::cp_pers_ub_update_ok
main, /prod/APP/vobs/ssp/cgi-bin/cp_pers_ub.pl, 39, main::cp_pers_ub_main
Apache::PerlRun,
/opt/perl-5.005_03/lib/site_perl/5.005/sun4-solaris/Apache/PerlRun.pm,
122, (eval)
Apache::PerlRun,
/opt/perl-5.005_03/lib/site_perl/5.005/sun4-solaris/Apache/PerlRun.pm,
296, Apache::PerlRun::compile
Apache::Constants,
/prod/ssp_2.8_mp_prod_sv.001212/vobs/ssp_perl/cgi-bin/opa_common_zc.pl, 0,
Apache::PerlRun::handler
Apache::Constants,
/prod/ssp_2.8_mp_prod_sv.001212/vobs/ssp_perl/cgi-bin/opa_common_zc.pl, 0,
(eval)
---------
> perl -V
Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
Platform:
osname=solaris, osvers=2.6, archname=sun4-solaris
uname='sunos atlas 5.6 generic_105181-19 sun4u sparc sunw,ultra-250 '
hint=recommended, useposix=true, d_sigaction=define
usethreads=undef useperlio=undef d_sfio=undef
Compiler:
cc='gcc -B/usr/ccs/bin/', optimize='-O', gccversion=2.95.2 19991024
(release)
cppflags='-I/usr/local/include'
ccflags ='-I/usr/local/include'
stdchar='unsigned char', d_stdstdio=define, usevfork=false
intsize=4, longsize=4, ptrsize=4, doublesize=8
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
alignbytes=8, usemymalloc=y, prototype=define
Linker and Libraries:
ld='gcc -B/usr/ccs/bin/', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
libs=-lsocket -lnsl -ldl -lm -lc -lcrypt
libc=, so=so, useshrplib=false, libperl=libperl.a
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'
Characteristics of this binary (from libperl):
Built under solaris
Compiled at Jul 11 2000 15:12:53
@INC:
/opt/perl5.005_03-gcc/lib/5.00503/sun4-solaris
/opt/perl5.005_03-gcc/lib/5.00503
/opt/perl5.005_03-gcc/lib/site_perl/5.005/sun4-solaris
/opt/perl5.005_03-gcc/lib/site_perl/5.005
.
