1. Problem Description:
Our production apache22/mod_perl2 web server segfaults regularly. I have tracked down this problem as much as I can, and could use some help. This is running on FreeBSD; the same problem occurs on FreeBSD 7.0, 7.2, and 8.0. apache22 is installed from ports, with WITH_DEBUG and WITH_EXECPTION_HOOK; mod_perl2 is installed from ports, my attempts to compile it with debug flags appear so far to be unsuccessful. (detailed version info below) For no reason I can fathom, I can not seem to get apache to dump core when it seg faults. I've set CoreDumpDirectory, I've set ulimit coredumpsize, nothing. No matter; I was able to install mod_whatkilledus and mod_backtrace. mod_whatkilledus shows that the segfaults happen on a wide variety of requests; most requests to our main web server are handled by mod_perl, either with a ResponseHandler or a FilterHandler. mod_backtrace shows that the seg fault is always happening in modperl_perl_global_request_save: [Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace backtrace for sig 11 (thread "pid" 6955) [Wed Dec 30 15:05:41 2009] pid 6955 mod_backtrace main() is at 8063880 0x8085a26 <ap_run_fatal_exception+70> at /usr/local/sbin/httpd 0x808752e <ap_mpm_set_exception_hook+238> at /usr/local/sbin/httpd 0x8087569 <ap_mpm_set_exception_hook+297> at /usr/local/sbin/httpd 0xbfbfffb4 0x287a83c5 <modperl_perl_global_request_save+213> at /usr/local/libexec/apache22/mod_perl.so 0x2879982d <modperl_response_handler_cgi+141> at /usr/local/libexec/apache22/mod_perl.so 0x807bc66 <ap_run_handler+70> at /usr/local/sbin/httpd ... So, any general pointers for what direction to go next would be appreciated. I would like to have mod_backtrace tell me what in my perl code is triggering the error, in which case I could work around it. Should I continue to pursue recompiling mod_perl2 and possibly perl itself with more debugging turned on? (http://perl.apache.org/docs/1.0/guide/help.html) 2. Used Components and their Configuration: *** mod_perl version 2.000004 *** using /usr/local/lib/perl5/site_perl/5.8.9/mach/Apache2/BuildConfig.pm *** Makefile.PL options: MP_APR_LIB => aprext MP_APXS => /usr/local/sbin/apxs MP_COMPAT_1X => 1 MP_GENERATE_XS => 1 MP_LIBNAME => mod_perl MP_USE_DSO => 1 *** The httpd binary was not found # I ran it by hand instead: bsdtest# /usr/local/sbin/httpd -V Server version: Apache/2.2.14 (FreeBSD) Server built: Dec 30 2009 09:46:16 Server's Module Magic Number: 20051115:23 Server loaded: APR 1.3.8, APR-Util 1.3.9 Compiled using: APR 1.3.8, APR-Util 1.3.9 Architecture: 32-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_USE_FLOCK_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/usr/local" -D SUEXEC_BIN="/usr/local/bin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="/var/log/httpd-error.log" -D AP_TYPES_CONFIG_FILE="etc/apache22/mime.types" -D SERVER_CONFIG_FILE="etc/apache22/httpd.conf" [Wed Dec 30 15:49:00 2009] [notice] Apache/2.2.14 (FreeBSD) mod_ssl/2.2.14 OpenSSL/0.9.8k mod_apreq2-20090110/2.7.1 mod_perl/2.0.4 Perl/v5.8.9 configured -- resuming normal operations *** (apr|apu)-config linking info -L/usr/local/lib -laprutil-1 -lexpat -liconv -L/usr/local/lib -L/usr/local/lib -lapr-1 -lcrypt -pthread *** /usr/local/bin/perl -V Summary of my perl5 (revision 5 version 8 subversion 9) configuration: Platform: osname=freebsd, osvers=8.0-release, archname=i386-freebsd-64int uname='freebsd freebsd.org 8.0-release freebsd 8.0-release #0: fri sep 18 03:01:27 pdt 2009 k...@freebsd.org:usrsrcsysmagickernelpath i386 ' config_args='-sde -Dprefix=/usr/local -Darchlib=/usr/local/lib/perl5/5.8.9/mach -Dprivlib=/usr/local/lib/perl5/5.8.9 -Dman3dir=/usr/local/lib/perl5/5.8.9/perl/man/man3 -Dman1dir=/usr/local/man/man1 -Dsitearch=/usr/local/lib/perl5/site_perl/5.8.9/mach -Dsitelib=/usr/local/lib/perl5/site_perl/5.8.9 -Dscriptdir=/usr/local/bin -Dsiteman3dir=/usr/local/lib/perl5/5.8.9/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Ui_malloc -Ui_iconv -Uinstallusrbinperl -Dcc=cc -Duseshrplib -Dinc_version_list=none -Dccflags=-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -Doptimize=-O2 -pipe -fno-strict-aliasing -Ud_dosuid -Ui_gdbm -Dusethreads=n -Dusemymalloc=y -Duse64bitint' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=define use64bitall=undef uselongdouble=undef usemymalloc=y, bincompat5005=undef Compiler: cc='cc', ccflags ='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include', optimize='-O2 -pipe -fno-strict-aliasing', cppflags='-DAPPLLIB_EXP="/usr/local/lib/perl5/5.8.9/BSDPAN" -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='4.2.1 20070719 [FreeBSD]', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -Wl,-E -L/usr/local/lib' libpth=/usr/lib /usr/local/lib libs=-lm -lcrypt -lutil perllibs=-lm -lcrypt -lutil libc=, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' -Wl,-R/usr/local/lib/perl5/5.8.9/mach/CORE' cccdlflags='-DPIC -fPIC', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MYMALLOC PERL_MALLOC_WRAP USE_64_BIT_INT USE_FAST_STDIO USE_LARGE_FILES USE_PERLIO Locally applied patches: defined-or Built under freebsd Compiled at Sep 18 2009 10:03:02 %ENV: PERL_LWP_USE_HTTP_10="1" @INC: /usr/local/lib/perl5/5.8.9/BSDPAN /usr/local/lib/perl5/site_perl/5.8.9/mach /usr/local/lib/perl5/site_perl/5.8.9 /usr/local/lib/perl5/5.8.9/mach /usr/local/lib/perl5/5.8.9 . *** Packages of interest status: Apache2 : - Apache2::Request : 2.12 CGI : 3.42 ExtUtils::MakeMaker: 6.48, undef LWP : 5.834 mod_perl : - mod_perl2 : 2.000004 3. This is the core dump trace: (if you get a core dump): [CORE TRACE COMES HERE] This report was generated by /usr/local/bin/mp2bug on Wed Dec 30 20:46:25 2009 GMT. - Alex Aminoff BaseSpace.net National Bureau of Economic Research (nber.org)
