Hello list,
I am experiencing memory allocation problem reported by glibc when
running a web application under mod_perl2:
1. Problem Description:
When hitting our web application with more than one concurrent request
(using ApacheBench), very soon errors like
*** glibc detected *** double free or corruption (fasttop): 0x08d7a318 ***
*** glibc detected *** corrupted double-linked list: 0x40544a8c ***
*** glibc detected *** double free or corruption (fasttop): 0x4585d630 ***
appear in the error log. If I disable those glibc warnings with
MALLOC_CHECK_=0 the server produces Segmentation Faults :-(.
I systematically tried to play around with the PerlInterp* and
PerlOptions +Clone/+Parent directives because I suspected the sharing
of interpreters between multiple virtual hosts might cause problems,
but this didn't make things any better.
As far as I understand the attached GDB stack trace, trouble starts
while cloning new Perl intrepreters.
What I can't tell is if it is our web-application (Catalyst-based) my
server setup or an actual bug in mp2... Any Comments?
Best Regards,
Andreas
2. Used Components and their Configuration:
*** mod_perl version 2.000002
*** using /usr/lib/perl5/Apache2/BuildConfig.pm
*** Makefile.PL options:
MP_APR_LIB => aprext
MP_APXS => /usr/bin/apxs2
MP_CCOPTS => -g -Wall
MP_COMPAT_1X => 1
MP_GENERATE_XS => 1
MP_INCLUDE_DIR => /usr/include/apache2 /usr/include/apr-0
MP_LIBNAME => mod_perl
MP_TRACE => 0
MP_USE_DSO => 1
MP_USE_GTOP => 1
MP_USE_STATIC => 0
*** The httpd binary was not found
*** (apr|apu)-config linking info
(apr|apu)-config scripts were not found
*** /usr/bin/perl -V
Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
Platform:
osname=linux, osvers=2.6.15.4, archname=i486-linux-gnu-thread-multi
uname='linux ninsei 2.6.15.4 #1 smp preempt mon feb 20 09:48:53
pst 2006 i686 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
-Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr
-Dprivlib=/usr/share/perl/5.8 -Darchlib=/usr/lib/perl/5.8
-Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
-Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
-Dsitelib=/usr/local/share/perl/5.8.8
-Dsitearch=/usr/local/lib/perl/5.8.8 -Dman1dir=/usr/share/man/man1
-Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
-Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
-Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm
-Duseshrplib -Dlibperl=libperl.so.5.8.8 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBIAN
-fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='4.0.3 (Debian 4.0.3-1)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.3.6.so, so=so, useshrplib=true,
libperl=libperl.so.5.8.8
gnulibc_version='2.3.6'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'
Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_MALLOC_WRAP THREADS_HAVE_PIDS USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
Built under linux
Compiled at Apr 4 2006 22:23:52
%ENV:
PERL_LWP_USE_HTTP_10="1"
@INC:
/etc/perl
/usr/local/lib/perl/5.8.8
/usr/local/share/perl/5.8.8
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8
/usr/share/perl/5.8
/usr/local/lib/site_perl
/usr/local/lib/perl/5.8.4
/usr/local/share/perl/5.8.4
.
*** Packages of interest status:
Apache2 : -
Apache2::Request : -
CGI : 3.15
ExtUtils::MakeMaker: 6.30
LWP : 5.805
mod_perl : -
mod_perl2 : 2.000002
*** here is some additional information about installed packages on
this Debian box:
Debian 3.1 Sarge Kernel 2.4.27-2-686-smp
weba03:~# apt-show-versions | grep apache2
libapache2-mod-perl2-dev/testing uptodate 2.0.2-2
libapache2-mod-perl2/testing uptodate 2.0.2-2
apache2/testing uptodate 2.0.55-4
apache2-common/testing uptodate 2.0.55-4
apache2-utils/testing uptodate 2.0.55-4
apache2-mpm-worker/testing uptodate 2.0.55-4
weba03:~# apt-show-versions | grep postgre
postgresql-client-8.1/testing uptodate 8.1.4-4
postgresql-client-common/testing uptodate 57
weba03:~# apt-show-versions | grep pq
libpq4/testing uptodate 8.1.4-4
weba03:~# apt-show-versions | grep dbd
libdbd-pg-perl/testing uptodate 1.49-1
weba03:~# apt-show-versions | grep dbi
libclass-dbi-loader-perl/testing uptodate 0.33-1
libclass-dbi-perl/testing uptodate 3.0.14-1
libclass-dbi-fromcgi-perl/testing uptodate 1.00-1
libclass-dbi-plugin-type-perl/testing uptodate 0.02-4
libapache-dbi-perl/stable uptodate 0.94-2
libdbix-contextualfetch-perl/testing uptodate 1.03-1
libclass-dbi-fromform-perl/testing uptodate 0.04-1
dbishell/stable uptodate 0.8.9-7
libdbi-perl/testing uptodate 1.51-2
libdbix-class-schema-loader-perl/testing uptodate 0.02007-1
libclass-dbi-plugin-retrieveall-perl/testing uptodate 1.04-1
libima-dbi-perl/testing uptodate 0.34-1
libdbix-class-perl/testing uptodate 0.06.003-1
libclass-dbi-asform-perl/testing uptodate 2.42-1
libcatalyst-model-cdbi-perl/testing uptodate 0.11-1
3. This is the core dump trace: (if you get a core dump):
weba03:~# MALLOC_CHECK_=2 /usr/sbin/apache2 -X
weba03:~# gdb /usr/sbin/apache2 <pid>
GNU gdb 6.3-debian
Copyright 2004 Free Software Foundation, Inc.
...
Program received signal SIGABRT, Aborted.
[Switching to Thread 229391 (LWP 808)]
0x4044e6f1 in kill () from /lib/libc.so.6
(gdb) where
#0 0x4044e6f1 in kill () from /lib/libc.so.6
#1 0x403d7030 in pthread_kill () from /lib/libpthread.so.0
#2 0x403d73fb in raise () from /lib/libpthread.so.0
#3 0x4044e464 in raise () from /lib/libc.so.6
#4 0x4044f9d8 in abort () from /lib/libc.so.6
#5 0x40489fb9 in malloc_usable_size () from /lib/libc.so.6
#6 0x4048a935 in free () from /lib/libc.so.6
#7 0x406d85da in Perl_newCONSTSUB () from /usr/lib/libperl.so.5.8
#8 0x406e89c7 in Perl_do_dump_pad () from /usr/lib/libperl.so.5.8
#9 0x4072db4c in Perl_pp_anoncode () from /usr/lib/libperl.so.5.8
#10 0x4070c989 in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#11 0x406ab2cd in Perl_get_cv () from /usr/lib/libperl.so.5.8
#12 0x406af065 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#13 0x406af57a in Perl_call_list () from /usr/lib/libperl.so.5.8
#14 0x406e35a8 in Perl_newATTRSUB () from /usr/lib/libperl.so.5.8
#15 0x406e1f90 in Perl_utilize () from /usr/lib/libperl.so.5.8
#16 0x406d44c2 in Perl_yyparse () from /usr/lib/libperl.so.5.8
#17 0x40748740 in Perl_find_runcv () from /usr/lib/libperl.so.5.8
#18 0x4074b0cc in Perl_pp_require () from /usr/lib/libperl.so.5.8
#19 0x40673ade in modperl_perl_global_request_restore ()
from /usr/lib/apache2/modules/mod_perl.so
#20 0x4070c989 in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#21 0x406ab2cd in Perl_get_cv () from /usr/lib/libperl.so.5.8
---Type <return> to continue, or q <return> to quit---
#22 0x406af065 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#23 0x406af57a in Perl_call_list () from /usr/lib/libperl.so.5.8
#24 0x406e35a8 in Perl_newATTRSUB () from /usr/lib/libperl.so.5.8
#25 0x406e1f90 in Perl_utilize () from /usr/lib/libperl.so.5.8
#26 0x406d44c2 in Perl_yyparse () from /usr/lib/libperl.so.5.8
#27 0x40748740 in Perl_find_runcv () from /usr/lib/libperl.so.5.8
#28 0x4074b0cc in Perl_pp_require () from /usr/lib/libperl.so.5.8
#29 0x40673ade in modperl_perl_global_request_restore ()
from /usr/lib/apache2/modules/mod_perl.so
#30 0x4070c989 in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#31 0x406ab2cd in Perl_get_cv () from /usr/lib/libperl.so.5.8
#32 0x406af065 in Perl_call_sv () from /usr/lib/libperl.so.5.8
#33 0x406af57a in Perl_call_list () from /usr/lib/libperl.so.5.8
#34 0x406e35a8 in Perl_newATTRSUB () from /usr/lib/libperl.so.5.8
#35 0x406e1f90 in Perl_utilize () from /usr/lib/libperl.so.5.8
#36 0x406d44c2 in Perl_yyparse () from /usr/lib/libperl.so.5.8
#37 0x40748740 in Perl_find_runcv () from /usr/lib/libperl.so.5.8
#38 0x4074b0cc in Perl_pp_require () from /usr/lib/libperl.so.5.8
#39 0x40673ade in modperl_perl_global_request_restore ()
from /usr/lib/apache2/modules/mod_perl.so
#40 0x4070c989 in Perl_runops_standard () from /usr/lib/libperl.so.5.8
#41 0x406ab2cd in Perl_get_cv () from /usr/lib/libperl.so.5.8
#42 0x406ae65b in Perl_eval_sv () from /usr/lib/libperl.so.5.8
---Type <return> to continue, or q <return> to quit---
#43 0x40669de8 in modperl_require_module ()
from /usr/lib/apache2/modules/mod_perl.so
#44 0x40670800 in modperl_mgv_resolve ()
from /usr/lib/apache2/modules/mod_perl.so
#45 0x40668f06 in modperl_handler_resolve ()
from /usr/lib/apache2/modules/mod_perl.so
#46 0x406670ff in modperl_callback () from
/usr/lib/apache2/modules/mod_perl.so
#47 0x40667b3b in modperl_callback_run_handlers ()
from /usr/lib/apache2/modules/mod_perl.so
#48 0x40667e13 in modperl_callback_per_dir ()
from /usr/lib/apache2/modules/mod_perl.so
#49 0x40662000 in modperl_response_finish ()
from /usr/lib/apache2/modules/mod_perl.so
#50 0x4066210b in modperl_response_handler ()
from /usr/lib/apache2/modules/mod_perl.so
#51 0x0807af75 in ap_run_handler ()
#52 0x0807b3b1 in ap_invoke_handler ()
#53 0x0806a638 in ap_process_request ()
#54 0x080652f8 in _start ()
#55 0x085a0f70 in ?? ()
#56 0x00000004 in ?? ()
#57 0x085a0f70 in ?? ()
#58 0x4067c7de in modperl_process_connection_handler ()
---Type <return> to continue, or q <return> to quit---
from /usr/lib/apache2/modules/mod_perl.so
#59 0x080865b5 in ap_run_process_connection ()
#60 0x08077b3e in ap_graceful_stop_signalled ()
#61 0x40346216 in apr_threadattr_stacksize_set () from
/usr/lib/libapr-0.so.0
#62 0x403d3c37 in pthread_start_thread () from /lib/libpthread.so.0
#63 0x404e52ba in clone () from /lib/libc.so.6
(gdb)
This report was generated by /usr/bin/mp2bug on Fri Aug 4 13:27:00
2006 GMT.
