ID: 14333
User updated by: [EMAIL PROTECTED]
Reported By: [EMAIL PROTECTED]
Old Status: Feedback
Status: Open
Bug Type: Apache related
Operating System: RH 6.2 SMP
PHP Version: 4.0.6
New Comment:
I just reconfigured and installed Apache 1.3.22 with PHP 4.1.0RC5 support. When I did
this, I also added --enable-debug to the PHP configure command-line. I added a
CFLAGS="-g" and --without-execstrip to the apache configure command-line.
Once I installed and restarted Apache, I waited about 3 or 4 minutes before one of the
Apache processes started using up 88% of the CPU. This process grew to consume 99%
within the next minute or so.
I then ran gdb and attached it to the PID. The following is the output, including a
backtrace:
[root@s1 apache_1.3.22]# gdb /usr/local/apache/bin/httpd 28070
GNU gdb 19991004
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
/root/atb/Apachetoolbox-1.5.45/apache_1.3.22/28070: No such file or directory.
Attaching to program: /usr/local/apache/bin/httpd, Pid 28070
Reading symbols from /lib/libpam.so.0...done.
Reading symbols from /lib/libdl.so.2...done.
Reading symbols from /usr/local/lib/libgd.so...done.
Reading symbols from /usr/local/lib/php/t1libs/lib/libt1.so.1...done.
Reading symbols from /usr/local/lib/libfreetype.so.6...done.
Reading symbols from /usr/local/lib/libz.so.1...done.
Reading symbols from /usr/local/lib/libjpeg.so.62...done.
Reading symbols from /lib/libcrypt.so.1...done.
Reading symbols from /lib/libresolv.so.2...done.
Reading symbols from /lib/libm.so.6...done.
Reading symbols from /lib/libnsl.so.1...done.
Reading symbols from /lib/libdb.so.3...done.
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
Reading symbols from /lib/libnss_files.so.2...done.
Reading symbols from /lib/libnss_dns.so.2...done.
Reading symbols from /usr/lib/gconv/ISO8859-1.so...done.
Reading symbols from /usr/local/apache/libexec/libjrun.so...done.
Reading symbols from /usr/local/apache/libexec/mod_caucho.so...done.
0x4020922e in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
3121 malloc.c: No such file or directory.
(gdb) bt
#0 0x4020922e in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
#1 0x40208f9a in __libc_free (mem=0x8ccf458) at malloc.c:3023
#2 0x815765a in shutdown_memory_manager (silent=0, clean_cache=1) at zend_alloc.c:485
#3 0x80c85a5 in php_module_shutdown () at main.c:1007
#4 0x80c855c in php_module_shutdown_wrapper (sapi_globals=0x8367ba0) at main.c:972
#5 0x80c6a15 in php_child_exit_handler (s=0x83aa8a8, p=0x8cf6578) at mod_php4.c:816
#6 0x81c4ac1 in ap_child_exit_modules (p=0x8cf6578, s=0x83aa8a8) at
http_config.c:1701
#7 0x81ca9c5 in clean_child_exit (code=0) at http_main.c:537
#8 0x81cc9e7 in just_die (sig=10) at http_main.c:3068
#9 0x81cca09 in usr1_handler (sig=10) at http_main.c:3078
#10 0x401cdc48 in __restore () at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127
#11 0x81cab96 in accept_mutex_on_sysvsem () at http_main.c:841
#12 0x81cdd36 in child_main (child_num_arg=17) at http_main.c:4301
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=17, now=1007468333) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415
I then quit gdb and killed the high-CPU process (kill -9 28070). There were around 10
idle httpd processes at that time. The httpd process with the highest PID (#28066)
then immediately started using up excessive CPU time. Doing a backtrace of it
resulted in the same kind of output:
0 0x40209233 in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
#1 0x40208f9a in __libc_free (mem=0x8ccf458) at malloc.c:3023
#2 0x815765a in shutdown_memory_manager (silent=0, clean_cache=1) at zend_alloc.c:485
#3 0x80c85a5 in php_module_shutdown () at main.c:1007
#4 0x80c855c in php_module_shutdown_wrapper (sapi_globals=0x8367ba0) at main.c:972
#5 0x80c6a15 in php_child_exit_handler (s=0x83aa8a8, p=0x8cf6578) at mod_php4.c:816
#6 0x81c4ac1 in ap_child_exit_modules (p=0x8cf6578, s=0x83aa8a8) at
http_config.c:1701
#7 0x81ca9c5 in clean_child_exit (code=0) at http_main.c:537
#8 0x81cc9e7 in just_die (sig=10) at http_main.c:3068
#9 0x81cca09 in usr1_handler (sig=10) at http_main.c:3078
#10 0x401cdc48 in __restore () at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127
#11 0x81cab96 in accept_mutex_on_sysvsem () at http_main.c:841
#12 0x81cdd36 in child_main (child_num_arg=16) at http_main.c:4301
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=16, now=1007468328) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415
Every time I kill the CPU consuming process, the highest PID apache process takes over
doing the same thing and consumes 95%+ CPU time immediately.
Looking at a normal httpd process (with 0.0% CPU), the output looks like this:
#0 0x4025c15e in __select () from /lib/libc.so.6
#1 0x81cab80 in accept_mutex_on_sysvsem () at http_main.c:837
#2 0x81ce35c in make_child (s=0x83aa8a8, slot=14, now=1007468327) at http_main.c:4723
#3 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#4 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#5 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415
Here is the output of a process that is active, but still not consuming hardly any
CPU:
#0 0x40255ab4 in __libc_read () from /lib/libc.so.6
#1 0x81ab484 in ssl_io_hook_read (fb=0x83b65e8,
buf=0x83b6628 "GET /lemonnier/forum/immagini/testatina_forum.jpg
HTTP/1.1\r\nAccept: */*\r\nReferer:
http://www.japhost.com/lemonnier/forum/ConclusioniSecondaria/LetMinis.html\r\nAccept-Language:
it\r\nAccept-Encoding: gzip"..., len=4096) at ssl_engine_io.c:339
#2 0x81e48ac in ap_hook_call_func (ap=0xbfffd864, he=0x83b30d8, hf=0x83b6570) at
ap_hook.c:649
#3 0x81e3fc1 in ap_hook_call (hook=0x8336fad "ap::buff::read") at ap_hook.c:382
#4 0x81c01a0 in ap_read (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:285
#5 0x81c1d9e in buff_read (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:329
#6 0x81c1d48 in saferead_guts (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:702
#7 0x81c084f in read_with_errors (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at
buff.c:753
#8 0x81c0ba7 in ap_bgets (buff=0xbfffd948 "", n=8192, fb=0x83b65e8) at buff.c:906
#9 0x81d09d8 in getline (s=0xbfffd948 "", n=8192, in=0x83b65e8, fold=0) at
http_protocol.c:839
#10 0x81d0cf1 in read_request_line (r=0x8d045d0) at http_protocol.c:962
#11 0x81d134e in ap_read_request (conn=0x83b9640) at http_protocol.c:1125
#12 0x81ce0c2 in child_main (child_num_arg=12) at http_main.c:4544
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=12, now=1007468293) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415
One more interesting thing is that most of the time, the high-CPU PID is not listed in
an Apache /server-status list of processes. If it is listed, it has a Mode of
Operation of "Waiting for connection".
Lastly, sometimes only one high-CPU processes appears and consumes 95%+ CPU. Other
times, multiple high-CPU processes appear. When this happens, they each consume less
CPU. For instance, right now there are 3 httpd processes, each consuming 50%+ CPU
time. All three process' backtraces looked the same as the first backtrace listed
above (the high-CPU processes).
Note that I have several additional modules included in Apache during these tests.
This is a production server, so I can't do any long-term tests that do not include
essential Apache mods. In addition to many of the "standard" modules, the following
modules were enabled in Apache:
mod_perl - static
WebDAV - static
mod_auth_ldap - static
mod_gzip - static
mod_layout - static
mod_headers - static
PHP - static
mod_php, v4.1.0RC5, --enable-debug
GD
mod_ssl - static
mod_jrun (JRun servlet engine) - as DSO
mod_caucho (Resin servlet engine) - as DSO
I've been able to duplicate this problem in both PHP 4.0.6 and PHP 4.1.0RC5 with only
the bare minimum modules included. I'm pretty confident that the problem would still
occur even if Apache was configured with only PHP. The backtraces do not seem to
indicate any of the modules being suspicious except for PHP and the core apache.
I hope this information is useful. Please let me know if there is anything else I can
provide.
Tauren
Previous Comments:
------------------------------------------------------------------------
[2001-12-04 02:22:02] [EMAIL PROTECTED]
Can you possibly try php-4.1.0RC5 from www.php.net/~zeev/ and make sure you do a clean
build. My guess is that your build is slightly broken. Please also try it as an static
apache module.
regards,
Derick
------------------------------------------------------------------------
[2001-12-04 00:59:59] [EMAIL PROTECTED]
It's interesting that you aren't experiencing the same problems. I wonder why I (and
several others) are having trouble. Here's the info you've requested.
Kernel/glibc version:
Red Hat Linux release 6.2 (Zoot)
Kernel 2.2.14-5.0smp on a 2-processor i686
Apache version:
Apache/1.3.22 (Unix) mod_ssl/2.8.5 OpenSSL/0.9.6
Please note that I'm not manually building Apache with PHP. I've built Apache
manually many times, but by the time I wanted to use PHP, I was using ApacheToolbox
(ATB) to build my Apache distributions. ATB is a script that eases the download and
configuration of Apache with many supported modules. It really simplifies the
building of Apache+ModSSL+PHP+OtherStuff. It can be downloaded at
www.apachetoolbox.com. Below is the PHP configure line that it generates and lets me
edit before PHP is configured.
PHP configure line:
CPPFLAGS=-I/usr/local/include
LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local \
--with-apache=/root/atb/Apachetoolbox-1.5.45/apache_1.3.22 \
--enable-exif \
--enable-track-vars \
--with-calendar=shared \
--enable-safe-mode \
--enable-magic-quotes \
--enable-trans-sid \
--enable-wddx \
--enable-ftp \
--with-mysql \
--with-ldap \
ATB then generates this Apache configuration line and lets me edit it before
configuring Apache:
export CFLAGS=""
export LIBS=""
export INCLUDES=""
./configure --prefix=/usr/local/apache \
--enable-suexec \
--suexec-caller=nobody \
--enable-module=so \
--enable-module=access \
--disable-module=auth_db \
--disable-module=digest \
--enable-module=imap \
--enable-module=mime \
--enable-module=setenvif \
--disable-module=usertrack \
--enable-module=auth \
--disable-module=cern_meta \
--disable-module=expires \
--enable-module=log_config \
--disable-module=proxy \
--disable-module=vhost_alias \
--disable-module=auth_anon \
--enable-module=cgi \
--disable-module=headers \
--disable-module=log_referer \
--enable-module=rewrite \
--enable-module=userdir \
--enable-module=asis \
--enable-module=autoindex \
--disable-module=example \
--disable-module=log_agent \
--enable-module=negotiation \
--enable-module=status \
--enable-module=actions \
--disable-module=auth_dbm \
--enable-module=dir \
--enable-module=include \
--disable-module=mime_magic \
--disable-module=unique_id \
--enable-module=alias \
--disable-module=auth_digest \
--enable-module=env \
--disable-module=info \
--disable-module=mmap_static \
--disable-module=speling \
--enable-module=ssl \
--activate-module=src/modules/php4/libphp4.a \
A couple other things... The MM library (mm-1.1.3) is being used with Apache. I've
been using this with my Apache builds for years and it hasn't caused any trouble
before. Also, Apache is built as a static binary with DSO support. PHP is built into
it, not as a DSO.
I'll give PHP 4.1.0RC5 a try and let you know what happens.
Also, yes I am using ./apachectl stop and start, not restart or graceful. Doesn't
hurt to be thorough with your questions...
Thanks for your help!
Tauren
------------------------------------------------------------------------
[2001-12-03 20:56:04] [EMAIL PROTECTED]
I'm also running PHP / Apache in a SMP machine but I never
have noticed anything like this. Some things we need to know about
your system:
- Kernel/glibc version
- Apache version
- PHP configure line
And you should also try with PHP 4.1.0RC5:
http://www.php.net/~zeev/php-4.1.0RC5.tar.gz
--Jani
p.s. This might sound stupid but did you do './apachectl stop ; ./apachectl start' or
did you use './apachectl restart' ?
------------------------------------------------------------------------
[2001-12-03 19:35:22] [EMAIL PROTECTED]
Actually, there is a sample PHP script installed on the server in several user
directories. But this is the ONLY file with a *.php* extension within our web root
directories. It is called "hello.php3" and it looks like this:
<html><head><title>PHP Test</title></head>
<body>
<?php echo "Hello World<P>"; ?>
</body></html>
I really don't think this script is being run. I don't think that any script is being
run to cause the problem.
Thanks,
Tauren
------------------------------------------------------------------------
[2001-12-03 19:21:39] [EMAIL PROTECTED]
I've been experiencing the exact same problem that is described in bug #14290 and
#8446 for quite some time now but did not know which Apache module was causing it. Up
until now, I've had a cron job that simply kills off (with a -9) any httpd processes
that are using 99% or more cpu time.
Today I've been trying to track down what exactly is causing the problem. I've
eliminated all extra Apache modules and did not experience the problem. When I added
PHP back in, the problem started immediately. Within one minute of starting Apache
back up, the high-CPU processes started appearing again.
The Apache "server-status" didn't indicate that ANY php script had been hit. The
processes just start going out of control after some time. In fact, there isn't even
a single *.php* file on the server. I really don't think this is happening because of
a PHP script being run.
I'm currently testing this out on a Red Hat 6.2 Linux (SMP) box with dual CPUs. From
the sound of things in the other bug reports (#14290 and #8446), the problem only
seems to be happening on SMP servers. I did not compile with any extra PHP modules
except for the core PHP 4.0.6.
I haven't really done a lot with PHP, so I'm not sure how to help debug this problem.
But I do want a stable Apache environment with PHP support for my hosting customers.
If there is anything I can do to help debug things, please let me know.
I've read the page on using gdb, but I'm think this is a different kind of situation.
Apache isn't crashing, but certain processes are going "out-of-control". Is there a
way to get a backtrace of a particular process while it is still running?
Until this problem can be resolved, I'm going to have to remove PHP from my servers.
I really don't want to have to do this, but the instabilities are becoming too much to
handle and very hard to explain to our customers.
Please let me know what I can do to help debug and solve this problem.
Thanks!
Tauren
------------------------------------------------------------------------
Edit this bug report at http://bugs.php.net/?id=14333&edit=1
--
PHP Development Mailing List <http://www.php.net/>
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]
