ID: 43105
Comment by: hwallenstone at gmx dot de
Reported By: ian at onlineloop dot com
Status: Assigned
Bug Type: Apache2 related
Operating System: Solaris 10
PHP Version: 5.2.5RC2-dev
Assigned To: ab5602
New Comment:
I think we have the same problem here. I have updated one server of a
cluster of busy servers from a patched 5.2.1 to 5.2.5 . The number of
apache processes is growing and as a consequence of this, the number of
open files increases. We have about 50 processes running on average
machines; on the 5.2.5-one the number constantly grows until it reaches
my MaxClients Limit. Trying to stop apache, I get hundreds of entries
like
[Sun Nov 25 14:14:55 2007] [error] could not make child process 28546
exit, attempting to continue anyway
This problem **definitely** has come with the upgrade from 5.2.1 to
5.2.5. Nothing else was changed. So it doesn't look like this is a old
apache bug.
Previous Comments:
------------------------------------------------------------------------
[2007-11-20 09:40:33] [EMAIL PROTECTED]
IIRC, this is actually an Apache bug. PHP is not the only module which
suffers as 3rd party of it..
------------------------------------------------------------------------
[2007-11-12 20:52:58] ian at onlineloop dot com
Hi,
I don't want to post the attahced file to the bug report as this may
expose more information that I like about our web server. It is a
pfiles output from the apache processes. I will send the file to you
directly in an email.
The exact situation is difficult to reproduce, and happened most
frequently when the server was under load. We are typically handling
about 100-120 requests per second, and under PHP 5.1.6 (we are back
on it now), there was usually between 500 and 1000 open files at any
one time between the apache processes.
The symptom of the problem - system error messages, masses of open
files, not being able to execute ssmtp and not being able to open
files, also generally exists for just a couple of seconds, then goes
away again. The total memory usage of apache climbs constantly,
usually hitting 6-8Gb within 24 hours of an apache restart, and
continuing to climb if apache is not restarted. After about 36
hours, the ram usage is over 10Gb, at which point I restart the
server as we need it to be available. Since running 5.1.6 again (one
week), the memory usage is constantly around 1.5-2Gb and there are no
problems from 5.2.5rc-x. I fear reproducing this will be difficult
for you.
ssmtp is generally able to send mails, just occasionally it is hit by
this problem. Apache also has problems opening other files under
this condition, apparently mainly for writing. I have tried on the
command line to send mails with ssmtp when the error messages start
coming out, however that worked and the test mails came in. "cat"ing
files into another file, and things like this also worked.
The problem of not being able to open files also occurs when not
trying to send mails, as such my impression the problem is more
general than just being connected with mail function. I did put
sendmail in briefly today to see what happened, the problem still
occured. A restart in the middle of the day for the apache process
is not an option here as the service needs to be available. I can
only make changes to the system between 2 and 4am.
I will do what I can to help pinpoint this problem, please however
understand the restrictions I have as I do this on a productive
system.
------------------------------------------------------------------------
[2007-11-11 19:18:31] [EMAIL PROTECTED]
1) What file(s) are being held open? (lsof/ptree may help).
2) Is 'ssmtp -t' successfully opening, delivering the email and closing
it's own network connections under this condition? Every open network
connection requires 2 file descriptors.
3) Does this happen without sending mail()?
4) Does this happen with other delivery programs such as 'sendmail -t'?
Thanks,
-Rob
------------------------------------------------------------------------
[2007-11-11 19:03:23] [EMAIL PROTECTED]
So, how can we reproduce this?
------------------------------------------------------------------------
[2007-11-06 11:19:24] ian at onlineloop dot com
Changing this whole report as I have now more information.
After further examining the problem, it is clear that PHP is sometimes
failing to close the files it opens, and is generally failing to cleanly
exit, if it exits at all. It is running with Apache 2.0.61. As time
passes, PHP has more and more files open, the apache child processes do
not exit, the status of the child processes stays on "W", writing, for
long periods of time although the network status on the server itsel
clearly shows that all connections to the remote address have closed.
The memory usage also climbs constantly until the apache child process
dies, if the child process dies at all.
In the system, this is noticable by the number of open files, in the
last 24 hours, this has changed from 4.2Gb and 30,711 open files to now
6.6Gb and 48,462 open files. This increase in memory consumption and
the number of open files is continuous. Additionally, there are more
and more apache child processes that simply do not exit. It is this
behaviour that is clearly responsible for the problems of opening other
files, and for the failure to be able to execute some system calls, such
as sending mail and writing log files.
The cause for this behaviour is unclear, however it is definately PHP
5.2.x related as under PHP 5.1.6, everything is ok.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
http://bugs.php.net/43105
--
Edit this bug report at http://bugs.php.net/?id=43105&edit=1
