#43105 [Asn]: PHP seems to fail to close open files.
ID: 43105 User updated by: ian at onlineloop dot com Reported By: ian at onlineloop dot com Status: Assigned Bug Type: Apache2 related Operating System: Solaris 10 PHP Version: 5.2.5RC2-dev Assigned To: ab5602 New Comment: Coming back to the bug report here now. In the meantime some private emails were exchanged, including a pfiles output from Solaris showing that PHP had over 210,000 open files after 24 hours running on our servers. Within 48 hours (we let it go this far onyl once), apache/php eats around 12Gb of RAM and has between 170 and 220 child processes with over 230,000 open files. Under 5.1.6 the usage is more around 1.5-2Gb ram, and 30-70 child processes, with rarely more than 100 open files (only when we are really under load do we get to more than about 800 open files at any one time). A small patch was sent to me to try, however this has changed nothing. I was also asked to compile with gcc if possible, however this is not feasible as too many other things would have to be recompiled. Besides, we specifically went away from gcc because everything we had that was compiled with gcc was seg faulting all the time, however with the Sun Studio compiler suite, everything is stable. I seriously doubt this is an apache bug, why were things working with previous versions of PHP, and not this one? Previous Comments: [2007-11-28 15:02:11] grknight at iwon dot com I am also experiencing this issue. We are running Apache 2.2.3 on Redhat EL 3 and recently tried to update to 5.2.5 from 5.2.3 to fix the security issues. The moment 5.2.5 was activated, connections failed to close in apache and resulted in hung processes. I also tried 5.2.4 with the same results. No configurations were changed nor PHP scripts. Something changed in the PHP processes that prevents the apxs2handler from exiting between 5.2.3 and 5.2.4. Configs available on request. [2007-11-26 14:08:48] marcus dot mueller at grintsch dot com I doubt this is an Apache issue since we're experiencing the same symptons as hwallenstone at gmx dot here on two debian linux boxes, one using the 64bit version of Apache's httpd 2.2.4, the other using the 32bit version of httpd 2.0.59. httpd processes seem to hang i.e. they don't close the connection (telling from /server-status) resulting in the issues mentioned above. I first noticed this behaviour when switching from PHP 5.2.3 to PHP 5.2.4, both self-compiled using the same configure options. PHP 5.2.3, unlike 5.2.4 and 5.2.5, does not expose this behaviour. I hope this info might narrow down your search path a little bit. [2007-11-25 14:21:58] hwallenstone at gmx dot de I think we have the same problem here. I have updated one server of a cluster of busy servers from a patched 5.2.1 to 5.2.5 . The number of apache processes is growing and as a consequence of this, the number of open files increases. We have about 50 processes running on average machines; on the 5.2.5-one the number constantly grows until it reaches my MaxClients Limit. Trying to stop apache, I get hundreds of entries like [Sun Nov 25 14:14:55 2007] [error] could not make child process 28546 exit, attempting to continue anyway This problem **definitely** has come with the upgrade from 5.2.1 to 5.2.5. Nothing else was changed. So it doesn't look like this is a old apache bug. [2007-11-20 09:40:33] [EMAIL PROTECTED] IIRC, this is actually an Apache bug. PHP is not the only module which suffers as 3rd party of it.. [2007-11-12 20:52:58] ian at onlineloop dot com Hi, I don't want to post the attahced file to the bug report as this may expose more information that I like about our web server. It is a pfiles output from the apache processes. I will send the file to you directly in an email. The exact situation is difficult to reproduce, and happened most frequently when the server was under load. We are typically handling about 100-120 requests per second, and under PHP 5.1.6 (we are back on it now), there was usually between 500 and 1000 open files at any one time between the apache processes. The symptom of the problem - system error messages, masses of open files, not being able to execute ssmtp and not being able to open files, also generally exists for just a couple of seconds, then goes away again. The total memory usage of apache climbs constantly, usually hitting 6-8Gb within 24 hours of an apache restart, and continuing to climb if apache is not restarted. After about 36 hours, the ram usage is over 10Gb, at which point I restart the server as we need it to be
#43105 [Asn]: PHP seems to fail to close open files.
ID: 43105 Updated by: [EMAIL PROTECTED] Reported By: ian at onlineloop dot com Status: Assigned Bug Type: Apache2 related Operating System: Solaris 10 PHP Version: 5.2.5RC2-dev Assigned To: ab5602 New Comment: IIRC, this is actually an Apache bug. PHP is not the only module which suffers as 3rd party of it.. Previous Comments: [2007-11-12 20:52:58] ian at onlineloop dot com Hi, I don't want to post the attahced file to the bug report as this may expose more information that I like about our web server. It is a pfiles output from the apache processes. I will send the file to you directly in an email. The exact situation is difficult to reproduce, and happened most frequently when the server was under load. We are typically handling about 100-120 requests per second, and under PHP 5.1.6 (we are back on it now), there was usually between 500 and 1000 open files at any one time between the apache processes. The symptom of the problem - system error messages, masses of open files, not being able to execute ssmtp and not being able to open files, also generally exists for just a couple of seconds, then goes away again. The total memory usage of apache climbs constantly, usually hitting 6-8Gb within 24 hours of an apache restart, and continuing to climb if apache is not restarted. After about 36 hours, the ram usage is over 10Gb, at which point I restart the server as we need it to be available. Since running 5.1.6 again (one week), the memory usage is constantly around 1.5-2Gb and there are no problems from 5.2.5rc-x. I fear reproducing this will be difficult for you. ssmtp is generally able to send mails, just occasionally it is hit by this problem. Apache also has problems opening other files under this condition, apparently mainly for writing. I have tried on the command line to send mails with ssmtp when the error messages start coming out, however that worked and the test mails came in. cating files into another file, and things like this also worked. The problem of not being able to open files also occurs when not trying to send mails, as such my impression the problem is more general than just being connected with mail function. I did put sendmail in briefly today to see what happened, the problem still occured. A restart in the middle of the day for the apache process is not an option here as the service needs to be available. I can only make changes to the system between 2 and 4am. I will do what I can to help pinpoint this problem, please however understand the restrictions I have as I do this on a productive system. [2007-11-11 19:18:31] [EMAIL PROTECTED] 1) What file(s) are being held open? (lsof/ptree may help). 2) Is 'ssmtp -t' successfully opening, delivering the email and closing it's own network connections under this condition? Every open network connection requires 2 file descriptors. 3) Does this happen without sending mail()? 4) Does this happen with other delivery programs such as 'sendmail -t'? Thanks, -Rob [2007-11-11 19:03:23] [EMAIL PROTECTED] So, how can we reproduce this? [2007-11-06 11:19:24] ian at onlineloop dot com Changing this whole report as I have now more information. After further examining the problem, it is clear that PHP is sometimes failing to close the files it opens, and is generally failing to cleanly exit, if it exits at all. It is running with Apache 2.0.61. As time passes, PHP has more and more files open, the apache child processes do not exit, the status of the child processes stays on W, writing, for long periods of time although the network status on the server itsel clearly shows that all connections to the remote address have closed. The memory usage also climbs constantly until the apache child process dies, if the child process dies at all. In the system, this is noticable by the number of open files, in the last 24 hours, this has changed from 4.2Gb and 30,711 open files to now 6.6Gb and 48,462 open files. This increase in memory consumption and the number of open files is continuous. Additionally, there are more and more apache child processes that simply do not exit. It is this behaviour that is clearly responsible for the problems of opening other files, and for the failure to be able to execute some system calls, such as sending mail and writing log files. The cause for this behaviour is unclear, however it is definately PHP 5.2.x related as under PHP 5.1.6, everything is ok. [2007-10-25 15:03:26] ian at onlineloop dot com Description:
#43105 [Asn]: PHP seems to fail to close open files.
ID: 43105 User updated by: ian at onlineloop dot com Reported By: ian at onlineloop dot com Status: Assigned Bug Type: Unknown/Other Function Operating System: Solaris 10 PHP Version: 5.2.5RC2-dev Assigned To: ab5602 New Comment: Hi, I don't want to post the attahced file to the bug report as this may expose more information that I like about our web server. It is a pfiles output from the apache processes. I will send the file to you directly in an email. The exact situation is difficult to reproduce, and happened most frequently when the server was under load. We are typically handling about 100-120 requests per second, and under PHP 5.1.6 (we are back on it now), there was usually between 500 and 1000 open files at any one time between the apache processes. The symptom of the problem - system error messages, masses of open files, not being able to execute ssmtp and not being able to open files, also generally exists for just a couple of seconds, then goes away again. The total memory usage of apache climbs constantly, usually hitting 6-8Gb within 24 hours of an apache restart, and continuing to climb if apache is not restarted. After about 36 hours, the ram usage is over 10Gb, at which point I restart the server as we need it to be available. Since running 5.1.6 again (one week), the memory usage is constantly around 1.5-2Gb and there are no problems from 5.2.5rc-x. I fear reproducing this will be difficult for you. ssmtp is generally able to send mails, just occasionally it is hit by this problem. Apache also has problems opening other files under this condition, apparently mainly for writing. I have tried on the command line to send mails with ssmtp when the error messages start coming out, however that worked and the test mails came in. cating files into another file, and things like this also worked. The problem of not being able to open files also occurs when not trying to send mails, as such my impression the problem is more general than just being connected with mail function. I did put sendmail in briefly today to see what happened, the problem still occured. A restart in the middle of the day for the apache process is not an option here as the service needs to be available. I can only make changes to the system between 2 and 4am. I will do what I can to help pinpoint this problem, please however understand the restrictions I have as I do this on a productive system. Previous Comments: [2007-11-11 19:18:31] [EMAIL PROTECTED] 1) What file(s) are being held open? (lsof/ptree may help). 2) Is 'ssmtp -t' successfully opening, delivering the email and closing it's own network connections under this condition? Every open network connection requires 2 file descriptors. 3) Does this happen without sending mail()? 4) Does this happen with other delivery programs such as 'sendmail -t'? Thanks, -Rob [2007-11-11 19:03:23] [EMAIL PROTECTED] So, how can we reproduce this? [2007-11-06 11:19:24] ian at onlineloop dot com Changing this whole report as I have now more information. After further examining the problem, it is clear that PHP is sometimes failing to close the files it opens, and is generally failing to cleanly exit, if it exits at all. It is running with Apache 2.0.61. As time passes, PHP has more and more files open, the apache child processes do not exit, the status of the child processes stays on W, writing, for long periods of time although the network status on the server itsel clearly shows that all connections to the remote address have closed. The memory usage also climbs constantly until the apache child process dies, if the child process dies at all. In the system, this is noticable by the number of open files, in the last 24 hours, this has changed from 4.2Gb and 30,711 open files to now 6.6Gb and 48,462 open files. This increase in memory consumption and the number of open files is continuous. Additionally, there are more and more apache child processes that simply do not exit. It is this behaviour that is clearly responsible for the problems of opening other files, and for the failure to be able to execute some system calls, such as sending mail and writing log files. The cause for this behaviour is unclear, however it is definately PHP 5.2.x related as under PHP 5.1.6, everything is ok. [2007-10-25 15:03:26] ian at onlineloop dot com Description: Working with PHP 5.2.5-RC2, build from php5.2-200710241430, problem also occurred in php5.2-200710171030. PHP for what ever reason is occasionally unable to execute the mail program that we have installed,
