Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. State-Changed-From-To: feedback-analyzed State-Changed-By: dgaudet State-Changed-When: Sat May 1 10:43:13 PDT 1999 State-Changed-Why: This sounds like the same kernel bug as reported in PR#3312: http://bugs.apache.org/index/full/3312 no workaround known at the moment Dean
Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. State-Changed-From-To: analyzed-feedback State-Changed-By: dgaudet State-Changed-When: Sat May 1 10:49:35 PDT 1999 State-Changed-Why: Hey are you still having this problem? You indicated that it happens when some mailing list cron job runs. Do you use sendmail or qmail? When the problem happens, could you use strace -p pid_of_parent to find out what the parent is doing and mail me the output? Thanks Dean
Re: os-linux/3343: Server dies after 1-20 hours of usage. (fwd)
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Dean Gaudet [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. (fwd) Date: Sat, 1 May 1999 10:41:56 -0700 (PDT) -- Forwarded message -- Date: Wed, 17 Mar 1999 12:13:59 -0600 From: Darrin Martin [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. As per your request, the following is the netstat -nt command issued after the server hung... [EMAIL PROTECTED] /root]# netstat -nt Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 368 0 206.65.73.212:80169.207.68.4:6337 CLOSE_WAIT tcp 368 0 206.65.73.212:80169.207.68.4:6326 CLOSE_WAIT tcp 391 0 206.65.73.205:80142.103.168.111:3458 CLOSE_WAIT tcp 416 0 206.65.73.205:80216.199.5.169:46679 ESTABLISHED tcp 343 0 206.65.73.205:80205.188.154.138:48871 CLOSE_WAIT tcp 311 0 206.65.73.205:80152.2.179.50:2072 CLOSE tcp 391 0 206.65.73.205:80142.103.168.111:3459 CLOSE_WAIT tcp 311 0 206.65.73.205:80152.2.179.50:2073 CLOSE tcp 289 0 206.65.73.205:8012.4.54.254:2060 CLOSE tcp 322 0 206.65.73.205:80139.78.213.97:3066 CLOSE_WAIT tcp 311 0 206.65.73.205:80152.2.179.50:2074 CLOSE tcp 416 0 206.65.73.205:80216.199.5.148:46680 ESTABLISHED tcp 1715 0 206.65.73.205:80142.103.168.111:3460 CLOSE_WAIT tcp 140 0 206.65.73.205:80209.67.244.80:2155 ESTABLISHED tcp 289 0 206.65.73.205:8012.4.54.254:2061 CLOSE tcp 270 0 206.65.73.205:80139.78.213.97:3094 CLOSE_WAIT tcp 289 0 206.65.73.205:8012.4.54.254:2062 CLOSE tcp 320 0 206.65.73.205:80128.171.242.1:23279 CLOSE_WAIT tcp 343 0 206.65.73.205:80205.188.154.138:60184 CLOSE_WAIT tcp 391 0 206.65.73.205:80142.103.168.111:3461 CLOSE_WAIT tcp 264 0 206.65.73.212:80206.101.224.99:34452 CLOSE_WAIT tcp 348 0 206.65.73.205:8012.4.54.254:2073 ESTABLISHED tcp0 0 206.65.73.205:80142.103.168.111:3462 ESTABLISHED tcp 385 0 206.65.73.205:80142.103.168.111:3463 CLOSE_WAIT tcp 264 0 206.65.73.212:80206.101.224.99:34554 CLOSE_WAIT tcp0124 206.65.73.205:23206.65.72.12:3243 ESTABLISHED tcp 241 0 206.65.73.205:80142.103.168.111:3471 ESTABLISHED tcp 342 0 206.65.73.205:80205.188.154.138:6924 ESTABLISHED Also, please be aware of the fact that the problem seemed to have changed when we upgraded to the Redhat 5.2rpm version of Apache.. now, instead of crashing every 20-24 hours, it is not expiring all requests, and eventually will lock up all available processes... I'll try and include a copy or you can view it yourself at http://www.v6fbody.com/server-status Login:apache Password:group Thank you. Darrin Martin ---snip- Comment-Added-By: dgaudet Comment-Added-When: Tue Mar 16 08:34:26 PST 1999 Comment-Added: When the problem happens, do a netstat -nt -- that should show to where the ESTABLISHED port 80 connections are ... I suspect something is just holding connections open. If that doesn't help, use strace on a few of the children to see what they're doing: strace -p pid_of_child Dean
Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: dgaudet Comment-Added-When: Tue Mar 16 08:34:26 PST 1999 Comment-Added: When the problem happens, do a netstat -nt -- that should show to where the ESTABLISHED port 80 connections are ... I suspect something is just holding connections open. If that doesn't help, use strace on a few of the children to see what they're doing: strace -p pid_of_child Dean
Re: os-linux/3343: Server dies after 1-20 hours of usage.
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark Herman II [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Sun, 21 Feb 1999 18:40:47 -0600 [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Thu Feb 11 12:51:43 PST 1999 Comment-Added: [This is a standard response.] This Apache problem report has not been updated recently. Please reply to this message if you have any additional information about this issue, or if you have answers to any questions that have been posed to you. If there are no outstanding questions, please consider this a request to try to reproduce the problem with the latest software release, if one has been made since last contact. If we don't hear from you, this report will be closed. If you have information to add, BE SURE to reply to this message and include the [EMAIL PROTECTED] address so it will be attached to the problem report! Hi, The server still dies, but we did find something that may be contributing to it. We just are not sure why. He runs a mailbag script that mails the traffic of his bulletin board to its subscribers every hour. If he disables this script, the web server doesn't die. We haven't found anything in this script that we believe would kill the web server, but if you would like to see it, I can forward it to you. Mark
Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Thu Feb 11 12:51:43 PST 1999 Comment-Added: [This is a standard response.] This Apache problem report has not been updated recently. Please reply to this message if you have any additional information about this issue, or if you have answers to any questions that have been posed to you. If there are no outstanding questions, please consider this a request to try to reproduce the problem with the latest software release, if one has been made since last contact. If we don't hear from you, this report will be closed. If you have information to add, BE SURE to reply to this message and include the [EMAIL PROTECTED] address so it will be attached to the problem report!
Re: os-linux/3343: Server dies after 1-20 hours of usage.
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark [EMAIL PROTECTED] To: Marc Slemko [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Mon, 09 Nov 1998 21:34:07 + Marc Slemko wrote: On 6 Nov 1998, Mark wrote: The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Fri, 06 Nov 1998 21:36:06 + [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Fri Nov 6 10:26:46 PST 1998 Comment-Added: Are you using NFS? Any third party modules? No. NFS is disabled, and we did not compile any 3rd party modules into the server. The only extra module that I enabled was the status module. The server ran fine for about 6 months before this started. Is there anything in the OS itself that may have changed to cause this problem? Did you upgrade your kernel version or any libraries, especially libc around when it started happening? No, we did not make any such changes to the machine since the end of Septemper. This problem started well afterwards. It is a friend's machine, and he did tell me one thing that is interesting about this problem: it seems to be happening consistently between 6:30am and 8:30am CST. We checked for cron jobs running at this time, and we didn't find any. Are there any new attacks against the server that you know of that could cause this behavior?
Re: os-linux/3343: Server dies after 1-20 hours of usage.
On 6 Nov 1998, Mark wrote: The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Fri, 06 Nov 1998 21:36:06 + [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Fri Nov 6 10:26:46 PST 1998 Comment-Added: Are you using NFS? Any third party modules? No. NFS is disabled, and we did not compile any 3rd party modules into the server. The only extra module that I enabled was the status module. The server ran fine for about 6 months before this started. Is there anything in the OS itself that may have changed to cause this problem? Did you upgrade your kernel version or any libraries, especially libc around when it started happening?
Re: os-linux/3343: Server dies after 1-20 hours of usage.
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Marc Slemko [EMAIL PROTECTED] To: Mark [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Sun, 8 Nov 1998 00:06:06 -0800 (PST) On 6 Nov 1998, Mark wrote: The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Fri, 06 Nov 1998 21:36:06 + [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Fri Nov 6 10:26:46 PST 1998 Comment-Added: Are you using NFS? Any third party modules? No. NFS is disabled, and we did not compile any 3rd party modules into the server. The only extra module that I enabled was the status module. The server ran fine for about 6 months before this started. Is there anything in the OS itself that may have changed to cause this problem? Did you upgrade your kernel version or any libraries, especially libc around when it started happening?
Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Fri Nov 6 10:26:46 PST 1998 Comment-Added: Are you using NFS? Any third party modules?
Re: os-linux/3343: Server dies after 1-20 hours of usage.
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Fri, 06 Nov 1998 21:36:06 + [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. Comment-Added-By: lars Comment-Added-When: Fri Nov 6 10:26:46 PST 1998 Comment-Added: Are you using NFS? Any third party modules? No. NFS is disabled, and we did not compile any 3rd party modules into the server. The only extra module that I enabled was the status module. The server ran fine for about 6 months before this started. Is there anything in the OS itself that may have changed to cause this problem?
Re: os-linux/3343: Server dies after 1-20 hours of usage.
[In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. State-Changed-From-To: open-feedback State-Changed-By: lars State-Changed-When: Wed Nov 4 18:30:46 PST 1998 State-Changed-Why: Are there any messages in your error log? What are your MaxClients, MaxRequestsPerChild and Min/MaxSpareServers settings? Release-Changed-From-To: 1.2.6 and 1.3.3-1.3.3 Release-Changed-By: lars Release-Changed-When: Wed Nov 4 18:30:46 PST 1998
Re: os-linux/3343: Server dies after 1-20 hours of usage.
The following reply was made to PR os-linux/3343; it has been noted by GNATS. From: Mark Herman II [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: os-linux/3343: Server dies after 1-20 hours of usage. Date: Thu, 05 Nov 1998 00:14:10 -0600 Hi, Thanks for the quick reply. Here are the settings you asked for: MaxRequestsPerChild is set to 64. It was set to 30, but I increased it. This seemed to make the server take longer to die. MaxClients is set to 150 MinSpareServers is 5 MaxSpareServers is 10 BTW, we are running a custom transfer log, but we weren't when the problem started. I noticed another message in the database regarding custom logs, but this shouldn't be the cause of the problem. There were no unusual messages in the error log file. I also checked the syslog messages file, and I did notice several messages about possible SYN floods, but the times don't appear to correspond with the unresponsiveness of httpd. Thanks [EMAIL PROTECTED] wrote: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ] Synopsis: Server dies after 1-20 hours of usage. State-Changed-From-To: open-feedback State-Changed-By: lars State-Changed-When: Wed Nov 4 18:30:46 PST 1998 State-Changed-Why: Are there any messages in your error log? What are your MaxClients, MaxRequestsPerChild and Min/MaxSpareServers settings? Release-Changed-From-To: 1.2.6 and 1.3.3-1.3.3 Release-Changed-By: lars Release-Changed-When: Wed Nov 4 18:30:46 PST 1998
os-linux/3343: Server dies after 1-20 hours of usage.
Number: 3343 Category: os-linux Synopsis: Server dies after 1-20 hours of usage. Confidential: no Severity: critical Priority: medium Responsible:apache State: open Class: sw-bug Submitter-Id: apache Arrival-Date: Wed Nov 4 14:40:00 PST 1998 Last-Modified: Originator: [EMAIL PROTECTED] Organization: apache Release:1.2.6 and 1.3.3 Environment: Red Hat Linux with kernel version 2.0.35. It has been patched to glibc2.0.7. I am using gcc2.7.2.3, but the 1.2.6 server we were using came with Red Hat Linux. Description: There is no core dump. The server-status report from the 1.2.6 apache server would show processes marked as running although they no longer existed. Once it reached the maximum number of requests for a child, it would kill the child, but it would still appear as running in the server-status report. It would then start a new process, since it can't use the dead ones. After several hours it would reach the maximum number of processes and stop answering requests. The 1.3.3 version does not exibit the same behavior in the server status report, but it still dies. How-To-Repeat: I don't know what is causing it, so I can't repeat it. You can access the server-status report at the following URL: http://www.v6fbody.com/server-status/ The username and password are: username: apache password: group The website address is http://www.v6fbody.com/ Fix: We started a cron job to restart the server on a periodic basis. Audit-Trail: Unformatted: [In order for any reply to be added to the PR database, ] [you need to include [EMAIL PROTECTED] in the Cc line ] [and leave the subject line UNCHANGED. This is not done] [automatically because of the potential for mail loops. ] [If you do not include this Cc, your reply may be ig- ] [nored unless you are responding to an explicit request ] [from a developer. ] [Reply only with text; DO NOT SEND ATTACHMENTS! ]
