RE: Apache Processes hanging
I can't help you, but I have been having the same problem. ( I had thought it was because we're running old code - Stronghold with mod_perl 1.24. I'm really hoping it goes away when we upgrade to 1.3.19 w 1.25, or at least I can have a better chance debugging it. ) My solution for the hung readers problem is to write another daemon that looks for them and kills them off. In order to do this, the daemon uses the /server-status URL to ask the web servers what processes are in the read state. If you want, I can post the code. -P -Original Message- From: Kevin Slean [mailto:[EMAIL PROTECTED]] Sent: Thursday, May 03, 2001 10:38 AM To: [EMAIL PROTECTED] Subject: Apache Processes hanging Mod_perlers, I have a problem with Apache and was looking for your thoughts on my problem or some additional mailing lists more focused on just Apache. I have 70 httpd daemons running and some of them just appear to hang. As time goes by, the number of hung processes increases such that there are no working ones left to perform real work. Consequently transaction processing performance drops eventually to zero. Our web transactions running through these httpd daemons should not take more than 60 seconds in a worst case scenario. Yet, some of these 'hung' processes have been on the same transaction for over 30 hours. I originally thought that this was a mod_perl problem and was buried in the CGI/Perl routines performing the transactions. However, upon closer inspection, I have found that these hanging processes are also running JAVA servlets and gif gets. This makes me suspect that it is an Apache problem. I ran truss on the hung processes and found that they fell into two camps. The first group was sitting at a read system call. The second group was in a loop with sigalarm going off every 10 seconds. We are running the following: Hardware: Sun Ultra-250 OS: Solaris 5.7 patch level 106541-12 Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24 secured_by_Raven/1.4.2 Any ideas, thoughts, and comments are welcome. Thanks. Kevin
Re: Apache Processes hanging
On Thu, 3 May 2001, Kevin Slean wrote: Mod_perlers, I have a problem with Apache and was looking for your thoughts on my problem or some additional mailing lists more focused on just Apache. I have 70 httpd daemons running and some of them just appear to hang. As time goes by, the number of hung processes increases such that there are no working ones left to perform real work. Consequently transaction processing performance drops eventually to zero. Our web transactions running through these httpd daemons should not take more than 60 seconds in a worst case scenario. Yet, some of these 'hung' processes have been on the same transaction for over 30 hours. I originally thought that this was a mod_perl problem and was buried in the CGI/Perl routines performing the transactions. However, upon closer inspection, I have found that these hanging processes are also running JAVA servlets and gif gets. This makes me suspect that it is an Apache problem. I ran truss on the hung processes and found that they fell into two camps. The first group was sitting at a read system call. The second group was in a loop with sigalarm going off every 10 seconds. We are running the following: Hardware: Sun Ultra-250 OS: Solaris 5.7 patch level 106541-12 Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24 secured_by_Raven/1.4.2 Any ideas, thoughts, and comments are welcome. Thanks. Since you already truss()ed the processes and know that it's not a mod_perl problem, you probably want to take the question to the httpd mailing list. Meanwhile you can use Apache::Watchdog::RunAway to monitor and kill the processes that hang. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://logilune.com/ http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: Apache Processes hanging
At 10:37 AM -0400 5/3/01, Kevin Slean wrote: Mod_perlers, I have a problem with Apache and was looking for your thoughts on my problem or some additional mailing lists more focused on just Apache. I have 70 httpd daemons running and some of them just appear to hang. As time goes by, the number of hung processes increases such that there are no working ones left to perform real work. Consequently transaction processing performance drops eventually to zero. Our web transactions running through these httpd daemons should not take more than 60 seconds in a worst case scenario. Yet, some of these 'hung' processes have been on the same transaction for over 30 hours. I originally thought that this was a mod_perl problem and was buried in the CGI/Perl routines performing the transactions. However, upon closer inspection, I have found that these hanging processes are also running JAVA servlets and gif gets. This makes me suspect that it is an Apache problem. I ran truss on the hung processes and found that they fell into two camps. The first group was sitting at a read system call. The second group was in a loop with sigalarm going off every 10 seconds. We are running the following: Hardware: Sun Ultra-250 OS: Solaris 5.7 patch level 106541-12 Apache: Apache/1.3.11 (Unix) ApacheJServ/1.1.2 mod_perl/1.24 secured_by_Raven/1.4.2 Any ideas, thoughts, and comments are welcome. Thanks. Kevin I'm having similar problems, but we think it's directly related to Oracle. Basically, a connection is made to the Oracle database, a transaction is started and finished, but the connection to the database doesn't go away and the statement (at least from the oracle side) never seems to finish. The data is present in the database (these are insert statement, btw). Over time, every process collects one of these hanging statements and it eventually overwhelms our oracle database. The only solution is to restart apache every 5 minutes to eliminate the built-up non-finished transactions. Has anyone seen this before? Rob -- As soon as you make something foolproof, someone will create a better fool.
RE: Apache Processes hanging
I'm having similar problems, but we think it's directly related to
Oracle. Basically, a connection is made to the Oracle database, a
transaction is started and finished, but the connection to the
database doesn't go away and the statement (at least from the oracle
side) never seems to finish. The data is present in the database
(these are insert statement, btw). Over time, every process collects
one of these hanging statements and it eventually overwhelms our
oracle database. The only solution is to restart apache every 5
minutes to eliminate the built-up non-finished transactions.
Yeah... two things: CONCURRENCY and TRANSACTIONS.
Concurrency: Are there any other processes/reports/queries running at the
time of insert? That will lock ALL of them, waiting for the insert to
complete so the lock is released. Or, Another Interesting Way To Lock A
Really Buff Linux Server (tm).
Transactions: how's this one for fun? I started experimenting with
Apache::Session::Oracle to see what I could see. Usually I run w/
$dbh-{AutoCommit} = 1, which is the default, because most of the time I'm
just running SELECT's. But ::Oracle wouldn't ever complete the transaction,
hanging that server process and eventually most of the httpd system, all
waiting for the commit() on the INSERT (from the new Session) that doesn't
complete. sigh I ended up having to do a local block, with Commit = 1:
{
local $dbh-{AutoCommit} = 0;
tie %session, 'Apache::Session::Oracle', $session_id, { Handle = $dbh,
Commit = 1};
$session_id = $session{_session_id}; # save a copy
_set_cookie( $r, SESSION_COOKIE, $session{_session_id} );
$session{referer} ||= $referer; # preserve prior entries
untie %session;
}
HTH!
L8r,
Rob
Re: Apache Processes hanging
This answer has nothing to do with modperl - sorry. I have had the same problem on a Sybase database with a 'normal' application. This situation can occur due to a database (b)locks, particularly if a transaction is composed of more than one update, or it fires a referential trigger which has the same effect. The first question is what is the lock-level. Is it at the row or at the page - I presume the row. If the insert causes a 'fault' such that an index page becomes full and has to split then the whole index page will be locked regardless of row-level locking. If the second part of transaction is waiting on someone else then we can get the deadly embrace situation. However, it can normally be cleared on time-outs. If however you are acquiring these blocks faster then they time out, then in very short order you will be .. er.. screwed. Sometimes it can help to do dirty reads if the data you need to present does not need to be up to date. One of the cures for this is an update pipe. Instead of each 'program' doing the insert on the database they funnel the updates to a single threading process. Reads of course can be be done by the individual 'programs'. Can you confirm that the connection is freed if you kill the process that is blocked? If so this gives you another way out. - Original Message - From: Robert Landrum [EMAIL PROTECTED] To: Kevin Slean [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Thursday, 03 May, 2001 03:40 PM Subject: Re: Apache Processes hanging At 10:37 AM -0400 5/3/01, Kevin Slean wrote: Mod_perlers, I have a problem with Apache and was looking for your thoughts on my problem or some additional mailing lists more focused on just Apache. I have 70 httpd daemons running and some of them just appear to hang. As time goes by, the number of hung processes increases such that there are no working ones left to perform real work. Consequently transaction processing performance drops eventually to zero. Our web transactions running through these httpd daemons should not take more than 60 seconds in a worst case scenario. Yet, some of these 'hung' processes have been on the same transaction for over 30 hours. I originally thought that this was a mod_perl problem and was buried in the CGI/Perl routines performing the transactions. However, upon closer inspection, I have found that these hanging processes are also running JAVA servlets and gif gets. This makes me suspect that it is an Apache problem. I ran truss on the hung processes and found that they fell into two camps. The first group was sitting at a read system call. The second group was in a loop with sigalarm going off every 10 seconds. I'm having similar problems, but we think it's directly related to Oracle. Basically, a connection is made to the Oracle database, a transaction is started and finished, but the connection to the database doesn't go away and the statement (at least from the oracle side) never seems to finish. The data is present in the database (these are insert statement, btw). Over time, every process collects one of these hanging statements and it eventually overwhelms our oracle database. The only solution is to restart apache every 5 minutes to eliminate the built-up non-finished transactions. Has anyone seen this before? Rob -- As soon as you make something foolproof, someone will create a better fool.
