Thanks Chris. You have been very helpful. I will implement your
recommendations and see what happens.
Quick question, when you look at a thread dump, how do you tell which
threads are idle?
Also, any inputs on my other postings regarding the session count and
catalina.out logging?
http://mail-archives.apache.org/mod_mbox/tomcat-users/200610.mbox/[EMAIL
PROTECTED]
http://mail-archives.apache.org/mod_mbox/tomcat-users/200610.mbox/[EMAIL
PROTECTED]
Thanks again,
-Riz.
Christopher Schultz wrote:
Riz,
here is the thread dump from when the app hung.
Any insights?
Well, a quick run-through of the thread dump suggests that your DBCP was
definitely stalled for some reason.
I counted about 70 idle threads, about 30 threads waiting on database
connections, and maybe 15 doing other things (handling requests,
accepting socket connections, running Telnet (wtf?) and a few threads
doing SMTP. There were also a few "Quartz" threads running, which I
assume were performing some kind of batch operations or something. Most
of them were either idle of sending email messages. Maybe one of two
were hung in the DBCP wait like the others.
Note that not all of your threads were hung while trying to
authenticate. That probably means that some of your users had
authenticated before your connection pool dried up, and were trying to
do other things.
I'm guessing that you have a connection leak, somewhere. Do you always
use Spring to manage your connections, or do you sometimes get a
connection using the DBCP directly?
I'm asking because it's more likely that you have a connection leak in
/your/ code than there is a connection leak in Spring (no offense ;).
Check for the places where you handle JDBC connections yourself and make
sure that you are following all the "rules":
1. Make sure you close your statements, result sets, and connection
objects in a finally block.
2. Make sure that you surround everything from #1 in try/catch blocks
and LOG ALL EXCEPTIONS.
Here are some helpful suggestions above and beyond the above "rules":
1. Make sure that you catch all possible exceptions (potentially
re-throwing them) and rollback if you are inside a transaction.
2. Turn on abandoned connection logging in development (and production
if you can afford it). There's nothing better than empirical evidence
to help you solve problems. The performance hit you take by turning
on this debugging information is well worth it compared to server
downtime due to connection pool exhaustion.
3. Set your connection pool size in development to a fixed size of 1
(yes, that's ONE). This will help you find connection leakages ASAP,
since the first time it happens, you're screwed right away.
If you have only one place in your code where connections are leaked,
it may take a long time to exhaust a connection pool containing 50
connections. If you drop it down to 1, you'll know right away ;)
Lastly, you should ask on the Spring list if there are any known
connection leaks in your version of Spring (you are not likely to get a
warm response to that question, but it needs to be asked), but only
after you have audited your own code.
-chris
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
