Hi All,
We're running multiple Tomcats (4.1.30) on Linux (2.4.21) fronted by different versions of Apache HTTP servers (1.3 and 2.0 flavors) using mod_jk and mod_jk2. These servers handle upwards of 60 Million pages a day. The app uses JDBC to talk to an Oracle database through the Tomcat DBCP connection pool.
After some time of high-usage, some of these Tomcats start hanging JK connections from the web servers, eventually consuming them all, therefore hanging the web servers as well.
Dumping the thread stack traces on these problematic Tomcats show several threads locked in a particular DBCP call:
"TP-Processor19" daemon prio=1 tid=0x08075a90 nid=0x22ec waiting for monitor entry [7beea000..7beebcd0]
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
- waiting to lock <0x85336b60> (a org.apache.commons.dbcp.PoolableConnectionFactory)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:816)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
...
After a while:
"TP-Processor19" daemon prio=1 tid=0x08075a90 nid=0x22ec waiting for monitor entry [7beea000..7beebcd0]
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
- waiting to lock <0x8527d4c8> (a org.apache.commons.dbcp.PoolableConnectionFactory)
Note that the same thread is still waiting. However, the makeObject() call reports that it's waiting to lock on some other number. I don't know if this is meaningful.
Later, additional threads appear as waiting on a different call:
"TP-Processor351" daemon prio=1 tid=0x79106258 nid=0x26c0 in Object.wait() [6dadf000..6dadfcd0]
at java.lang.Object.wait(Native Method)
- waiting on <0x8527aab8> (a org.apache.commons.pool.impl.GenericObjectPool)
at java.lang.Object.wait(Object.java:429)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:793)
- locked <0x8527aab8> (a org.apache.commons.pool.impl.GenericObjectPool)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
At first we thought this could be related to the Linux NPTL - JVM threading bug and set the env variable LD_ASSUME_KERNEL=2.4.19, but this didn't help. We have upgraded to the latest 1.4 JVM (1.4.2_07) and we're still seeing the problem.
Our code has been carefully audited twice and absolutely all ResultSets and Statements are closed after usage and all connections properly returned to the pool.
This happens predictably (although not periodically) when there are a high number (> 1.500) of concurrent sessions on a single Tomcat instance over a few minutes/hours.
Has anybody else seen these locks? Are we missing something obvious about DBCP configuration and usage?
Thanks for your help,
Ike
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
