Hi ALL,
We are using tomcat 6.0 in production environment and facing a critical problem
that tomcat hangs with variable frequency in a week. We have tried to figure
out
the problem but couldn't reach the root cause. To investigate the cause we used
the lambda probe tool. Lamba probe shows multiple threads are blocked when
tomcat hangs. Normally i have seen a similar type of stack trace for the thread
which lambda probe shows as blocked, I am pasting it as follows:
net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 )
net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 )
net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 )
net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 )
net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 )
net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 )
net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )
org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close (
DelegatingResultSet.java:187 )
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close (
DelegatingStatement.java:163 )
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate (
DelegatingConnection.java:426 )
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close (
DelegatingConnection.java:246 )
org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose (
PoolableConnection.java:122 )
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject (
PoolableConnectionFactory.java:628 )
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject (
GenericObjectPool.java:1247 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject (
AbandonedObjectPool.java:125 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned (
AbandonedObjectPool.java:158 )
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject (
AbandonedObjectPool.java:77 )
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection (
PoolingDataSource.java:106 )
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection (
BasicDataSource.java:1044 )
com.avanza.contactcenter.dbutil.DBConnection.getConnection (
DBConnection.java:154 )
com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList
( TableFieldsMappingProcessor.java:696 )
com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch
( CustomerDetailActionProcessor.java:187 )
com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo
( RdvInboundActionProcessor.java:817 )
com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage
( RendezvousMessageSource.java:431 )
com.avanza.contactcenter.integration.RendezvousMessageSource.run (
RendezvousMessageSource.java:136 )
As we have been investigating the issue from a month, so i would like to share
it with you though you can understand the scenario in depth. Initially we
thought that there are connection leak problems in our application because the
lambda probe showed us that db connections are continuously busy. We scanned
twice the DAO layer of our application to search any connection leaks, but we
didn't find any. At that time our application's Context.xml file didn't
contain
the removeAbandoned property. After DAO layer scan, we decide to set
the removeAbandoned = true, so that connection pool refreshes at run time. But
this also didn't work for us. Lambda probe still showed us blocked threads when
tomcat hanged, but yes by setting the removeAbandoned = true connection pool
continued to refresh.
When tomcat hanged today we noticed following statistics on lambda probe,
(along
with blocked threads for which stack trace is already provided above):
System OVerView
Memory utilization
-- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB
OS information
JVM: Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_10-b03 Java
HotSpot(TM) Client VM)
OS: Windows 2003 (Service Pack 2) x86 5.2
Processors: 8
Current time: Wed Dec 22 13:00:48 GMT+05:00 2010
Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0
Current memory usage
Used Committed Maximum Initial Group
Survivor Space: 1.26Mb 4.63Mb 7.88Mb 4.56Mb HEAP
Perm Gen: 34.28Mb 34.50Mb 64.00Mb 8.00Mb NON_HEAP
Tenured Gen: 115.20Mb 553.88Mb 945.25Mb 553.88Mb HEAP
Eden Space: 2.08Mb 37.00Mb 63.00Mb 37.00Mb HEAP
Code Cache 12.37Mb 12.47Mb 32.00Mb 192.00Kb NON_HEAP
Total 165.65Mb 642.47Mb 1.09Gb 603.63Mb TOTAL
OS information
OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free
RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb
(In above line,Free RAM is 2GB, it might be due to that operating system is 32
bit and hardware is also 32 bit)
STATUS:
ajp-8009
Current thread count: 0 Current threads busy: 0 Max threads: 200 Max spare
threads: 0 Min spare threads: 0
Max time (ms): 0 Processing time (ms): 0 Request count: 0 Error count: 0
Received: 0b Sent: 0b
No processors
http-8080
Current thread count: 60 Current threads busy: 5 Max threads: 200 Max spare
threads: 0 Min spare threads: 0
Max time (ms): 430500 Processing time (ms): 133505594 Request count: 390058
Error count: 8904 Received: 24Mb Sent: 4Gb
Context.xml
<Resource factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory"
auth="Application" removeAbandoned="true" logAbandoned="true"
validationQuery="select 1" driverClassName="net.sourceforge.jtds.jdbc.Driver"
maxActive="60" maxIdle="20" maxWait="10000" name="DB_Pool"
type="javax.sql.DataSource" url="jdbc:jtds:sqlserver://10.22.4.29/XYZ" />
Note that we have noticed multiple times that there are no deadlocks at
database
end when tomcat crashed and memory and CPU utilization of application server
machine and database machine is always normal. We are using SQL Server 2005
Enterprise in production environment. The number of concurrent users to use the
application simultaneously is 50.
Will appreciate if any one can give the idea by looking at above statistics
that
what is area of problem. Will appreciate.
Regards,
David
