Hi ALL,
We are using tomcat 6.0 in production environment and facing a critical problem that tomcat hangs with variable frequency in a week. We have tried to figure out the problem but couldn't reach the root cause. To investigate the cause we used the lambda probe tool. Lamba probe shows multiple threads are blocked when tomcat hangs. Normally i have seen a similar type of stack trace for the thread which lambda probe shows as blocked, I am pasting it as follows: net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 ) net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 ) net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 ) net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 ) net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 ) net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 ) net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close ( DelegatingResultSet.java:187 ) org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close ( DelegatingStatement.java:163 ) org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate ( DelegatingConnection.java:426 ) org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close ( DelegatingConnection.java:246 ) org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose ( PoolableConnection.java:122 ) org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject ( PoolableConnectionFactory.java:628 ) org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject ( GenericObjectPool.java:1247 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject ( AbandonedObjectPool.java:125 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned ( AbandonedObjectPool.java:158 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject ( AbandonedObjectPool.java:77 ) org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection ( PoolingDataSource.java:106 ) org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection ( BasicDataSource.java:1044 ) com.avanza.contactcenter.dbutil.DBConnection.getConnection ( DBConnection.java:154 ) com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList ( TableFieldsMappingProcessor.java:696 ) com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch ( CustomerDetailActionProcessor.java:187 ) com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo ( RdvInboundActionProcessor.java:817 ) com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage ( RendezvousMessageSource.java:431 ) com.avanza.contactcenter.integration.RendezvousMessageSource.run ( RendezvousMessageSource.java:136 ) As we have been investigating the issue from a month, so i would like to share it with you though you can understand the scenario in depth. Initially we thought that there are connection leak problems in our application because the lambda probe showed us that db connections are continuously busy. We scanned twice the DAO layer of our application to search any connection leaks, but we didn't find any. At that time our application's Context.xml file didn't contain the removeAbandoned property. After DAO layer scan, we decide to set the removeAbandoned = true, so that connection pool refreshes at run time. But this also didn't work for us. Lambda probe still showed us blocked threads when tomcat hanged, but yes by setting the removeAbandoned = true connection pool continued to refresh. When tomcat hanged today we noticed following statistics on lambda probe, (along with blocked threads for which stack trace is already provided above): System OVerView Memory utilization -- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB OS information JVM: Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_10-b03 Java HotSpot(TM) Client VM) OS: Windows 2003 (Service Pack 2) x86 5.2 Processors: 8 Current time: Wed Dec 22 13:00:48 GMT+05:00 2010 Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0 Current memory usage Used Committed Maximum Initial Group Survivor Space: 1.26Mb 4.63Mb 7.88Mb 4.56Mb HEAP Perm Gen: 34.28Mb 34.50Mb 64.00Mb 8.00Mb NON_HEAP Tenured Gen: 115.20Mb 553.88Mb 945.25Mb 553.88Mb HEAP Eden Space: 2.08Mb 37.00Mb 63.00Mb 37.00Mb HEAP Code Cache 12.37Mb 12.47Mb 32.00Mb 192.00Kb NON_HEAP Total 165.65Mb 642.47Mb 1.09Gb 603.63Mb TOTAL OS information OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb (In above line,Free RAM is 2GB, it might be due to that operating system is 32 bit and hardware is also 32 bit) STATUS: ajp-8009 Current thread count: 0 Current threads busy: 0 Max threads: 200 Max spare threads: 0 Min spare threads: 0 Max time (ms): 0 Processing time (ms): 0 Request count: 0 Error count: 0 Received: 0b Sent: 0b No processors http-8080 Current thread count: 60 Current threads busy: 5 Max threads: 200 Max spare threads: 0 Min spare threads: 0 Max time (ms): 430500 Processing time (ms): 133505594 Request count: 390058 Error count: 8904 Received: 24Mb Sent: 4Gb Context.xml <Resource factory="org.apache.tomcat.dbcp.dbcp.BasicDataSourceFactory" auth="Application" removeAbandoned="true" logAbandoned="true" validationQuery="select 1" driverClassName="net.sourceforge.jtds.jdbc.Driver" maxActive="60" maxIdle="20" maxWait="10000" name="DB_Pool" type="javax.sql.DataSource" url="jdbc:jtds:sqlserver://10.22.4.29/XYZ" /> Note that we have noticed multiple times that there are no deadlocks at database end when tomcat crashed and memory and CPU utilization of application server machine and database machine is always normal. We are using SQL Server 2005 Enterprise in production environment. The number of concurrent users to use the application simultaneously is 50. Will appreciate if any one can give the idea by looking at above statistics that what is area of problem. Will appreciate. Regards, David