Have you found any java.lang.Thread.State: RUNNABLE threads? They are usually more interesting if it comes to a high cpu :-) Also, as David posted, what is the HEAP usage? it's usually at the end of the dump.
regards Leon On Mon, Jan 26, 2009 at 5:37 PM, Pieter Temmerman <ptemmerman....@sadiel.es> wrote: > Hi all. > > I've been investigating why one of our applications (running in Tomcat > 5.5.7) suddenly freezes after a variable amount of time (sometimes > 10min, sometimes 2 hours). > > Disclaimer: I'm not the developer of the application, nor do I know the > exact details of how stuff is implemented. I know..it sucks. > > Memory usage looks healthy, but CPU usage goes sky high (mainly caused > by the Java Tomcat process). > So I made a thread dump and the first thing I noticed was the large > amount of TP-ProcessorXX threads, most of them in WAITING state. > > A small snippet of the thread dump (it's very very big). > > "TP-Processor290" daemon prio=1 tid=0x00002aaab47acd30 nid=0x6486 in > Object.wait() [0x0000000056ae4000..0x0000000056ae6e10] > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aabadaabff8> (a > org.apache.commons.pool.impl.GenericObjectPool) > at java.lang.Object.wait(Object.java:474) > at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810) > - locked <0x00002aabadaabff8> (a > org.apache.commons.pool.impl.GenericObjectPool) > at > org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > at > org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81) > at > org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423) > at > org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144) > at > org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139) > at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1547) > at org.hibernate.loader.Loader.doQuery(Loader.java:673) > > My first question is, what is a TP-Processor exactly? Is each client > connection to Tomcat being assigned a TP-Processor or am I wrong? > > Anyway, back to the thread dump itself. > There are a lot (read: +100) of those TP-Processor threads in waiting > state which mention org.hibernate.blablabla. So my guess is that the > freeze is being caused by a database connection pool that is out of open > connections, and thus the application those threads are waiting until > there is a free one. But for some kind of reason, there is never a free > one available, and the application just won't work until Tomcat is > restarted. > > In the assumption that this is the reason for the application to hang, > my "thread dump decoding knowledge" is too limited to be sure what is > causing this situation. Is the thread pool just too small, is the > application not closing it's connections, and thus running out of > connections. I have no idea. > > Therefor, my second question. > > "TP-Processor290" daemon prio=1 tid=0x00002aaab47acd30 nid=0x6486 in > Object.wait() [0x0000000056ae4000..0x0000000056ae6e10] > at java.lang.Object.wait(Native Method) > - waiting on <0x00002aabadaabff8> (a > org.apache.commons.pool.impl.GenericObjectPool) > at java.lang.Object.wait(Object.java:474) > at > org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810) > - locked <0x00002aabadaabff8> (a > org.apache.commons.pool.impl.GenericObjectPool) > > > Note the line "locked <0x00002aabadaabff8>" and "waiting on > <0x00002aabadaabff8>" later on. So first it's locking that "thing" and > then it's waiting on that "thing". This same number is coming back in > each TP-Processor that is in waiting state. That seems rather weird to > me. > So I was wondering: > a. Is that normal behavior? > b. Is there any way to know what the 0x00002aabadaabff8 means? > > My scientific calculator says it's rather an insane number when trying > to convert it to decimal. > Maybe it's just as easy as it reads: Waiting on 0x0002aabad....which is > a GenericObjectPool. > > Any help to confirm my supposition will be kindly appreciated. > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
