Thanks for your feedback, Rainer.
The DAO objects/methods that you've mentioned are used for almost
every page on our website. So I am not surprised to find them in the
thread dumps. However, as you pointed out, the parameters to these
methods change depending on the page they are on and if the user is
logged in or not.
I did not check the CPU usage when the issue happened. I just checked
the web application logs and found that first there was an
OutOfMemoryError, followed by MySQL errors when
TWFotoSetListDAO.getAllFotosets tried to get a database connection.
I am *guessing* :
I changed the session timeout to 4 hours for the Jasig CAS
Single-Signon web application. This stores a ton of tickets in memory.
During a 4 hour period when there are more http requests, the 512MB of
Java Heap space is not sufficient to store all these tickets. During
that time OutOfMemoryError happens and all hell breaks loose
subsequently.
I probably should decrease the session-timeout to 1 hour or so and see
if that changes things. Would you agree, Rainer?
Thanks,
Joe
2009 Oct 06 / 10:48:43 ERROR -
[org.apache.catalina.core.ContainerBase] : Exception invoking periodic
operation:
java.lang.OutOfMemoryError: Java heap space
2009 Oct 06 / 10:48:43 FATAL - [tw.conn.TWConnectionManager] :
com.mysql.jdbc.CommunicationsException: Communications link failure
due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2351)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2862)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3649)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1176)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2558)
at com.mysql.jdbc.Connection.<init>(Connection.java:1485)
at
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at
org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
at
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
at
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:771)
at
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
at tw.conn.TWConnectionManager.getConnection(Unknown Source)
at tw.beans.TWFotoSetListDAO.getAllFotosets(Unknown Source)
at tw.beans.TWViewBean.getAllFotosets(Unknown Source)
at tw.servlet.TWQueryServlet.doPost(Unknown Source)
at tw.servlet.TWQueryServlet.doGet(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at
org.apache.jsp.groupby_005fitems_jsp._jspService(groupby_005fitems_jsp.java:152)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:332)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
** END NESTED EXCEPTION **
On Tue, Oct 6, 2009 at 1:11 PM, Rainer Jung <rainer.j...@kippdata.de> wrote:
> On 06.10.2009 19:44, Joe Hansen wrote:
> I will only comment the threads for the AJP pool. Everything else seems
> not relevant:
>
> Dump 1:
>
> 20 threads connected to Apache, waiting for the next request
> 3 threads idle in the pool
> 1 thread waiting for the next connection
> 0 threads working on requests
>
> Dump 2:
>
> 12 threads connected to Apache, waiting for the next request
> 5 threads idle in the pool
> 1 thread waiting for the next connection
> 14 threads working on requests
>
> Dump 3:
>
> 11 threads connected to Apache, waiting for the next request
> 5 threads idle in the pool
> 1 thread waiting for the next connection
> 15 threads working on requests
>
> Dump 4:
>
> 11 threads connected to Apache, waiting for the next request
> 5 threads idle in the pool
> 1 thread waiting for the next connection
> 15 threads working on requests
>
> Those busy threads in dump 2, 3 and 4 are the same except for one, which
> started only in dump 3.
>
> Most of them are busy working on data. They are *not* waiting for some
> other system, like database or similar.
>
> Out of the 14+15+15 thread stacks, that are interesting, 42 work on some
> DAOs.
>
> Those are the DAO methods they work on:
>
> 18 tw.beans.TWFotoSetDAO.getFotosetFotos
> 11 tw.beans.TWEmailUpdateListDAO.getEmailUpdateList
> 3 tw.beans.TWFotoSetListDAO.getAllFotosets
> 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery2
> 3 tw.beans.TWFotoSetDAO.getFotosetFotosQuery
> 3 tw.beans.TWEmailUpdateDAO.getEmailUpdate
> 1 tw.beans.TWEmailUpdateListDAO.getEmailUpdateListQuery
>
> It seems your application is CPU heavy. Either the data objects handled
> are to heavy weight (maybe some user having huge Fotoset or Email list)
> or the request rate is simply to large. Is the CPU saturated during the
> problems?
>
> I would activate a good access log and try to find out from that and
> your webapp logs what maybe special about these web requests or users.
>
> Regards,
>
> Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
