Hi Chuck -
How the application is doing its locking is definitely exacerbating
the problem of the thread just sitting on that socket read to the
database and never getting to release the lock, this is just one
particular application out of two different tomcats that are locking
up with same core issue. The second app also has a runnable thread
hanging onto a lock with lots of other threads blocking waiting for
that object. This time the lock is not being made in our code but in
DBCP code and it hangs on a socket read to the database as well never
releasing it's locks. Thread stacktrace from this thread below. Our
DBCP and Pool libs are out of date. I'm hoping updating those libs
will allow DBCP to better handle socket read hangups. Still very
concerned about why socket reads are hanging up to db bringing these
locking issues to light.
Almost all other threads are blocked waiting for the lock this thread
has on SharedPoolDataSource below
"catalina-exec-company1-2" daemon prio=10 tid=0x000000004bf5b800
nid=0x41a2 runnable [0x0000000042be1000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at oracle.net.ns.Packet.receive(Packet.java:293)
at oracle.net.ns.DataPacket.receive(DataPacket.java:92)
at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:174)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:119)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:94)
at oracle.net.ns.NetInputStream.read(NetInputStream.java:79)
at
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:112)
at
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:73)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1040)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1016)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:183)
at
oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:780)
at
oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:855)
at
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1187)
at
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1378)
- locked <0x00002aaad1f64df8> (a oracle.jdbc.driver.T4CConnection)
at
oracle.jdbc.driver.OracleStatementWrapper.executeQuery(OracleStatementWrapper.java:387)
at
org.apache.commons.dbcp.datasources.KeyedCPDSConnectionFactory.validateObject(KeyedCPDSConnectionFactory.java:163)
at
org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:837)
- locked <0x00002aaace878d18> (a
org.apache.commons.pool.impl.GenericKeyedObjectPool)
at
org.apache.commons.dbcp.datasources.SharedPoolDataSource.getPooledConnectionAndInfo(SharedPoolDataSource.java:165)
- locked <0x00002aaaced306f8> (a
org.apache.commons.dbcp.datasources.SharedPoolDataSource)
at
org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:631)
at
org.apache.commons.dbcp.datasources.InstanceKeyDataSource.getConnection(InstanceKeyDataSource.java:615)
at
com.mycompany.db.ConnectionHandler.getConnection(ConnectionHandler.java:150)
at
com.mycompany.community.dataaccess.CustomerDAO.getBasicInfo(CustomerDAO.java:108)
at
com.mycompany.community.domain.CommunityUser.loadBasicInfo(CommunityUser.java:207)
at
com.mycompany.community.domain.CommunityUser.isRegisteredMember(CommunityUser.java:598)
at
org.apache.jsp.templates.home.leftnav_jsp._jspService(leftnav_jsp.java:626)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at
org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
at
org.apache.jsp.templates.home.home_005ftemplate_jsp._jspService(home_005ftemplate_jsp.java:149)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:377)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551)
at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488)
at
com.mycompany.servlet.filters.NavigationFilter.retrieveTemplate(NavigationFilter.java:381)
at
com.mycompany.servlet.filters.NavigationFilter.doFilter(NavigationFilter.java:185)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:427)
at
org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:384)
at
org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2087)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
On Fri, Oct 22, 2010 at 6:34 AM, Caldarale, Charles R
<chuck.caldar...@unisys.com> wrote:
>
> From: Jason Britton [mailto:jbritto...@gmail.com]
> > Subject: Re: tomcat 6.0.29 hung
>
>
> > My thoughts are this is the thread causing all the backup
> > AJP Thread 1 RUNNABLE, sitting on socketRead0, Has locked
> > <0x00002aaad303cad0> (a com.acompany.model.customer.CustomerData)
> > which almost all other threads are blocked waiting for lock on
>
> Which means you have an extremely poor application design. The webapp should
> never set a global lock and then make a database request. Either the lock
> must be of much finer granularity, or not set until after the DB request is
> responded to.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
