On Tue, Mar 12, 2013 at 5:51 PM, Mark Thomas <ma...@apache.org> wrote:
> On 12/03/2013 21:47, Colin Ingarfield wrote: > > Hello, > > > > We're using Jetty 8.1.3.v20120416 w/ JDBCSessionManager as our > application > > server with Tomcat 7's JDBC Connection pool 7.0.28. We've run this > > particular combination in production since at least Sept 2012. (Java 6 > 64 > > bit/Ubuntu, Amazon RDS/MySQL db.) > > > > This morning one Jetty instance stopped accepting requests and the > problem > > appears to be a deadlock btw a jetty connection handler thread and the > conn > > pool's "PoolCleaner" thread. From jstack trace: > > Deadlock Detection: > > > > Found one Java-level deadlock: > > ============================= > > > > "qtp1840392480-3740": > > waiting to lock Monitor@0x00007f4350001fd0 (Object@0x00000006c01a0e88, > a > > com/mysql/jdbc/JDBC4Connection), > > which is held by "PoolCleaner[2009981184:1363034108768]" > > "PoolCleaner[2009981184:1363034108768]": > > waiting to lock Monitor@0x00007f4350001f28 (Object@0x00000006c1ed5738, > a > > com/mysql/jdbc/JDBC4ResultSet), > > which is held by "qtp1840392480-3740" > > > > Once this happened all other worker threads blocked at various points > > within JDBCSession manager. Obviously restarting the instance resolved > the > > issue for now. > > > > How does one go about diagnosing / resolving such an issue? It's not > clear > > to me if either component is "at fault" here...I have the full jstack > > output as well as a full heap dump of the java process, if that would > help. > > Deadlocks occur when two threads attempt to acquire the same locks but > in a different order. Normally, one of the code paths is at fault. To > figure out which we need to see the full stack trace for the two threads > that are deadlocked. > > Mark > > I experienced deadlock today while revisiting development of some code that would automatically insert some data in the background via @Schedule timers. While first testing the code, I had CDI @RequestScoped bean referencing @Stateless EJBs to perform inserts against many different tables. That is when I experienced the deadlock, but then I changed the bean from CDI @RequestScoped back to @Singleton and @Lock(WRITE) (as advised by David Blevins of TomEE/openEJB...months ago), and immediately the deadlock went away. Using TomEE (tomcat7), ActiveMQ/JMS, @Schedule, @Singleton @Lock(WRITE) and plenty of @Stateless EJBs (1 per table) to get it all done.... and on my development server, takes few seconds for everything to work, and 1 second on the database insert and commit. I assume that the CDI @RequestScoped and @Stateless EJBs were executing concurrently/simultaneously, which I assume is the 2 threads that Mark was talking about here. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
