We solved this problem. Turns out it has nothing to do with JBoss Messaging and everything to do with our code. I'm relating our experiences here in case anyone else encounters a similar problem.
Queue clients are not only a JBoss server hosting MDB's listening to those queues, but also freestanding programs. Because there is a fair number of these freestanding programs, they each cache a QueueConnection. These programs are multithreaded. In short, there was a race condition in our code which caused the QueueConnection.close() to hang. After this occurred enough times, the JBoss session pool ran out of sessions. We observed other strange behavior as well, including the fact that we could not undeploy applications from JBoss listening in on queue's affected by this problem. Our initial approach was to interrupt() one of the deadlocked threads, drop the reference, and create a new one with a fresh QueueConnection. However, onMessage() was called immediately once again, before the deadlock condition could resolve itself. A side-effect of this nonsense was that the JVM filled up with orphaned threads all waiting on QueueConnection.close(). Eventually we had OutOfMemoryE! rrors. The race condition never occurred in JBossMQ because it seems that the thread which calls onMessage() in a MessageListener was the same thread as calls onException() in an ExceptionListener. In JBoss Messaging, it appears these are different threads. Our problem was that an onException() called while onMessage() was being processed caused a deadlock under a particular, application-specific condition. After this, QueueConnection.close() would wait for onMessage() to complete, which was forever. View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4033144#4033144 Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4033144 _______________________________________________ jboss-user mailing list jboss-user@lists.jboss.org https://lists.jboss.org/mailman/listinfo/jboss-user
