Deadlock when TCP connection breaks
-----------------------------------
Key: AMQNET-366
URL: https://issues.apache.org/jira/browse/AMQNET-366
Project: ActiveMQ .Net
Issue Type: Bug
Components: ActiveMQ
Affects Versions: 1.5.2
Reporter: Mikael Finstad
Assignee: Jim Gomes
When an ActiveMQ TCP connection gets broken (for instance when the ActiveMQ
server goes down), the AsyncCallExceptionListener is used by helper classes
like SingleConnectionFactory and SimpleMessageListenerContainer from
springframework.net to reconnect when connection becomes available again.
>From this thread, these classes will call Close() on the connection which in
>turn calls Shutdown on the ThreadPoolExecutor (from the final block in
>Connection.cs:Close()).
Inside Shutdown(), the execution hangs on this.executionComplete.WaitOne(), and
there is a deadlock because the AsyncCallExceptionListener call never returns,
and nobody else triggers the executionComplete event
Simple test case to reproduce the problem:
using System;
using System.Threading;
using Apache.NMS.ActiveMQ;
using NUnit.Framework;
using Spring.Messaging.Nms.Listener;
[TestFixture]
public class ActiveMQTests
{
[Test]
[Ignore]
public void TestConnection()
{
var connectionFactory = new ConnectionFactory("tcp://localhost:61616")
{UserName = "qwe", Password = "asd"};
var messageListenerContainer = new SimpleMessageListenerContainer
{
ConnectionFactory = connectionFactory,
DestinationName = "test.test.in"
};
messageListenerContainer.Start();
Thread.Sleep(TimeSpan.FromMinutes(10));
}
}
Start debugging. Wait until it connects tothe ActiveMQ server, shutdown the
ActiveMQ server, pause execution and look at the Threads.
Example call stack:
[In a sleep, wait, or join]
[External Code]
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.Shutdown()
Line 121 + 0xe bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Connection.Close() Line 592 + 0xd
bytes
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Connections.ConnectionFactoryUtils.ReleaseConnection(Apache.NMS.IConnection
connection, Apache.NMS.IConnectionFactory cf, bool started) Line 74 + 0x9
bytes
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.AbstractListenerContainer.RefreshSharedConnection()
Line 432
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.SimpleMessageListenerContainer.RefreshConnectionUntilSuccessful()
Line 242
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.SimpleMessageListenerContainer.OnException(System.Exception
exception) Line 210
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Connection.AsyncCallExceptionListener(object
error) Line 945 + 0x11 bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.Future.Run()
Line 63 + 0x14 bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.QueueProcessor(object
unused) Line 150 + 0xa bytes
[External Code]
Should it not be possible to Close the connection from the
AsyncCallExceptionListener thread?
I've tried running executor.Shutdown() in Connection.Close (under finally
block) in a new thread and recompiling the NMS.ActiveMQ DLL, Like so:
if(executor != null)
{
var thread = new Thread(delegate()
{
executor.Shutdown();
});
thread.Start();
}
This seems to solve the problem, however I am not certain that this will not
break anything else.
I am using Spring framework.net version 1.3.2 and apache.NMS version 1.5.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
