[
https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rob Davies resolved AMQ-2332.
-----------------------------
Fix Version/s: 5.4.0
(was: 5.4.1)
Resolution: Fixed
Socket timeouts are OS specific - but you can set the property on the
transport used by the broker - see
http://activemq.apache.org/tcp-transport-reference.html
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
> Assignee: Rob Davies
> Fix For: 5.4.0
>
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be
> causing it by a multi-threading problem in the client, we are currently
> looking into this. However, I feel that this is still a bug in the broker
> since it shouldn't be possible to deadlock the broker by incorrect use of the
> client.
> The situation is that we have two clients connected to the broker, one over
> TCP using the Java library, and one connecting over SSL (our own
> implementation, this may also be involved in creating the problem) using the
> NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and
> starts sending messages to the topics. The NMS client connects and attempts
> to connect to some of the same topics, as well as creating a temporary queue.
> As it is doing this initial setup stuff it hangs and goes no further. From
> this point I don't think any more messages get delivered on the topic, and
> Java clients trying to send messages to it hang on the producers send() call
> indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data
> back to the NMS client (the bottom stack trace below), another thread
> (originating from the Java client) is trying to acquire the write lock so
> that it too can write to the client (middle stack trace) while still owning a
> lock on the list of consumers of a topic, meanwhile a third thread (this one
> origination from the NMS client again) is trying to get that lock on the list
> of consumers in order to add to it (top stack trace). For some reason the
> first thread trying to write to the NMS client never finishes, even when the
> NMS client is killed, and so the lock is never released and everything
> involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from
> jconsole, and tracked down the locks involved in the deadlock using a
> debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.copyonwritearrayl...@509c6c30 owned
> by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> copyonwritearrayl...@509c6c30 lock acquired at
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.obj...@58c1c2f1 owned by: TempQueue:
> c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> obj...@58c1c2f1 lock acquired at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
