1. What tells you that the three threads at the bottom are deadlocked? Usually I'd expect to see that those threads were holding locks that the other threads were waiting to acquire.
2. When you see the javax.jms.InvalidClientIDException, which broker (hub master, hub slave, or spoke) do you see that in the logs for? It looks like the spoke log; is that accurate? And you said you thought a network outage and failover had happened; do you have any more information about what you think happened, how long it lasted, what each broker did in response, etc.? 3. Are the Camel routes the only consumers on that topic? Tim On Tue, Sep 6, 2016 at 2:14 PM, ranjitnethi <[email protected]> wrote: > We are using ActiveMQ, using Network of Brokers in Hub-Spoke topology. > Active MQ is deployed in Karaf Container > > Software Stack Used > > Active MQ Version 5.10.0 > Container - Karaf 3.0.3 > Camel 2.14.1 > > Hub is having 2 brokers- Master Active MQ and Slave Active MQ, configured > using shared file system (GFS) and Kaha DB Data Store. Operate as failover. > Spokes or Site connect to Hub using network connectors configuration below- > > <networkConnectors> > <networkConnector > name="T:broker1->broker2" > > uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616 > ) > duplex="true" > decreaseNetworkConsumerPriority="true" > networkTTL="2" > dynamicOnly="true"> > </networkConnector> > > <networkConnector > name="Q:broker1->broker2" > > uri="masterslave:(tcp://masterbroker.com:61616,tcp://slavebroker.com:61616 > ) > duplex="true" > decreaseNetworkConsumerPriority="true" > networkTTL="2" > dynamicOnly="true"> > </networkConnector> > </networkConnectors> > > *Issue* > > > * > We have topic on HUB AMQ broker called Shop.Canonical.WorkOrder.Notify, > and > the producers are at the spoke level. The producers write to the local > topic > Shop.Canonical.WorkOrder.Notify (This is the same topic AMQ creates when > using network of brokers). > > We have Durable Topic Subscribers on Central (HUB ) broker, camel client > routes subscribed to this topic. > Apparently due to network connections lost or force shutdowns of servers > hosting these karaf containers at spoke level, we are ending up having > offline durable subscribers. > > It looks like we had network outage and network failover tried to stop the > network bridge and at the same time active mq tried to start another > connection causing deadlock. I see the following exception from log file. > Also the details of the threads causing deadlock towards the end of this > message. > * > > > javax.jms.InvalidClientIDException: Broker: taxxx-amq-broker - Client: > T:broker1->broker2_central-amq-broker_inbound_taxxx-amq-broker already > connected from vm://taxxx-amq-broker#4548 > at > org.apache.activemq.broker.region.RegionBroker.addConnection > (RegionBroker.java:246)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.jmx.ManagedRegionBroker.addConnec > tion(ManagedRegionBroker.java:231)[83:org.apache.activemq.ac > tivemq-osgi:5.10.0] > at > org.apache.activemq.broker.BrokerFilter.addConnection(Broker > Filter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.advisory.AdvisoryBroker.addConnection(Ad > visoryBroker.java:89)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.BrokerFilter.addConnection(Broker > Filter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.BrokerFilter.addConnection(Broker > Filter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.BrokerFilter.addConnection(Broker > Filter.java:92)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.security.JaasAuthenticationBroker.addCon > nection(JaasAuthenticationBroker.java:87)[83:org.apache. > activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.MutableBrokerFilter.addConnection > (MutableBrokerFilter.java:97)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.TransportConnection.processAddCon > nection(TransportConnection.java:764)[83:org.apache. > activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.command.ConnectionInfo.visit(ConnectionI > nfo.java:139)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.TransportConnection.service(Trans > portConnection.java:294)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:148)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.ResponseCorrelator.onCommand(R > esponseCorrelator.java:116)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTr > ansport.java:138)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTran > sport.java:130)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransp > ort.java:107)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:68)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.ResponseCorrelator.asyncReques > t(ResponseCorrelator.java:81)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.transport.ResponseCorrelator.request(Res > ponseCorrelator.java:86)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.network.DemandForwardingBridgeSupport.st > artLocalBridge(DemandForwardingBridgeSupport.java:453)[83: > org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.network.DemandForwardingBridgeSupport.do > StartLocalAndRemoteBridges(DemandForwardingBridgeSupport.jav > a:420)[83:org.apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.network.DemandForwardingBridgeSupport.ac > cess$500(DemandForwardingBridgeSupport.java:105)[83:org. > apache.activemq.activemq-osgi:5.10.0] > at > org.apache.activemq.network.DemandForwardingBridgeSupport$5. > run(DemandForwardingBridgeSupport.java:335)[83:org.apache.ac > tivemq.activemq-osgi:5.10.0] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1145)[:1.7.0_75] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:615)[:1.7.0_75] > at java.lang.Thread.run(Thread.java:745)[:1.7.0_75] > > > > > *Above scenario also creates a dead lock condition - 3 threads * > > *Thread 1 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-17676* > State: WAITING on > java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: > ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480 > Total blocked: 5 Total waited: 10 > > Stack trace: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn > dCheckInterrupt(AbstractQueuedSynchronizer.java:834) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > eQueued(AbstractQueuedSynchronizer.java:867) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > e(AbstractQueuedSynchronizer.java:1197) > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock( > ReentrantLock.java:214) > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:66) > org.apache.activemq.transport.ResponseCorrelator.oneway(Resp > onseCorrelator.java:60) > org.apache.activemq.network.DemandForwardingBridgeSupport.se > rviceLocalCommand(DemandForwardingBridgeSupport.java:970) > org.apache.activemq.network.DemandForwardingBridgeSupport$2. > onCommand(DemandForwardingBridgeSupport.java:206) > org.apache.activemq.transport.ResponseCorrelator.onCommand(R > esponseCorrelator.java:116) > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTr > ansport.java:138) > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTran > sport.java:130) > - locked java.util.concurrent.atomic.AtomicBoolean@1e5f8fb > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107) > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:68) > org.apache.activemq.transport.ResponseCorrelator.oneway(Resp > onseCorrelator.java:60) > org.apache.activemq.broker.TransportConnection.dispatch(Tran > sportConnection.java:1370) > org.apache.activemq.broker.TransportConnection.processDispat > ch(TransportConnection.java:889) > org.apache.activemq.broker.TransportConnection.iterate(Trans > portConnection.java:935) > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTa > skRunner.java:133) > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTask > Runner.java:48) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > * Thread 2 - > Name: ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480* > State: WAITING on > java.util.concurrent.locks.ReentrantLock$NonfairSync@1c83e84 owned by: > ActiveMQ BrokerService[taxxx-amq-broker] Task-17676 > Total blocked: 0 Total waited: 2 > > Stack trace: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn > dCheckInterrupt(AbstractQueuedSynchronizer.java:834) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > eQueued(AbstractQueuedSynchronizer.java:867) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > e(AbstractQueuedSynchronizer.java:1197) > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock( > ReentrantLock.java:214) > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:66) > org.apache.activemq.transport.ResponseCorrelator.oneway(Resp > onseCorrelator.java:60) > org.apache.activemq.broker.TransportConnection.dispatch(Tran > sportConnection.java:1370) > org.apache.activemq.broker.TransportConnection.processDispat > ch(TransportConnection.java:889) > org.apache.activemq.broker.TransportConnection.dispatchSync( > TransportConnection.java:849) > org.apache.activemq.broker.TransportConnection$1.onCommand( > TransportConnection.java:150) > org.apache.activemq.transport.ResponseCorrelator.onCommand(R > esponseCorrelator.java:116) > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > org.apache.activemq.transport.vm.VMTransport.doDispatch(VMTr > ansport.java:138) > org.apache.activemq.transport.vm.VMTransport.dispatch(VMTran > sport.java:130) > - locked java.util.concurrent.atomic.AtomicBoolean@12796af > org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:107) > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:68) > org.apache.activemq.transport.ResponseCorrelator.asyncReques > t(ResponseCorrelator.java:81) > org.apache.activemq.transport.ResponseCorrelator.request(Res > ponseCorrelator.java:86) > org.apache.activemq.network.DemandForwardingBridgeSupport.ad > dSubscription(DemandForwardingBridgeSupport.java:905) > org.apache.activemq.network.DemandForwardingBridgeSupport.ad > dConsumerInfo(DemandForwardingBridgeSupport.java:1178) > org.apache.activemq.network.DemandForwardingBridgeSupport.se > rviceRemoteConsumerAdvisory(DemandForwardingBridgeSupport.java:763) > - locked java.net.URI@15e6d6 > org.apache.activemq.network.DemandForwardingBridgeSupport.se > rviceRemoteCommand(DemandForwardingBridgeSupport.java:614) > org.apache.activemq.network.DemandForwardingBridgeSupport$3. > onCommand(DemandForwardingBridgeSupport.java:224) > org.apache.activemq.transport.ResponseCorrelator.onCommand(R > esponseCorrelator.java:116) > org.apache.activemq.transport.MutexTransport.onCommand(Mutex > Transport.java:50) > org.apache.activemq.transport.failover.FailoverTransport$3.o > nCommand(FailoverTransport.java:208) > org.apache.activemq.transport.WireFormatNegotiator.onCommand > (WireFormatNegotiator.java:113) > org.apache.activemq.transport.AbstractInactivityMonitor.onCo > mmand(AbstractInactivityMonitor.java:270) > org.apache.activemq.transport.TransportSupport.doConsume(Tra > nsportSupport.java:83) > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTran > sport.java:214) > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > java.lang.Thread.run(Thread.java:745) > > > > *Thread 3 - Name: ActiveMQ BrokerService[taxxx-amq-broker] Task-18436* > State: WAITING on > java.util.concurrent.locks.ReentrantLock$NonfairSync@138f39e owned by: > ActiveMQ Transport: tcp://shopesb1-prd-sl/10.98.1.92:61616@54480 > Total blocked: 0 Total waited: 44 > > Stack trace: > sun.misc.Unsafe.park(Native Method) > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn > dCheckInterrupt(AbstractQueuedSynchronizer.java:834) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > eQueued(AbstractQueuedSynchronizer.java:867) > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir > e(AbstractQueuedSynchronizer.java:1197) > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock( > ReentrantLock.java:214) > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > org.apache.activemq.transport.MutexTransport.oneway(MutexTra > nsport.java:66) > org.apache.activemq.transport.ResponseCorrelator.oneway(Resp > onseCorrelator.java:60) > org.apache.activemq.network.DemandForwardingBridgeSupport$4. > run(DemandForwardingBridgeSupport.java:288) > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool > Executor.java:1145) > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo > lExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > > > > -- > View this message in context: http://activemq.2283324.n4.nab > ble.com/Durable-Subscribers-going-offline-Deadlock- > Network-Of-Brokers-tp4716206.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >
