Client deadlock during failover ------------------------------- Key: AMQ-1093 URL: https://issues.apache.org/activemq/browse/AMQ-1093 Project: ActiveMQ Issue Type: Bug Components: Transport Affects Versions: 4.1.0 Environment: Linux 2.6, java 1.5.0 Reporter: Danielius Jurna Priority: Critical
In 4.1.0 there's deadlock on connection failover. There is the scenario: 1. Client consumes message using message listener 2. Conection is lost 3. Client sends message to another queue from messagle listener and waits until connection is restored. 4. Reconnect task blocks on reconnecting. This bug is new to 4.1.0. The problem is in ActiveMQMessageConsumre.dispatch . There is new lock on unconsumedMessages.getMutex() . So the reconnect task cannot invoke ActiveMQMessageConsumre.clearMessagesInProgress(), because lock is acquired by message listener, which waits untill message is sent (untill connection is resumed). Here is stack traces: "ActiveMQ Session Task" daemon prio=1 tid=0x0000002b27774260 nid=0x4778 in Object.wait() [0x0000000040ef3000..0x0000000040ef4db0] at java.lang.Object.wait(Native Method) - waiting on <0x0000002b0020a7c8> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at java.lang.Object.wait(Object.java:474) at edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar.await(CondVar.java:75) - locked <0x0000002b0020a7c8> (a edu.emory.mathcs.backport.java.util.concurrent.locks.CondVar) at edu.emory.mathcs.backport.java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:318) at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:42) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:75) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171) at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1548) at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:465) at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:75) - locked <0x0000002b173fa480> (a org.apache.activemq.ActiveMQMessageProducer) at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:60) at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:537) at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) at lt.elitnet.dbp.das.impl.storage.HI2StorageImpl.storeHI2Message(HI2StorageImpl.java:57) at lt.elitnet.dbp.das.impl.hi2.HI2PersistanceBase.saveIRIContent(HI2PersistanceBase.java:77) at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:203) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:162) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:104) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at lt.elitnet.dbp.das.impl.alarming.DataBaseConnectionAlarmsPublisher.invoke(DataBaseConnectionAlarmsPublisher.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:209) at $Proxy24.saveIRIContent(Unknown Source) at lt.elitnet.dbp.das.impl.hi2.HI2MessageListener.onMessage(HI2MessageListener.java:40) at lt.elitnet.dbp.commons.messaging.TextMessageListener.deliverMessage(TextMessageListener.java:84) at lt.elitnet.dbp.commons.messaging.TextMessageListener.onMessage(TextMessageListener.java:50) at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:854) at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:793) at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:763) at org.springframework.jms.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:743) at org.springframework.jms.listener.SimpleMessageListenerContainer$2.onMessage(SimpleMessageListenerContainer.java:190) at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:840) - locked <0x0000002b1771ddc0> (a java.lang.Object) at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:96) at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:165) at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111) at org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26) at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) "ActiveMQ Transport: tcp:///192.168.2.55:61616" prio=1 tid=0x0000002b2e814660 nid=0x4773 waiting for monitor entry [0x0000000040e82000..0x0000000040e82c30] at org.apache.activemq.MessageDispatchChannel.clear(MessageDispatchChannel.java:135) - waiting to lock <0x0000002b1771ddc0> (a java.lang.Object) at org.apache.activemq.ActiveMQMessageConsumer.clearMessagesInProgress(ActiveMQMessageConsumer.java:546) at org.apache.activemq.ActiveMQSession.clearMessagesInProgress(ActiveMQSession.java:524) at org.apache.activemq.ActiveMQConnection.transportInterupted(ActiveMQConnection.java:1547) at org.apache.activemq.transport.TransportFilter.transportInterupted(TransportFilter.java:100) at org.apache.activemq.transport.TransportFilter.transportInterupted(TransportFilter.java:100) at org.apache.activemq.transport.failover.FailoverTransport.handleTransportFailure(FailoverTransport.java:236) at org.apache.activemq.transport.failover.FailoverTransport.access$300(FailoverTransport.java:55) at org.apache.activemq.transport.failover.FailoverTransport$1.onException(FailoverTransport.java:120) at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:96) at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:147) at org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:150) at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:97) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:150) at java.lang.Thread.run(Thread.java:595) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/activemq/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira