Hi all,
We are currently testing failover for our application in Jboss 7.2 with
HornetQ cluster active/passive in XA transacted mode. It seems like failover
works at random, but often we get the following error on the client side and
the one below on the server side. It seems to be stable when running
transacted mode without XA. We are using Camel/Spring/activemq-jms-pool
5.9.0/XaPooledConnectionFactory. Has anyone successfully run this
configuration with failover? Any help would be greatly appreciated!
My interpretation of what happens is this.
1. Active and Passive queue servers are running
2. We shut down the active queue server.
3. Our application tries to continue and is given a session by activemq
pool.
4. The queue server recognizes the session as already being in use by
another transaction and throws an exception.
However - the session/transaction never seem to time out on the queue server
and is also never discarded on the activemq session pool?
Please...any help will be greatly appreciated!
Cheers,
Jesper
Application server exception
13:50:48,113 ERROR [org.hornetq.core.client] (Camel (camelContext) thread
#21 - JmsConsumer[asynchronousPaymentProcessingQueue]) HQ214007: XA start
operation failed Cannot start, session is already doing work in
a transaction XidImpl (574641522
bq:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.113.-113.22.0.0.126.113.0.0.0.0.0.0.0.0
formatID:131077
gtxid:0.0.0.0.0.0.0.0.0.0.-1.-1.10.20.103.76.-72.-12.91.64.83.11
3.-113.22.0.0.126.111.49
base64:AAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-cQAAAAAAAAAAAAAAAAAAAAAAAP__ChRnTLj0W0BTcY8WAAB-bzEHAgIA
code:-6
13:50:48,113 WARN [com.arjuna.ats.jta] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) ARJUNA016061:
TransactionImple.enlistResource - XAResource.start returned: XAException.X
AER_PROTO for < formatId=131077, gtrid_length=29, bqual_length=36,
tx_uid=0:ffff0a14674c:-470ba4c0:53718f16:4316e, node_name=1,
branch_uid=0:ffff0a14674c:-470ba4c0:53718f16:43170,
subordinatenodename=null, eis_na
me=unknown eis name >: javax.transaction.xa.XAException
at
org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775)
[hornetq-core-client.jar:]
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:636)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:397)
at
org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70)
[activemq-jms-pool.jar:5.9.0]
at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
[activemq-jms-pool.jar:5.9.0]
at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(AbstractPollingMessageListenerContainer.java:77)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerContainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
[spring-jms.jar:3.2.2.RE
LEASE]
at
org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(ConnectionFactoryUtils.java:300)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:288)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1096)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
[spring-jms.jar:3.2.2.RELEASE]
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:985)
[spring-jms.jar:3.2.2.RELEASE]
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
[rt.jar:1.6.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
[rt.jar:1.6.0_45]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_45]
13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue])
javax.transaction.xa.XAException
13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1775)
13:50:48,114 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
ctionImple.java:636)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(Transa
ctionImple.java:397)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.apache.activemq.jms.pool.XaConnectionPool.createSession(XaConnectionPool.java:70)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.apache.activemq.jms.pool.PooledConnection.createSession(PooledConnection.java:167)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.access$200(Abstr
actPollingMessageListenerContainer.java:77)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer$MessageListenerC
ontainerResourceFactory.createSession(AbstractPollingMessageListenerContainer.java:531)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.connection.ConnectionFactoryUtils.doGetTransactionalSession(Conne
ctionFactoryUtils.java:300)
13:50:48,115 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExec
ute(AbstractPollingMessageListenerContainer.java:288)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecut
e(AbstractPollingMessageListenerContainer.java:243)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.invokeListener(DefaultMessageListenerContainer.java:1096)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.executeOngoingLoop(DefaultMessageListenerContainer.java:1088)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvo
ker.run(DefaultMessageListenerContainer.java:985)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
13:50:48,116 ERROR [stderr] (Camel (camelContext) thread #21 -
JmsConsumer[asynchronousPaymentProcessingQueue]) at
java.lang.Thread.run(Thread.java:662)
13:50:50,165 WARN
[org.apache.camel.component.jms.DefaultJmsMessageListenerContainer] (Camel
(camelContext) thread #20 - JmsConsumer[asynchronousPaymentProcessingQueue])
Setup of JMS message listener invoker failed for destination
'asynchronousPaymentProcessingQueue' - trying to recover. Cause: JTA
transaction unexpectedly rolled back (maybe due to a timeout); nested
exception is javax.transaction.RollbackException: ARJUNA016053: Could not
commit transaction.
Queue server exception
13:21:38,295 ERROR [org.hornetq.core.server] (Old I/O server worker
(parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224029:
Caught XA exception: org.hornetq.core.exception.HornetQXAException: Invalid
transaction state
at
org.hornetq.core.server.impl.ServerSessionImpl.individualAcknowledge(ServerSessionImpl.java:634)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:415)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281)
[netty-3.6.2.Final.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
[netty-3.6.2.Final.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[rt.jar:1.6.0_30]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.6.0_30]
at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]
13:21:38,354 ERROR [org.hornetq.core.server] (Old I/O server worker
(parentId: -1887525330, [id: 0x8f7ea62e, /10.20.103.88:5595])) HQ224030:
Caught exception: HornetQException[errorType=ILLEGAL_STATE message=HQ119057:
Could not find reference on consumer ID=67, messageId = 51,567,696,761 queue
= jms.queue.asynchronousPaymentProcessingQueue]
at
org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:645)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:624)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:562)
[hornetq-server-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281)
[netty-3.6.2.Final.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
[hornetq-core-client-2.3.0.CR1.jar:]
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[netty-3.6.2.Final.jar:]
at
org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
[netty-3.6.2.Final.jar:]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[rt.jar:1.6.0_30]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[rt.jar:1.6.0_30]
at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_30]
--
View this message in context:
http://activemq.2283324.n4.nabble.com/failover-session-pool-tp4681094.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
