I have a problem whereby the ActiveMQ broker crashes when my system has been
running for more than ~6 hours. By "crashes" I mean it's still running but
won't allow any sending/receiving of messages and needs to be restarted.
Here's my setup:
ActiveMQ 5.5.0 with the out-of-the-box configuration, except after it
crashed the first few times I set Xmx to 2048M and
UseDedicatedTaskRunner=false, not really sure why though.
Has around 200 (non-transacted) connections from locally networked PCs but
very minimal traffic - around two messages per second.
Runs on Ubuntu 9.04, using:
java version "1.6.0_16"
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
My data/activemq.log shows the following when it crashes:
2011-07-14 03:57:54,605 | INFO | Transport failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
rt: tcp:///146.169.52.132:53248
2011-07-14 03:58:00,925 | INFO | Transport failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transpo
rt: tcp:///146.169.52.133:50171
2011-07-14 03:58:03,795 | INFO | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@4dcd6d55
2011-07-14 03:58:07,275 | INFO | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
Async Exception Handler
2011-07-14 03:58:07,274 | INFO | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@5ff907ee
2011-07-14 03:58:03,802 | INFO | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@3bd1c463
2011-07-14 03:58:07,274 | INFO | Transport failed:
java.net.SocketException: Broken pipe |
org.apache.activemq.broker.TransportConnection.Transport |
InactivityMonitor Async Task:
java.util.concurrent.ThreadPoolExecutor$Worker@3a15e82a
etc. etc. lots more Broken pipes... then some of these interspersed with
many more broken pipes:
2011-07-14 03:58:10,201 | INFO | Transport failed:
org.apache.activemq.transport.InactivityIOException: Cannot send, channel
has already failed: /146
.169.53.87:46663 | org.apache.activemq.broker.TransportConnection.Transport
| Async Exception Handler
Then some of these (still interspersed with broken pipes and failed channel
messages):
2011-07-14 04:18:50,633 | WARN | Failed to unregister mbean:
org.apache.activemq:BrokerName=localhost,Type=Connection,ConnectorName=openwire,ViewType
=address,Name=/146.169.52.126_33912 |
org.apache.activemq.broker.jmx.ManagedTransportConnection | ActiveMQ
Task-103
Then the end is nigh with a few of these:
2011-07-14 04:19:54,077 | WARN | Failed to remove consumer: ID:"I removed
the address here"-60248-1310591381109-0:1:-1:1. Reason:
java.lang.OutOfMemoryError
: Java heap space | org.apache.activemq.broker.TransportConnection |
ActiveMQ Task-69
java.lang.OutOfMemoryError: Java heap space
at
java.util.concurrent.CopyOnWriteArrayList.remove(CopyOnWriteArrayList.java:481)
at
java.util.concurrent.CopyOnWriteArraySet.remove(CopyOnWriteArraySet.java:205)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.unregisterSubscription(ManagedRegionBroker.java:381)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.removeConsumer(ManagedRegionBroker.java:255)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
at
org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:225)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
at
org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:117)
at
org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123)
at
org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:575)
at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:613)
at
org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:726)
at
org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1025)
at
org.apache.activemq.broker.jmx.ManagedTransportConnection.doStop(ManagedTransportConnection.java:71)
at
org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:952)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2011-07-14 04:19:54,077 | WARN | Problem scavenging sessions |
org.eclipse.jetty.util.log | HashSessionScavenger-1
java.lang.OutOfMemoryError: Java heap space
After the broker "crashes" my clients dump the following when attempting to
create a temporary queue (though "createTemporaryQueue" is just the first
method of the local "Session" that they call):
Exception in thread "Thread-4" javax.jms.JMSException: Timer already
cancelled.
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1295)
at
org.apache.activemq.ActiveMQConnection.createTempDestination(ActiveMQConnection.java:1930)
at
org.apache.activemq.ActiveMQSession.createTemporaryQueue(ActiveMQSession.java:1330)
at project.hub.Client.receive(Client.scala:54)
at project.Node$.colony(Node.scala:52)
at
project.Node$$anonfun$main$1$$anonfun$apply$1.apply$mcV$sp(Node.scala:69)
at project.util.Ops$$anon$1.run(Ops.scala:5)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:354)
at java.util.Timer.schedule(Timer.java:222)
at
org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:49)
at org.apache.activemq.broker.region.Queue.start(Queue.java:852)
at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:129)
at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:304)
at
org.apache.activemq.broker.region.RegionBroker.addDestinationInfo(RegionBroker.java:367)
at
org.apache.activemq.advisory.AdvisoryBroker.addDestinationInfo(AdvisoryBroker.java:160)
at
org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
at
org.apache.activemq.broker.BrokerFilter.addDestinationInfo(BrokerFilter.java:217)
at
org.apache.activemq.broker.MutableBrokerFilter.addDestinationInfo(MutableBrokerFilter.java:223)
at
org.apache.activemq.broker.TransportConnection.processAddDestination(TransportConnection.java:480)
at
org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:122)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
at java.lang.Thread.run(Thread.java:619)
I also get the following from a different client thread that is blocked on
"MessageConsumer.receive":
Exception in thread "Thread-10" javax.jms.JMSException: Channel was inactive
for too (>30000) long: server/146.169.2.83:61678
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at
org.apache.activemq.ActiveMQMessageConsumer.dequeue(ActiveMQMessageConsumer.java:458)
at
org.apache.activemq.ActiveMQMessageConsumer.receive(ActiveMQMessageConsumer.java:504)
at project.hub.Client$$anonfun$2.apply$mcV$sp(Client.scala:32)
at project.util.Ops$$anon$1.run(Ops.scala:5)
at java.lang.Thread.run(Thread.java:636)
Another type of broker client I have dumps these:
Exception in thread "Thread-10" javax.jms.JMSException: Cannot send, channel
has already failed: server/146.169.2.83:61678
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:62)
at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1267)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1259)
at
org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1863)
at
org.apache.activemq.ActiveMQMessageProducer.close(ActiveMQMessageProducer.java:147)
at project.hub.Server$.theoryProvider(Server.scala:114)
at
project.hub.Server$$anonfun$main$2$$anonfun$apply$1.apply$mcV$sp(Server.scala:176)
at project.util.Ops$$anon$1.run(Ops.scala:5)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send,
channel has already failed: server/146.169.2.83:61678
at
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:246)
at
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1265)
... 7 more
Any help would be greatly appreciated.
Will
--
View this message in context:
http://activemq.2283324.n4.nabble.com/ActiveMQ-5-5-0-broker-crashing-tp3667090p3667090.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
