I'm experiencing an issue running ActiveMQ 5.2.0 where it unpredictably stops
processing messages after running for a few days (up to a week or two). A
stack trace shows up in activemq.log like so:
ERROR Service - Async error occurred:
javax.jms.ResourceAllocationException: SystemUsage memory limit reached
javax.jms.ResourceAllocationException: SystemUsage memory limit reached
at org.apache.activemq.broker.region.Queue.send(Queue.java:344)
at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
at java.lang.Thread.run(Thread.java:619)
Client producers and receivers output no errors and seem to continue sending
but no messages arrive at the receivers. Our config file is here:
http://www.nabble.com/file/p23384456/activemq.xml activemq.xml . Producer
flow control is turned off as is all persistence. When we send messages we
set a TTL of 5 seconds on the Producers. The volume of messages is a few
hundred a second at peak.
I managed to run jstack on the ActiveMQ process and got this output:
http://www.nabble.com/file/p23384456/jstack_oom.txt jstack_oom.txt . I run
jstack every day and a noticeable difference between its "usual" output and
when the server has gone into this state is the *much* higher number of
"ActiveMQ Connection Dispatcher" and "ActiveMQ Transport" threads. Surely if
the TTL expires these threads should disappear?
I also run jmap on the ActiveMQ process and got this output:
http://www.nabble.com/file/p23384456/jmap_oom.txt jmap_oom.txt . Note the
100% usage of the JVM "From Space". For completeness I also generated a jmap
with a histo: http://www.nabble.com/file/p23384456/jmaphisto_oom.txt
jmaphisto_oom.txt .
When ActiveMQ goes into this state the only solution is to restart it.
Anybody have an idea what is going wrong? Is there something wrong in our
config? Is there some other way of preventing this from happening?
--
View this message in context:
http://www.nabble.com/SystemUsage-memory-limit-reached-tp23384456p23384456.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
