[
https://issues.apache.org/jira/browse/AMQ-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dumitru HUSLEAG updated AMQ-6062:
---------------------------------
Description:
Hello,
The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse
enter the infinite loop in the same time on a multi-core) when a command line
brose on multiple ques is issued. When that arives the only way we found to
restore the situation was to kill the broker and restart because he was not
responding anymore to clients. Killing the clients and the browse process does
not lower the CPU usage of the broker.
I assume the infinite loop based on successive thread dumps that show the
browse thread remaines in the same code zone:
Here is the Browse thread call stack trace:
{quote}
at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
at java/util/HashMap.put(HashMap.java:680(Compiled Code))
at
org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
Code))
at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled
Code))
at
org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
Code)).
{quote}
Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in
beetwen are affected.
The cause seems to be the non-threadsafe usage of the *audit HashMap* member of
*QueueBrowserSubscription.java* class.
The scenario may be produced with a fresh non modified install (default config)
of ActiveMQ broker:
1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right
activemq, and Java version.
2. cd $ACTIVEMQ_HOME/bin; ./activemq start
3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID
--amqurl tcp://localhost:61616 QA.*
Another IBM Link on *HashMap problem*:
http://www-01.ibm.com/support/docview.wss?uid=swg21597581
The simplest fix that I suggest is to change in the
*QueueBrowserSubscription.java* class the type of *audit* member to
*ConcurrentHashMap*.
was:
Hello,
The Broker may enter a state of high CPU consumption (100% or 200% if 2 browse
enter the infinite loop in the same time on a multi-core) when a command line
brose on multiple ques is issued. When that arives the only way we found to
restore the situation was to kill the broker and restart because he was not
responding anymore to clients. Killing the clients and the browse process does
not lower the CPU usage of the broker.
I assume the infinite loop based on successive thread dumps that show the
browse thread remaines in the same code zone:
Here is the Browse thread call stack trace:
{quote}
at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
at java/util/HashMap.put(HashMap.java:680(Compiled Code))
at
org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
Code))
at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled
Code))
at
org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
Code)).
{quote}
Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in
beetwen are affected.
The cause seems to be the non-threadsafe usage of the *audit HashMap* member of
*QueueBrowserSubscription.java* class.
The scenario may be produced with a fresh non modified install (default config)
of ActiveMQ broker:
1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right
activemq, and Java version.
2. cd $ACTIVEMQ_HOME/bin; ./activemq start
3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID
--amqurl tcp://localhost:61616 QA.*
Another IBM Link on HashMap problem:
http://www-01.ibm.com/support/docview.wss?uid=swg21597581
The simplest fix that I suggest is to change the type of *audit* member to
*ConcurrentHashMap*.
> Broker goes 100% CPU on multi-queue command line browse
> -------------------------------------------------------
>
> Key: AMQ-6062
> URL: https://issues.apache.org/jira/browse/AMQ-6062
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.1, 5.12.0
> Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either
> IBM or Oracle)
> Reporter: Dumitru HUSLEAG
> Priority: Critical
> Attachments: ActiveMQ_BrowseInfiniteLoopUseCase.jmx,
> ThreadStatusAnalysis_2015-11-25_14h13m33s.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hello,
> The Broker may enter a state of high CPU consumption (100% or 200% if 2
> browse enter the infinite loop in the same time on a multi-core) when a
> command line brose on multiple ques is issued. When that arives the only way
> we found to restore the situation was to kill the broker and restart because
> he was not responding anymore to clients. Killing the clients and the browse
> process does not lower the CPU usage of the broker.
> I assume the infinite loop based on successive thread dumps that show the
> browse thread remaines in the same code zone:
> Here is the Browse thread call stack trace:
> {quote}
> at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
> at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
> at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
> at java/util/HashMap.put(HashMap.java:680(Compiled Code))
> at
> org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
> Code))
> at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled
> Code))
> at
> org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
> Code)).
> {quote}
> Tested versions: 5.9.1 and the last, 5.12.0, but of course I suppose all in
> beetwen are affected.
> The cause seems to be the non-threadsafe usage of the *audit HashMap* member
> of *QueueBrowserSubscription.java* class.
> The scenario may be produced with a fresh non modified install (default
> config) of ActiveMQ broker:
> 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right
> activemq, and Java version.
> 2. cd $ACTIVEMQ_HOME/bin; ./activemq start
> 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
> 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID
> --amqurl tcp://localhost:61616 QA.*
> Another IBM Link on *HashMap problem*:
> http://www-01.ibm.com/support/docview.wss?uid=swg21597581
> The simplest fix that I suggest is to change in the
> *QueueBrowserSubscription.java* class the type of *audit* member to
> *ConcurrentHashMap*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
