[
https://issues.apache.org/jira/browse/AMQ-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035968#comment-15035968
]
Dumitru HUSLEAG edited comment on AMQ-6062 at 12/2/15 3:52 PM:
---------------------------------------------------------------
I modified again the test to increase the number of browse tasks and the
timeout because on command line it is faster.
And yes I reproduced the bug wit HotSpot java and I attached the thread dump.
{code}
$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
{code}
was (Author: doomytroo):
I modified again the test to increase the number of browse tasks.
And yes I reproduced the bug wit HotSpot java and I attached the thread dump.
{code}
$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
{code}
> Broker goes 100% CPU on multi-queue command line browse
> -------------------------------------------------------
>
> Key: AMQ-6062
> URL: https://issues.apache.org/jira/browse/AMQ-6062
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.1, 5.12.0
> Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either
> IBM or Oracle)
> Reporter: Dumitru HUSLEAG
> Assignee: Christopher L. Shannon
> Attachments: AMQ6062Test.java,
> ActiveMQ_BrowseInfiniteLoopUseCase.jmx, HotSpotThreadDump.txt,
> IBMjava7AndAMQMaster.javacore.20151202.161643.18441.0001.txt,
> ThreadStatusAnalysis_2015-11-25_14h13m33s.png, activemq-test.log
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hello,
> The Broker may enter a state of high CPU consumption (100% or 200% if 2
> browse enter the infinite loop in the same time on a multi-core) when a
> command line browse on multiple queues is issued. When that arrives the only
> way we found to restore the situation was to kill the broker and restart
> because he was not responding anymore to clients. Killing the clients and the
> browse process does not lower the CPU usage of the broker.
> I assume the infinite loop based on successive thread dumps that show the
> browse thread remaines in the same code zone:
> Here is an example of the Browse thread call stack trace, when Broker is in
> high CPU usage state:
> {quote}
> at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
> at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
> at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
> at java/util/HashMap.put(HashMap.java:680(Compiled Code))
> at
> org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
> Code))
> at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled
> Code))
> at
> org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
> Code)).
> {quote}
> Tested ActiveMQ versions: 5.9.1 and the last, 5.12.0, but of course I suppose
> all in beetwen are affected.
> The cause seems to be the non-threadsafe usage of the *audit HashMap* member
> of *QueueBrowserSubscription.java* class.
> Here are some facts on non thread-safe HashMap use problems:
> *High CPU / Hang on java.util.HashMap.findNonNullKeyEntry() due to
> non-thread-safe usage of HashMap*
> http://www-01.ibm.com/support/docview.wss?uid=swg21597581
> *IZ73767: INFINITE LOOP CAN OCCUR IN HASHMAP*
> http://www-01.ibm.com/support/docview.wss?uid=swg1IZ73767
> I must underline that *this is not an IBM JDK specific problem as the IBM JDK
> is based on Oracle JDK* and changing the JDK or JRE is not a solution.
> The scenario may be produced with a fresh non modified install (default
> config) of ActiveMQ broker:
> 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right
> activemq, and Java version.
> 2. cd $ACTIVEMQ_HOME/bin; ./activemq start
> 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
> 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID
> --amqurl tcp://localhost:61616 QA.*
> The simplest fix that I suggest is to change in the
> *QueueBrowserSubscription.java* class the type of *audit* member to
> *ConcurrentHashMap*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
