[
https://issues.apache.org/jira/browse/AMQ-6062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035801#comment-15035801
]
Dumitru HUSLEAG edited comment on AMQ-6062 at 12/2/15 1:59 PM:
---------------------------------------------------------------
JUnit test that produces the bug and a thread dump when it occurs.
Just put it in
/activemq-parent-5.9.1/activemq-unit-tests/src/test/java/org/apache/activemq/console/command/
and test it.
It works for me with 5.9.1 tag sources and IBM java6 on Ubuntu 12.04 LTS.
The test is a bit complicated because I tried to make it fail() and fail fast,
do a thread dump and not block or kill the JVM without a JUnit failure.
I run the test from Eclipse. I just came to realise you might test from command
line or other ways (in Jenkins, etc) and I did a test on command line and JUnit
does not fail because the kill is faster and maven test is in success in spite
of the bug detection.
Anyway the thread dump is generated to prove the bug.
was (Author: doomytroo):
JUnit test that produces the bug and a thread dump when it occurs.
Just put it in
/activemq-parent-5.9.1/activemq-unit-tests/src/test/java/org/apache/activemq/console/command/
and test it.
It works for me with 5.9.1 tag sources and IBM java6 on Ubuntu 12.04 LTS.
The test is a bit complicated because I tried to make it fail() and fail fast,
do a thread dump and not block or kill the JVM without a JUnit failure.
I run the test from Eclipse. I just came to realise you might test from command
line or othe ways (in Jenkins, etc) and I did a test on command line and JUnit
does not fail because the kill is faster and maven test is in success in spite
of the bug detection.
Anyway the thread dump is generated to prove the bug.
> Broker goes 100% CPU on multi-queue command line browse
> -------------------------------------------------------
>
> Key: AMQ-6062
> URL: https://issues.apache.org/jira/browse/AMQ-6062
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.9.1, 5.12.0
> Environment: Linux Ubuntu, Aix, with both Java 6 and Java 7 (either
> IBM or Oracle)
> Reporter: Dumitru HUSLEAG
> Attachments: AMQ6062Test.java,
> ActiveMQ_BrowseInfiniteLoopUseCase.jmx,
> ThreadStatusAnalysis_2015-11-25_14h13m33s.png
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hello,
> The Broker may enter a state of high CPU consumption (100% or 200% if 2
> browse enter the infinite loop in the same time on a multi-core) when a
> command line browse on multiple queues is issued. When that arrives the only
> way we found to restore the situation was to kill the broker and restart
> because he was not responding anymore to clients. Killing the clients and the
> browse process does not lower the CPU usage of the broker.
> I assume the infinite loop based on successive thread dumps that show the
> browse thread remaines in the same code zone:
> Here is an example of the Browse thread call stack trace, when Broker is in
> high CPU usage state:
> {quote}
> at java/util/HashMap.rehash(HashMap.java:782(Compiled Code))
> at java/util/HashMap.rehash(HashMap.java:819(Compiled Code))
> at java/util/HashMap.putImpl(HashMap.java:702(Compiled Code))
> at java/util/HashMap.put(HashMap.java:680(Compiled Code))
> at
> org/apache/activemq/broker/region/*QueueBrowserSubscription.isDuplicate(QueueBrowserSubscription.java:72*(Compiled
> Code))
> at org/apache/activemq/broker/region/*Queue.iterate(Queue.java:1688*(Compiled
> Code))
> at
> org/apache/activemq/thread/PooledTaskRunner.runTask(PooledTaskRunner.java:133(Compiled
> Code)).
> {quote}
> Tested ActiveMQ versions: 5.9.1 and the last, 5.12.0, but of course I suppose
> all in beetwen are affected.
> The cause seems to be the non-threadsafe usage of the *audit HashMap* member
> of *QueueBrowserSubscription.java* class.
> Here are some facts on non thread-safe HashMap use problems:
> *High CPU / Hang on java.util.HashMap.findNonNullKeyEntry() due to
> non-thread-safe usage of HashMap*
> http://www-01.ibm.com/support/docview.wss?uid=swg21597581
> *IZ73767: INFINITE LOOP CAN OCCUR IN HASHMAP*
> http://www-01.ibm.com/support/docview.wss?uid=swg1IZ73767
> I must underline that *this is not an IBM JDK specific problem as the IBM JDK
> is based on Oracle JDK* and changing the JDK or JRE is not a solution.
> The scenario may be produced with a fresh non modified install (default
> config) of ActiveMQ broker:
> 1. set ACTIVEMQ_HOME, JAVA_HOME and PATH to point accordingly to the right
> activemq, and Java version.
> 2. cd $ACTIVEMQ_HOME/bin; ./activemq start
> 3. inject at least 2000 messages in each of two ques, say QA.ONE and QA.TWO
> 4. $ $ACTIVEMQ_HOME/bin/activemq browse --view JMSDestination,JMSMessageID
> --amqurl tcp://localhost:61616 QA.*
> The simplest fix that I suggest is to change in the
> *QueueBrowserSubscription.java* class the type of *audit* member to
> *ConcurrentHashMap*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
