Do you see the same performance impact from attaching JConsole or JVisualVM? On Nov 10, 2015 4:09 PM, "Basmajian, Raffi" <rbasmaj...@ofiglobal.com> wrote:
> I'm throwing a hail mary on this one, > > We've set up a broker cluster on A-MQ 5.11 (Fuse 6.2). > Six master/slave pairs, full graph topology network of brokers; 12 brokers > total. > The cluster is brand new, no message activity; network connectors are > active and working properly. > Java 8, RHEL 7.1, 1gb/4Gb min/max > > Problem > ======= > Using JMC (Mission Control) , we connect to a master to view JMX metrics. > Almost immediately, cpu activity on the broker shoots to 100%. Heap > consumption oscillates between 128Mb and 800Mb over 30s windows, and the > rest of the day is generally unpleasant. > > From a thread view, nearly all cpu activity is consumed by numerous > "ActiveMQ Task-xxx" threads; here's the stack trace from one, but they are > all nearly identical: > > ActiveMQ Task-220 [2656] (TIMED_WAITING) > sun.misc.Unsafe.park line: not available [native method] > java.util.concurrent.locks.LockSupport.parkNanos line: 215 > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill line: > 460 > java.util.concurrent.SynchronousQueue$TransferStack.transfer line: 362 > java.util.concurrent.SynchronousQueue.poll line: 941 > java.util.concurrent.ThreadPoolExecutor.getTask line: 1066 > java.util.concurrent.ThreadPoolExecutor.runWorker line: 1127 > java.util.concurrent.ThreadPoolExecutor$Worker.run line: 617 > java.lang.Thread.run line: 745 > > At first I thought it was related to this issue, and while I found five > threads with the name "JMX Server connection timeout" on this instance, > none of those threads are consuming cpu resources. > https://access.redhat.com/solutions/1169753 > > When we test this on a standalone instance with no network configuration, > this problem does not occur. In our QA environment (the config detailed > above), the broker config is identical to our standalone instances, the > only difference is the network connector. Here's a snippet, there are five > total pairs like this (for six total m/s pairs in the topology) in each > activemq.xml: > > <!-- Network #1 --> > <!-- (#1) Queue network link --> > <networkConnector > name="queues_nc1" > userName="${auth.user}" > password="${auth.password}" > uri="masterslave(tcp://whatever, tcp://whatever)" > consumerTTL="1" > messageTTL="100" > conduitSubscriptions="false" > decreaseNetworkConsumerPriority="true" > suppressDuplicateQueueSubscriptions="true"> > <dynamicallyIncludedDestinations> > <queue physicalName=">"/> > </dynamicallyIncludedDestinations> > </networkConnector> > <!-- (#1) Topic network link --> > <networkConnector > name="topics_nc1 " > userName="${auth.user}" > password="${auth.password}" > uri="masterslave(tcp://whatever, tcp://whatever)" > consumerTTL="1" > messageTTL="100 " > decreaseNetworkConsumerPriority="true"> > <dynamicallyIncludedDestinations> > <topic physicalName=">"/> > </dynamicallyIncludedDestinations> > </networkConnector> > > > > This e-mail transmission may contain information that is proprietary, > privileged and/or confidential and is intended exclusively for the > person(s) to whom it is addressed. Any use, copying, retention or > disclosure by any person other than the intended recipient or the intended > recipient's designees is strictly prohibited. If you are not the intended > recipient or their designee, please notify the sender immediately by return > e-mail and delete all copies. OppenheimerFunds may, at its sole discretion, > monitor, review, retain and/or disclose the content of all email > communications. >
