Thanks for responding, Tim. We only have about 50 destinations with almost no churn. The CPU at the time was minimal. Load was less than .5 on a 4 core system. RAM usage was less than 1G on 4G allocated to JVM heap.
Maybe a good place to start is understanding what the correct troubleshooting identification steps are when a broker misbehaves. I learned after that I could reload log4j via JMX. Is enabling DEBUG pretty much the only way to understand what is happening? Thanks On Sep 14, 2016, at 10:11 PM, Tim Bain <tb...@alumni.duke.edu<mailto:tb...@alumni.duke.edu>> wrote: <policyEntry queue=">" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" queuePrefetch="20"/> What's your volume of destinations, and how much churn do you have? Another user (Kevin Burton) experienced inefficiency in the destination GC algorithm with large numbers of short-lived destinations; if that sounds like your situation, I think he had some changes that never made it to trunk, which it might be possible to pass to you to rebase to 5.14.0 and then try as a patch. Overall, what does the process's performance look like? High CPU? Lots of memory? (In swap?) Lots of network IO? Doing lots of JVM full GCs (not destination GCs, but the actual ones from the JVM)? Knowing what the immediate symptoms of the slowness are could help narrow down the investigation. Tim On Wed, Sep 14, 2016 at 9:01 PM, Geoffrey Mina <gm...@connectfirst.com<mailto:gm...@connectfirst.com>> wrote: Greetings, I am pretty new to ActiveMQ as we just deployed into our stack. We had a network of brokers have a significant problem tonight due to (what appeared to be) a single broker. 2 of the 3 were processing messages OK and the third was queuing up and not processing messages quickly. This is for a real-time application and seconds (even milliseconds) count. There was absolutely nothing in the log file that was interesting. I even restarted the “bad” broker and when it came back online it was behaving identically. We have persistence completely disabled and there is not even a kahadb on the file system. I have attached my config here for one of the brokers (all configs identical except for the the network config). If anyone sees anything glaringly obvious, please let me know. We just deployed this cluster of servers last Friday. It has processed hundreds of millions of messages in the last few days – before it began to misbehave tonight. ActiveMQ 5.14.0 java version "1.8.0_102" Java(TM) SE Runtime Environment (build 1.8.0_102-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) Amazon EC2 Host CentOS 7 3.10.0-327.10.1.el7.x86_64 4 x Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz 8G RAM Thanks in advance! Geoff
