I am stress testing activemq 5.9.0 by sending a flood of non-persistent messages to a topic using jmeter.
My producers are 50 threads attempting to deliver 1000 msgs/sec; each of size 1K. There is merely a 1 millisecond delay in between message sends. There are 100 consumers on the topic. It runs fine for 10-20 seconds or so, then I see the consumer stop working and "GC overhead limit exceeded" errors appear in the consumer's console (Weirdly, monitoring the gc via visualvm it does not seem like a massive gc event...). Looking at the heap dump from the consumer client I see a large number of ActiveMQObjectMessage instances, owned by the GC root ActiveMDSessionExecutor.messageQueue(). Is the consumer message dispatching simply falling badly behind incoming message flow? Can I tune the consumer somehow to compensate? Or is this just the expected failure mode for this scenario? Thanks for any insights. Sample output from the consumers: Created the tree successfully using JMS_Subscribers_Forever.jmx Starting the test @ Thu Feb 13 11:33:51 PST 2014 (1392320031599) Waiting for possible shutdown message on port 4445 summary + 0 in 5s = 0.0/s Avg: 0 Min: 5001 Max: 5001 Err: 1 (0.00%) Active: 100 Started: 100 Finished: 0 summary + 0 in 35s = 0.0/s Avg: 0 Min: 4999 Max: 5006 Err: 600 (0.00%) Active: 100 Started: 100 Finished: 0 java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to java_pid3941.hprof ... Heap dump file created [654947324 bytes in 8.677 secs] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Session Task-1" Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Logging Error: Unknown error writing event. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. java.lang.OutOfMemoryError: GC overhead limit exceeded Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Logging Error: Unknown error writing event. java.lang.OutOfMemoryError: GC overhead limit exceeded Logging Error: Unknown error writing event. java.lang.OutOfMemoryError: GC overhead limit exceeded Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport: tcp://server/172.55.33.120:61616@41330" Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. Uncaught Exception java.lang.OutOfMemoryError: GC overhead limit exceeded. See log file for details. -- View this message in context: http://activemq.2283324.n4.nabble.com/Stress-testing-OOME-tp4677959.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.