Hi everyone, Some time I posted about ActiveMQ OOM that kept occurring on our staging environment. At that point, I was missing reproducible test scenarios which I would like to post here.
The original problem is about ActiveMQ starting to spend too much time with Full GC, and eventually fails with OOM: GC Overhead limit exceeded. The root cause is slow consumer on non-persistent topic, which makes ActiveMQ flow to disk (temp store). Producer flow control was switched off, and we were hoping that producer will fill temp store (limit 10 Gb) before it gets blocked. Surprisingly, it turned out that the system was most stable when memory limit was set to a *smaller number*, 300Mb. Having 4Gb or 8Gb of heap and 70% of heap memory set as memory limit makes system to crash faster. I realized that PFC off means broker has no means to provide backpressure, and it can eventually flood the broker memory with incoming messages. However, it turned out that even Producer flow control ON (and even having reasonable producer window size) can provoke OOM on a broker side, depending on conditions. I can reproduce it with a simple example I want to share with you. I'm running broker with 3Gb heap, having memory usage limit set as 1Gb; producer window size: 10Mb. Per-topic policy memoryLimit is 10Mb. With a demo producer, I'm sending 100Kb messages with no delay (which is up to 10,000 msgs/sec until consumer started!), my slow consumer delays 50ms after each message. I can see PFC kicks in, but soon after I get OOM: 2018-08-07 15:28:33 INFO BrokerService:773 - Apache ActiveMQ 5.15.0 (localhost, ID:a-thinkpad-46421-1533644913567-0:1) started 2018-08-07 15:28:33 INFO BrokerService:774 - For help or more information please see: http://activemq.apache.org 2018-08-07 15:28:33 INFO TransportConnector:263 - Connector vm://localhost started 2018-08-07 15:29:01 INFO Topic:386 - topic://amq.topic, Usage Manager memory limit reached 10485760. Producers will be throttled to the rate at which messages are removed from this destination to prevent flooding it. See http://activemq.apache.org/producer-flow-control.html for more info. 2018-08-07 15:29:17 WARN Transport:237 - Transport Connection to: tcp://127.0.0.1:55802 failed: java.io.IOException: Unexpected error occurred: java.lang.OutOfMemoryError: GC overhead limit exceeded For convenience, I wrapped all the scenario on GitHub using the embedded broker, and demo producer/consumer in the same entry class. However, during my tests I was running producer/consumer in a separate JVMs, and still getting the same OOM result. You might object that such a 'heavy' message flow is disproportional to heap ActiveMQ gets, but I'd want to see the stable backpressure we would have during peaks. Thank you, Arturs -- Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
