this looks like an instance of the regression from https://issues.apache.org/jira/browse/AMQ-6815
On Wed, 10 Jan 2018 at 19:42 neon18 <neo...@nngo.net> wrote: > We run the broker with max heap of 4G and initial of 1G (-Xmx4G -Xms1G). > We use non-persistent messages on these particular queues (3 of them in > this > test). > The number of messages sent to the broker in my last "flood gate" test was > around 40,000 (40k) in 5 minutes or about 8K msgs/min. After this flood of > messages, the producers send messages at a much much lower rate. I have > pretty much the factory default activemq.xml with > systemUsage/memoryUsage/percentOfJvmHeap=70 and queuePrefetch=20 on these 3 > queues. > > So I have seen two different scenarios when lots of non-persistent messages > are put on queue. > 1. Async Writer Thread Shutdown errors (with no prior warnings/errors), > then > OutOfMemoryErrors > 2. INFO PListStore: ...tmp_storage initialized, ~10 seconds later WARN: > IOException: OutOfMemoryError: GC overhead limit exceeded ... ActiveMQ > Transport: tcp:..., then that repeats and other errors follow. Also there > is > no warnings/errors prior to the tmp_storage init info log msg. FYI: the web > console was responsive until I saw the tmp_storage initialized (KahaDB) > INFO > msg (~4.5 minutes into my test), then it stops responding. The last count > of > messages on queues via web console is ~30K msgs under ActiveMQ 5.15.2 > broker. Under the 5.14.5 broker, I was able to see the flood of ~40K msgs > added to the 3 queues in ~6 minutes. > > In more controlled testing in the past 2 days, where I clear the AMQ_DATA > dir before each test run, I have not seen issue #1 (Async Writer Thread > Shutdown / OutOfMemoryError). I see issue #2 (OutOfMemoryError) a few > seconds after KahaDB tmp_storage is initialized, then the web console stop > responding and lots of OoMerrors and other errors in the activemq.log. > > Running with ActiveMQ 5.14.5 and 5.12.2 brokers, we do not have any > OutOfMemoryErrors with this same load or higher load vs running under > ActiveMQ 5.15.2. Running with 5.15.2 broker it seems like there might be an > issue with throttling the producers of the queue when the JVM hit's the > configured memoryUsage (default of 70%). > > Running on that thought, I did another test with > systemUsage/memoryUsage/percentOfJvmHeap=50 %, but same thing (except that > the OoM error occurs about 20 seconds after the tmp_storage init info log. > > So, I ran the test again with systemUsage/memoryUsage to 20%, same thing, > except the OoM error occurs about 40 seconds after the tmp_storage init > info > log. This time, I also monitored the memory percent used and temp memory > percent used via the web console. right after I see the tmp_storage init > info log, I can see memUsed=39% tempUsed=1%, ~10 seconds later memUsed=56% > tempUsed=2%, ~10 seconds later memUsed=69% tempUsed=2%, next refresh failed > and of course in the activemq.log I see the OutOfMemoryErrors and other > warnings and errors appearing in the log. > > Also, I grepped my old logs for "Journal failed" and did see some results, > but they happen after the a few OutOfMemoryErrors, so I did not include > them > in this thread. > > I can pretty reliably recreate the problem in about 6 minutes (with a clean > amq_data_dir) running ActiveMQ 5.15.2 broker and no issues under 5.14.5 or > 5.12.2 brokers. > > Regards, > > Neon > > > > -- > Sent from: > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html >
