Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

2018-07-20 Thread Tom Hall 
Hi Art,
What per destination limit are you looking for? The per destination limits will 
deal with memory used per queue.
http://activemq.apache.org/per-destination-policies.html 

 
   

-Tom

> On Jul 20, 2018, at 3:50 AM, art.licis  wrote:
> 
> 300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
> it was more. While I totally agree 300 Mb memory limit is far from the
> optimal setting, and even better per-destination limits are needed.
> 
> However, I have stress tested ActiveMQ with much less memory, and I've seen
> the following in logs:
> 
> topic://topic-3, Usage Manager memory
> limit reached 5242880. Producers will be throttled to the rate at which
> messages are removed from this destination to prevent flooding it.
> 
> << just 5Mb of memory limit on this test!
> 
> .. and eventually this:
> 
> TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
> destinations=1, dispatched=32766, delivered=15963293, matched=1460,
> discarded=0, prefetchExtension=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
> is full, temp usag (100%) or memory usage (0%) limit reached, blocking
> message add() pending the release of resources.
> 
> << all 10Gb of temp store filled!
> 
> 
> Which means filling up temp store and blocking the producer, but NOT
> crashing with out of memory, having all heap filled with PageFile-related
> data.
> 
> - Art
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

2018-07-20 Thread art.licis 
300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
it was more. While I totally agree 300 Mb memory limit is far from the
optimal setting, and even better per-destination limits are needed.

However, I have stress tested ActiveMQ with much less memory, and I've seen
the following in logs:

topic://topic-3, Usage Manager memory
 limit reached 5242880. Producers will be throttled to the rate at which
messages are removed from this destination to prevent flooding it.

<< just 5Mb of memory limit on this test!

.. and eventually this:

TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
destinations=1, dispatched=32766, delivered=15963293, matched=1460,
discarded=0, prefetchExtension=0: Pending message cursor
[org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
is full, temp usag (100%) or memory usage (0%) limit reached, blocking
message add() pending the release of resources.

<< all 10Gb of temp store filled!


Which means filling up temp store and blocking the producer, but NOT
crashing with out of memory, having all heap filled with PageFile-related
data.

- Art



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

2018-07-19 Thread Tom Hall 
Are you really limiting ram to 300 mb? Why so low if you are on a 8G box?
Thanks,
Tom

> On Jul 19, 2018, at 7:13 AM, art.licis  wrote:
> 
> Dear community,
> 
> We've been evaluation AMQ which is based on version 5.11.0. Several tests in
> staging environments which would have a big number of producers and
> consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
> However, it looks like it's just one slow consumer that is causing this
> memory problem (details below).
> 
> Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
> had the same outcome.
> 
> Configuration: we have intentionally producer flow control turned off, and
> expecting temp store to fill up before the producer is blocked
> (non-persistent topic messages). I can simulate this with a simple and
> controlled tests involving very fast producer, and several fast and some
> slow consumers. With manual testing we achieve the desired result: when temp
> store reaches the limit, the producer is blocked.
> 
> We analyzed GC outputs and heap dump which we got upon the crash. Time spent
> in GC keeps significantly increasing before the crash, and heap dump shows
> the majority of memory occupied by PageFile object tree (lots of
> PageFile$PageWrite objects, which indicate that too much data remains in
> memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
> PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
> for non-persistent messages).
> 
> During my analysis I also discovered some bug that was fixed recently -
> *AMQ-6815*. It looks like it affects the exact class family that could have
> caused the memory leak. 
> 
> I'd be grateful for any help or suggestions. I'm currently preparing the
> bigger test environment with many producers/consumers (and I'm also gonna
> run Activemq in docker to limit I/O speed while I'm sure it wasn't the
> issue), but the fact is that out of potentially thousands of
> producers/consumers there was only one slow consumer. Eventually, we will
> try running whole system against latest ActiveMQ, however, it will take lots
> of arrangements and I'd like to see if I can make reproducable and
> controlled test environment, and if there're some pitfalls in current setup.
> 
> activemq.xml is below (the slow consumer was not the one listening to
> virtual topic). And I will post more info if I'm able to solve the issue or
> at least get more hints.
> 
> Thank you.
> -Art
> 
>   xmlns="http://www.springframework.org/schema/beans";
>  xmlns:amq="http://activemq.apache.org/schema/core";
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd";>
> 
> 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>
>
>
>
> 
>http://activemq.apache.org/schema/core";
> brokerName="${broker-name}" dataDirectory="${data}" start="false"
> restartAllowed="false">
> 
>
>
>  
>
>
> optimizedDispatch="true" >
>
>  
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
> 
>
> uri="tcp://0.0.0.0:0?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
>
> 
>
>
>
> forwardOnly="false">
>
> physicalName="rpc.request.virtual.notifications" />
>
>
> forwardOnly="false">
>
> physicalName="rpc.response.virtual.notifications" />
>
>
>
>
>
> 
>
> 
> 
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

2018-07-19 Thread art.licis 
Dear community,

We've been evaluation AMQ which is based on version 5.11.0. Several tests in
staging environments which would have a big number of producers and
consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
However, it looks like it's just one slow consumer that is causing this
memory problem (details below).

Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
had the same outcome.

Configuration: we have intentionally producer flow control turned off, and
expecting temp store to fill up before the producer is blocked
(non-persistent topic messages). I can simulate this with a simple and
controlled tests involving very fast producer, and several fast and some
slow consumers. With manual testing we achieve the desired result: when temp
store reaches the limit, the producer is blocked.

We analyzed GC outputs and heap dump which we got upon the crash. Time spent
in GC keeps significantly increasing before the crash, and heap dump shows
the majority of memory occupied by PageFile object tree (lots of
PageFile$PageWrite objects, which indicate that too much data remains in
memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
for non-persistent messages).

During my analysis I also discovered some bug that was fixed recently -
*AMQ-6815*. It looks like it affects the exact class family that could have
caused the memory leak. 

I'd be grateful for any help or suggestions. I'm currently preparing the
bigger test environment with many producers/consumers (and I'm also gonna
run Activemq in docker to limit I/O speed while I'm sure it wasn't the
issue), but the fact is that out of potentially thousands of
producers/consumers there was only one slow consumer. Eventually, we will
try running whole system against latest ActiveMQ, however, it will take lots
of arrangements and I'd like to see if I can make reproducable and
controlled test environment, and if there're some pitfalls in current setup.

activemq.xml is below (the slow consumer was not the one listening to
virtual topic). And I will post more info if I'm able to solve the issue or
at least get more hints.

Thank you.
-Art

http://www.springframework.org/schema/beans";
  xmlns:amq="http://activemq.apache.org/schema/core";
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
 






 
http://activemq.apache.org/schema/core";
brokerName="${broker-name}" dataDirectory="${data}" start="false"
restartAllowed="false">
 


  




  


 



 



 



 


 













 



 
















 

 




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html