ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

[art.licis]

Dear community,

We've been evaluation AMQ which is based on version 5.11.0. Several tests in
staging environments which would have a big number of producers and
consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
However, it looks like it's just one slow consumer that is causing this
memory problem (details below).

Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
had the same outcome.

Configuration: we have intentionally producer flow control turned off, and
expecting temp store to fill up before the producer is blocked
(non-persistent topic messages). I can simulate this with a simple and
controlled tests involving very fast producer, and several fast and some
slow consumers. With manual testing we achieve the desired result: when temp
store reaches the limit, the producer is blocked.

We analyzed GC outputs and heap dump which we got upon the crash. Time spent
in GC keeps significantly increasing before the crash, and heap dump shows
the majority of memory occupied by PageFile object tree (lots of
PageFile$PageWrite objects, which indicate that too much data remains in
memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
for non-persistent messages).

During my analysis I also discovered some bug that was fixed recently -
*AMQ-6815*. It looks like it affects the exact class family that could have
caused the memory leak. 

I'd be grateful for any help or suggestions. I'm currently preparing the
bigger test environment with many producers/consumers (and I'm also gonna
run Activemq in docker to limit I/O speed while I'm sure it wasn't the
issue), but the fact is that out of potentially thousands of
producers/consumers there was only one slow consumer. Eventually, we will
try running whole system against latest ActiveMQ, however, it will take lots
of arrangements and I'd like to see if I can make reproducable and
controlled test environment, and if there're some pitfalls in current setup.

activemq.xml is below (the slow consumer was not the one listening to
virtual topic). And I will post more info if I'm able to solve the issue or
at least get more hints.

Thank you.
-Art

<beans
  xmlns="http://www.springframework.org/schema/beans";
  xmlns:amq="http://activemq.apache.org/schema/core";
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
 
    
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="properties">
            <bean class="io.fabric8.mq.fabric.ConfigurationProperties"/>
        </property>
    </bean>
 
    <broker xmlns="http://activemq.apache.org/schema/core";
brokerName="${broker-name}" dataDirectory="${data}" start="false"
restartAllowed="false">
 
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="false">
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="false"
optimizedDispatch="true" >
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>
 
        <managementContext>
            <managementContext createConnector="true" connectorPort="9001"/>
        </managementContext>
 
        <ioExceptionHandler>
            <defaultIOExceptionHandler ignoreNoSpaceErrors="false"/>
        </ioExceptionHandler>
 
        <persistenceAdapter>
            <kahaDB directory="${data}/kahadb"/>
        </persistenceAdapter>
 
        <plugins>
        </plugins>
 
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="300 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="10 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
 
        <transportConnectors>
            <transportConnector name="openwire"
uri="tcp://0.0.0.0:0?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
        </transportConnectors>
 
        <destinationInterceptors>
            <virtualDestinationInterceptor>
                <virtualDestinations>
                    <compositeQueue name="rpc.request.>"
forwardOnly="false">
                        <forwardTo>
                            <topic
physicalName="rpc.request.virtual.notifications" />
                        </forwardTo>
                    </compositeQueue>
                    <compositeQueue name="rpc.response.>"
forwardOnly="false">
                        <forwardTo>
                            <topic
physicalName="rpc.response.virtual.notifications" />
                        </forwardTo>
                    </compositeQueue>
                </virtualDestinations>
            </virtualDestinationInterceptor>
        </destinationInterceptors>
 
    </broker>
 
</beans>



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html