[
https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678860#comment-16678860
]
Alan Protasio edited comment on AMQ-7091 at 11/8/18 3:49 PM:
-------------------------------------------------------------
Hi [~gtully]
Thanks again! haha :D
{quote}the solution is to have the data in the pageFile and pageCache such that
it can get flushed from memory at the cost of accessing from pages in normal
operation.
{quote}
Did you see my last proposed change? This change does not requite new index
version (kahadb) and will not write anything new to the index.
I dont know why the bot didn't add the comment (updated the PR) here but i'm
not adding a new Btree Index anymore... I'm reusing the data that is already
there:
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18]
We already have this information in other format in the ListIndex<String,
SequenceSet> ackPositions; So i'm using it.
My comment before was.... nowadays to discover if one message is referenced by
one subscribe we only do a sd.messageReferences.get(sequenceId) and check if
the reference count is 0.
With my change, we will do when acking a message something similar with that
we already do when sending a message.
See:
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2924]
and
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18#diff-e3b8fff8c2133dfd70999705bbb558b3R2935]
was (Author: alanprot):
Hi [~gtully]
Thanks again! haha :D
{quote}the solution is to have the data in the pageFile and pageCache such that
it can get flushed from memory at the cost of accessing from pages in normal
operation.
{quote}
Did you see my last proposed change?
I dont know why the bot didn't add the comment (updated the PR) here but i'm
not flushing the data to the pagefile anymore... I'm reusing the data that is
already there:
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18]
We already have this information in other format in the ListIndex<String,
SequenceSet> ackPositions; So i'm using this.
My comment before was.... nowadays to discover if one message is referenced by
one subscribe we only do a sd.messageReferences.get(sequenceId) and check if
the reference count is 0.
With my change, we will have to do the same as we already do when we are
sending one message:
See:
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2924]
and
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18#diff-e3b8fff8c2133dfd70999705bbb558b3R2935]
As we already have this code when sending a message, i dont know if this is a
problem.
> O(n) Memory consumption when broker has inactive durable subscribes causing
> OOM
> -------------------------------------------------------------------------------
>
> Key: AMQ-7091
> URL: https://issues.apache.org/jira/browse/AMQ-7091
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
> Affects Versions: 5.15.7
> Reporter: Alan Protasio
> Priority: Major
> Attachments: After.png, Before.png,
> InactiveDurableSubscriberTest.java, memoryAllocation.jpg
>
>
> Hi :D
> One of our brokers was bouncing indefinitely due OOM even though the load
> (TPS) was pretty low.
> Getting the memory dump I could see that almost 90% of the memory was being
> used by
> [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368]
> TreeMap to keep track of what messages were already acked by all Subscribes
> in order to delete them.
> This seems to be a problem as if the broker has an inactive durable
> subscribe, the memory footprint increase proportionally (O) with the number
> of messages sent to the topic in question, causing the broker to die due OOM
> sooner or later (the high memory footprint continue even after a restart).
> You can find attached (memoryAllocation.jpg) a screen shot showing my broker
> using 90% of the memory to keep track of those messages, making it barely
> usable.
> Looking at the code, I could do a change to change the messageReferences to
> use a BTreeIndex:
> final TreeMap<Long, Long> messageReferences = new TreeMap<>();
> + BTreeIndex<Long, Long> messageReferences;
> Making this change, the memory allocation of the broker stabilized and the
> broker didn't run OOM anymore.
> Attached you can see the code that I used to reproduce this scenario, also
> the memory utilization (HEAP and GC graphs) before and after this change.
> Before the change the broker died in 5 minutes and I could send 480000. After
> then change the broker was still pretty healthy after 5 minutes and i could
> send 2265000 to the topic (almost 5x more due high GC pauses).
>
> All test are passing: mvn clean install -Dactivemq.tests=all
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
