[
https://issues.apache.org/jira/browse/QPID-8168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002913#comment-17002913
]
Alex Rudyy commented on QPID-8168:
----------------------------------
The reported issue impacts multiple use cases when small messages are published
with a lot of headers (into exchanges having queues bound with selectors). As
flow to disk is not triggered, the messages consume the heap and broker crushes
with {{OutOfMemoryError}}. It seems in order to address this issue and another
issue reported in QPID-7918, we need to track heap utilization by the messages.
There should be another flow to disk threshold for the heap memory utilization
by the messages. When that threshold is breached we need to start flowing
messages into disk.
Unfortunately, tracking heap memory utilization is not a trivial task. There
are several approaches we can utilize to evaluate heap usage:
* by estimating heap utilization by message and queue entries objects (by
estimating the size of message objects in memory using the approach similar to
[https://github.com/JetBrains/jdk8u_nashorn/blob/master/src/jdk/nashorn/internal/ir/debug/ObjectSizeCalculator.java]
). The BDB JE is using the similish approach for evaluating of heap usage by
BDB JE caches
* by using java instrumentation api
* by trying to listen to for events generated for Full GC via JMX notifications
issued by MemoryPoolMXBean
* using OpenJDK JOL tool http://openjdk.java.net/projects/code-tools/jol/
There is Stackoverflow discussion about heap usage evaluation at
[https://stackoverflow.com/questions/52353/in-java-what-is-the-best-way-to-determine-the-size-of-an-object]
I tried to play with JMX notifications and found that it does not work very
well with different Garbage Collector implementations. For example, the
notification from G1 GC are not send that often.
Instrumentation API looks like the easiest way to track memory but it requires
setting of javaagent. OpenJDK JOL API can use Instrumentation API to evaluate
the heap usage. I am not sure how performance would be impacted by utilizing
this approach.
I am inclined to track the message heap usage by estimating of object heap
consumption using the approach similar to the one used in BDB JE
(com.sleepycat.je.dbi.MemoryBudget) where Object overheads in memory is
evaluated based on estimated size of fields, object references, primitives,
arrays, etc. All implementations of {{ServerMessage}}, {{StoredMessage}},
{{StorableMessageMetaData}} and {{QueueEntry}} should allow to estimate the
heap usage before and after flowing message to disk. We can track the total
heap usage by messages on the broker object. When tracked heap usage
estimation exceeds the pre-defined threshold we can start flowing to disk all
incoming messages and already enqueued messages using the approach as for
direct memory.
> [Broker-J] Broker should flow to disk transient messages when heap usage
> reaches critical threshold in order to avoid running of heap memory due to
> consumption of heap by transient message metadata
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8168
> URL: https://issues.apache.org/jira/browse/QPID-8168
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-6.1.6, qpid-java-broker-7.0.3,
> qpid-java-broker-7.0.2, qpid-java-6.0.8, qpid-java-broker-7.0.0,
> qpid-java-broker-7.0.1, qpid-java-broker-7.1.0
> Reporter: Alex Rudyy
> Priority: Major
> Fix For: qpid-java-broker-8.0.0
>
>
> The flow to disk is currently triggered when direct memory usage reaches
> predefined threshold ({{broker.flowToDiskThreshold}} which is set to 75% of
> direct memory size by default). However, in some situations (for example,
> when direct memory size is big and heap memory size is small), the direct
> memory threshold is not reached but the heap is fully occupied by transient
> messages data, the broker runs out of heap memory.
> The broker behaviour can be improved by triggering flow to disk when a heap
> occupancy reaches the critical threshold in addition to direct memory usage
> threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
