[
https://issues.apache.org/jira/browse/QPID-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18135135#comment-18135135
]
Marco Geri commented on QPID-8789:
----------------------------------
Hi Daniil,
Thank you very much for the patch and the benchmarks!
We re-created the problem we hit in a simulation environment reproducing our
broker configuration, and compared 10.1.1 with and without QPID_8789.patch: BDB
virtual host, AMQP 1.0 (ProtonJ2 clients), flow to disk forced with
broker.flowToDiskThreshold=32MB, Java 21, G1, Linux, -Xmx1g,
MaxDirectMemorySize=768m, 5 minutes per run. Three scenarios:
* A: persistent messages fanned out to 4 LVQs with non-destructive consumers
using JMS selectors, plus browsed and consumed queues (two runs);
* B: the configuration where we hit the problem: standard queues with
messageDurability=ALWAYS and ensureNondestructiveConsumers, small
non-persistent messages routed by a direct exchange both to a shared queue and
to a dedicated queue, 6 browsers reconnecting every 30s and re-reading the
growing queues;
* C: as B, with a selector on every browser (in our configuration every
consumer has a default filter reading an application property).
Patch vs 10.1.1:
|| ||A (2 runs)||B||C||
|max drift (counted-real)|-5.4% / -5.4% |-8.4%|-10.5%|
|broker CPU per read|-6.5% / -8.0%|-8.4%|-10.5%|
|GC time per second|-6.5% / -8.0%|-12.7%|-18.2%|
|heap avg|-10.3% / -4.6%|+8.6%|-1.9%|
|heap max|-7.3% / +0.6%|=|+4.8%|
|throughput|unchanged (noise)|unchanged|unchanged|
To be explicit about the drift row: every 10 seconds we sampled the broker
statistics numberOfBuffersInUse (x 256 KiB, i.e. what
getAllocatedDirectMemorySize() reports, "counted") and usedDirectMemorySize
(the direct BufferPoolMXBean, "real").
With the patch the counted value never exceeded the direct memory actually
reserved by the JVM in any run, so the buffer accounting stays in line with the
real direct memory. Without it the counted value keeps growing while the real
one goes down: at the end of the B and C runs, 470 MiB and 560 MiB counted
against 106 MiB and 98 MiB reserved.
The patch removes the drift in all scenarios and reduces CPU and GC time, more
so in B and C.
Average heap shows no clear trend, but the peak tends to be slightly higher, in
line with your fanout measurements: in B and C, where a 1 GB heap was already
tight without the patch, it led to one compaction full GC (245-300 ms, "G1
Compaction Pause" after an evacuation failure) that the unpatched broker did
not have. The few full GCs in A were all "System.gc()", which we attribute to
our deliberately small direct memory limit.
>From our side the patch looks good, and we will keep an eye on the heap peak
>when we adopt it.
Thanks again for your help, it made a real difference for us. Happy to run
further tests if useful.
> [Broker-J] Pooled buffers leaked after a BDB flow to disk make the broker
> flow every message to disk until it is restarted
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8789
> URL: https://issues.apache.org/jira/browse/QPID-8789
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-10.1.0, qpid-java-broker-10.1.1
> Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0
> clients. The affected code is unchanged on main.
> Reporter: Marco Geri
> Assignee: Daniil Kirilyuk
> Priority: Major
> Attachments: QPID-XXXX-proposed-fix.patch,
> QPID-XXXX-reproducer-test.patch, QPID_8789.patch
>
>
> h2. Problem
> With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak
> pooled direct buffers from the accounting. The JVM frees their memory, but
> {{numberOfBuffersInUse}} never goes down, and
> {{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes
> {{broker.flowToDiskThreshold}}, the broker flows every incoming message to
> disk and reloads it from the store on delivery. Every such delivery flows the
> message to disk again and leaks more, so the broker never recovers: it stays
> very slow until it is restarted.
> In production (16 GiB max direct memory, messages routed to several queues,
> browsers, message filters) the counter reached 202,512 buffers (49 GiB) while
> the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass
> flow to disk; on days without one, the two values match.
> h2. Reproduce
> {{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}}
> ({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take
> the metadata of a message, flow the message to disk, copy a section as a
> delivery does, remove the message. On {{main}} it fails:
> {noformat}
> Pooled buffers still counted as in use after the message was removed ==>
> expected: <0> but was: <1>
> {noformat}
> It passes if the section is not read after the flow to disk.
> h2. Cause
> {{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}},
> which releases the encoded form of the metadata and drops the reference:
> {code:java}
> _metaData.clearEncodedForm();
> ...
> _metaData = null;
> {code}
> Threads that obtained the metadata earlier through
> {{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues
> delivering the same message, browsers, filters. When one of them reads a
> section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new
> pooled buffer. This happens in {{copy()}} on every delivery, in
> {{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that
> metadata any more, so the buffer is never disposed.
> Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold,
> and in {{AbstractServerMessageImpl#getContent}} on every delivery of a
> message whose content was on disk. That is why the leak feeds itself.
> h2. Attachments
> * *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on
> {{main}}.
> * *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only
> as a starting point. Once {{clearEncodedForm()}} has been called,
> {{AbstractSection}} re-encodes into a heap buffer, which needs no dispose.
> Sections that were never cleared are unchanged. With it, the test passes, and
> so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and
> {{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
> You know this code far better than we do, so please treat the patch as a
> suggestion to be reassessed rather than a finished fix. In particular, we are
> not sure whether the right place for it is the section or the store. Another
> option would be for {{MessageDataRef#clear(false)}} to keep owning the
> metadata until the message is removed, at the cost of keeping decoded
> metadata on the heap for messages on disk. There may well be a better
> approach we have not considered. We are happy to test any alternative against
> our workload, and thank you for taking a look.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]