[
https://issues.apache.org/jira/browse/QPID-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133574#comment-18133574
]
Daniil Kirilyuk commented on QPID-8789:
---------------------------------------
Hi Marco,
I’ve attached QPID_8789.patch for testing and assessment. It takes a slightly
different approach: {{clearEncodedForm()}} copies the original section bytes
into an exactly sized heap buffer and releases the original buffer. This
preserves lazy decoding and the AMQP encoding, avoiding eviction-time decoding
and subsequent re-encoding. Repeated clearing does not copy the same section
again.
The patch also adds regression coverage for BDB / JDBC eviction, concurrent
readers and mixed heap / direct buffers,.
I compared main, your proposed patch, and this implementation using persistent
AMQP 1.0 / BDB workloads: point-to-point and fanout to 4 queues. Measurements
used Java 17 / G1 on Windows, a 300 Mb heap, a 200 Mb direct-memory limit, and
sustained backlog to exercise flow-to-disk. Across three runs per configuration:
* Compared with main, median throughput improved approximately 7% for
point-to-point and 11% for fanout; CPU per delivery decreased 12–13%, and heap
allocation per delivery decreased 6–18%.
* Compared with your patch, throughput improved approximately 6% and 11%,
respectively; CPU per delivery decreased 10–15%, and allocation decreased 6–7%.
Both fixes eliminated the observed pooled-buffer accounting drift in the longer
fanout checks.
There is a heap / GC tradeoff. In fanout, median heap occupancy was
approximately 211 MiB on main, 223 MiB with your patch, and 234 MiB with this
patch. Total GC pause time per delivery increased from 11.52 µs on main to
13.53 µs, although it remained below your patch’s 14.32 µs. Collections were
more frequent; average individual pauses were slightly shorter.
Could you test this against your reproducer and workload, particularly the heap
footprint and GC behavior, and share your assessment?
> [Broker-J] Pooled buffers leaked after a BDB flow to disk make the broker
> flow every message to disk until it is restarted
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8789
> URL: https://issues.apache.org/jira/browse/QPID-8789
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-10.1.0, qpid-java-broker-10.1.1
> Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0
> clients. The affected code is unchanged on main.
> Reporter: Marco Geri
> Assignee: Daniil Kirilyuk
> Priority: Major
> Attachments: QPID-XXXX-proposed-fix.patch,
> QPID-XXXX-reproducer-test.patch, QPID_8789.patch
>
>
> h2. Problem
> With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak
> pooled direct buffers from the accounting. The JVM frees their memory, but
> {{numberOfBuffersInUse}} never goes down, and
> {{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes
> {{broker.flowToDiskThreshold}}, the broker flows every incoming message to
> disk and reloads it from the store on delivery. Every such delivery flows the
> message to disk again and leaks more, so the broker never recovers: it stays
> very slow until it is restarted.
> In production (16 GiB max direct memory, messages routed to several queues,
> browsers, message filters) the counter reached 202,512 buffers (49 GiB) while
> the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass
> flow to disk; on days without one, the two values match.
> h2. Reproduce
> {{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}}
> ({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take
> the metadata of a message, flow the message to disk, copy a section as a
> delivery does, remove the message. On {{main}} it fails:
> {noformat}
> Pooled buffers still counted as in use after the message was removed ==>
> expected: <0> but was: <1>
> {noformat}
> It passes if the section is not read after the flow to disk.
> h2. Cause
> {{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}},
> which releases the encoded form of the metadata and drops the reference:
> {code:java}
> _metaData.clearEncodedForm();
> ...
> _metaData = null;
> {code}
> Threads that obtained the metadata earlier through
> {{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues
> delivering the same message, browsers, filters. When one of them reads a
> section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new
> pooled buffer. This happens in {{copy()}} on every delivery, in
> {{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that
> metadata any more, so the buffer is never disposed.
> Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold,
> and in {{AbstractServerMessageImpl#getContent}} on every delivery of a
> message whose content was on disk. That is why the leak feeds itself.
> h2. Attachments
> * *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on
> {{main}}.
> * *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only
> as a starting point. Once {{clearEncodedForm()}} has been called,
> {{AbstractSection}} re-encodes into a heap buffer, which needs no dispose.
> Sections that were never cleared are unchanged. With it, the test passes, and
> so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and
> {{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
> You know this code far better than we do, so please treat the patch as a
> suggestion to be reassessed rather than a finished fix. In particular, we are
> not sure whether the right place for it is the section or the store. Another
> option would be for {{MessageDataRef#clear(false)}} to keep owning the
> metadata until the message is removed, at the cost of keeping decoded
> metadata on the heap for messages on disk. There may well be a better
> approach we have not considered. We are happy to test any alternative against
> our workload, and thank you for taking a look.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]