[
https://issues.apache.org/jira/browse/QPID-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140231#comment-18140231
]
Marco Geri commented on QPID-8789:
----------------------------------
Hi Daniil,
Thank you for the feedback, and for the work on reducing allocations in
bdb-store.
Mid-January 2027 for 10.2.0 works well for us. We hit the issue in one specific
scenario on a single queue, and we have mitigated it for now by raising
broker.flowToDiskThreshold above the direct memory limit, so that flow to disk
is not triggered, and by lowering qpid.compact_memory_interval to one minute.
We are monitoring the broker closely, and given how that queue is used, this
workaround keeps us sufficiently safe until the fix is released. If it becomes
urgent before then, we can apply the patch on top of 10.1.1 as a temporary
measure until 10.2.0 is out.
Regarding the heap usage and the full GC pauses: once your bdb-store
optimizations are ready, I would be happy to re-run the same benchmarks with
the patch on top of them, if that helps.
Kind regards,
Marco
> [Broker-J] Pooled buffers leaked after a BDB flow to disk make the broker
> flow every message to disk until it is restarted
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: QPID-8789
> URL: https://issues.apache.org/jira/browse/QPID-8789
> Project: Qpid
> Issue Type: Bug
> Components: Broker-J
> Affects Versions: qpid-java-broker-10.1.0, qpid-java-broker-10.1.1
> Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0
> clients. The affected code is unchanged on main.
> Reporter: Marco Geri
> Assignee: Daniil Kirilyuk
> Priority: Major
> Fix For: qpid-java-broker-10.2.0
>
> Attachments: QPID-XXXX-proposed-fix.patch,
> QPID-XXXX-reproducer-test.patch, QPID_8789.patch
>
>
> h2. Problem
> With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak
> pooled direct buffers from the accounting. The JVM frees their memory, but
> {{numberOfBuffersInUse}} never goes down, and
> {{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes
> {{broker.flowToDiskThreshold}}, the broker flows every incoming message to
> disk and reloads it from the store on delivery. Every such delivery flows the
> message to disk again and leaks more, so the broker never recovers: it stays
> very slow until it is restarted.
> In production (16 GiB max direct memory, messages routed to several queues,
> browsers, message filters) the counter reached 202,512 buffers (49 GiB) while
> the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass
> flow to disk; on days without one, the two values match.
> h2. Reproduce
> {{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}}
> ({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take
> the metadata of a message, flow the message to disk, copy a section as a
> delivery does, remove the message. On {{main}} it fails:
> {noformat}
> Pooled buffers still counted as in use after the message was removed ==>
> expected: <0> but was: <1>
> {noformat}
> It passes if the section is not read after the flow to disk.
> h2. Cause
> {{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}},
> which releases the encoded form of the metadata and drops the reference:
> {code:java}
> _metaData.clearEncodedForm();
> ...
> _metaData = null;
> {code}
> Threads that obtained the metadata earlier through
> {{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues
> delivering the same message, browsers, filters. When one of them reads a
> section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new
> pooled buffer. This happens in {{copy()}} on every delivery, in
> {{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that
> metadata any more, so the buffer is never disposed.
> Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold,
> and in {{AbstractServerMessageImpl#getContent}} on every delivery of a
> message whose content was on disk. That is why the leak feeds itself.
> h2. Attachments
> * *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on
> {{main}}.
> * *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only
> as a starting point. Once {{clearEncodedForm()}} has been called,
> {{AbstractSection}} re-encodes into a heap buffer, which needs no dispose.
> Sections that were never cleared are unchanged. With it, the test passes, and
> so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and
> {{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
> You know this code far better than we do, so please treat the patch as a
> suggestion to be reassessed rather than a finished fix. In particular, we are
> not sure whether the right place for it is the section or the store. Another
> option would be for {{MessageDataRef#clear(false)}} to keep owning the
> metadata until the message is removed, at the cost of keeping decoded
> metadata on the heap for messages on disk. There may well be a better
> approach we have not considered. We are happy to test any alternative against
> our workload, and thank you for taking a look.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]