[ 
https://issues.apache.org/jira/browse/QPID-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18140201#comment-18140201
 ] 

Daniil Kirilyuk commented on QPID-8789:
---------------------------------------

Hi Marco,

Thank you for the update and the corrected figures! It is good to see that the 
patch eliminates the accounting drift and reduces both CPU usage and overall GC 
time.

I am still a little concerned about the higher heap usage in some scenarios and 
the full GC pauses observed in the tests. We are working on several 
optimizations in bdb-store aimed at reducing allocations. These may help 
improve the memory profile, but the work is still in progress, and further 
benchmarking will be needed to assess their impact.

Our current plan is to include this fix in 10.2.0, which we are aiming to 
release around mid-January 2027. Would that timeframe work for you?

Kind regards,
Daniil

> [Broker-J] Pooled buffers leaked after a BDB flow to disk make the broker 
> flow every message to disk until it is restarted
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: QPID-8789
>                 URL: https://issues.apache.org/jira/browse/QPID-8789
>             Project: Qpid
>          Issue Type: Bug
>          Components: Broker-J
>    Affects Versions: qpid-java-broker-10.1.0, qpid-java-broker-10.1.1
>         Environment: Broker-J 10.1.0, Java 21, BDB virtual host, AMQP 1.0 
> clients. The affected code is unchanged on main.
>            Reporter: Marco Geri
>            Assignee: Daniil Kirilyuk
>            Priority: Major
>             Fix For: qpid-java-broker-10.2.0
>
>         Attachments: QPID-XXXX-proposed-fix.patch, 
> QPID-XXXX-reproducer-test.patch, QPID_8789.patch
>
>
> h2. Problem
> With a BDB virtual host and AMQP 1.0 messages, every flow to disk can leak 
> pooled direct buffers from the accounting. The JVM frees their memory, but 
> {{numberOfBuffersInUse}} never goes down, and 
> {{getAllocatedDirectMemorySize()}} is derived from it. Once that value passes 
> {{broker.flowToDiskThreshold}}, the broker flows every incoming message to 
> disk and reloads it from the store on delivery. Every such delivery flows the 
> message to disk again and leaks more, so the broker never recovers: it stays 
> very slow until it is restarted.
> In production (16 GiB max direct memory, messages routed to several queues, 
> browsers, message filters) the counter reached 202,512 buffers (49 GiB) while 
> the JVM had 1.9 GiB of direct memory reserved. The drift started after a mass 
> flow to disk; on days without one, the two values match.
> h2. Reproduce
> {{QPID-XXXX-reproducer-test.patch}} adds {{BDBFlowToDiskMetaDataTest}} 
> ({{broker-plugins/amqp-1-0-bdb-store}}), on a real {{BDBMessageStore}}: take 
> the metadata of a message, flow the message to disk, copy a section as a 
> delivery does, remove the message. On {{main}} it fails:
> {noformat}
> Pooled buffers still counted as in use after the message was removed ==> 
> expected: <0> but was: <1>
> {noformat}
> It passes if the section is not read after the flow to disk.
> h2. Cause
> {{StoredBDBMessage#flowToDisk()}} calls {{MessageDataRef#clear(false)}}, 
> which releases the encoded form of the metadata and drops the reference:
> {code:java}
> _metaData.clearEncodedForm();
> ...
> _metaData = null;
> {code}
> Threads that obtained the metadata earlier through 
> {{StoredBDBMessage#getMetaData()}} keep using it: consumers of other queues 
> delivering the same message, browsers, filters. When one of them reads a 
> section, {{AbstractSection#encodeIfNecessary()}} re-encodes it into a new 
> pooled buffer. This happens in {{copy()}} on every delivery, in 
> {{getEncodedForm()}}, {{writeTo()}} and {{getValue()}}. Nothing owns that 
> metadata any more, so the buffer is never disposed.
> Flow to disk on BDB happens in {{AbstractQueue#enqueue}} above the threshold, 
> and in {{AbstractServerMessageImpl#getContent}} on every delivery of a 
> message whose content was on disk. That is why the leak feeds itself.
> h2. Attachments
>  * *{{QPID-XXXX-reproducer-test.patch}}*: the test above. It fails on 
> {{main}}.
>  * *{{QPID-XXXX-proposed-fix.patch}}*: a first attempt at a fix, meant only 
> as a starting point. Once {{clearEncodedForm()}} has been called, 
> {{AbstractSection}} re-encodes into a heap buffer, which needs no dispose. 
> Sections that were never cleared are unchanged. With it, the test passes, and 
> so do the existing tests of {{amqp-1-0-protocol}}, {{bdbstore}} and 
> {{amqp-1-0-bdb-store}} (including with {{-Dprofile=java-bdb.1-0}}).
> You know this code far better than we do, so please treat the patch as a 
> suggestion to be reassessed rather than a finished fix. In particular, we are 
> not sure whether the right place for it is the section or the store. Another 
> option would be for {{MessageDataRef#clear(false)}} to keep owning the 
> metadata until the message is removed, at the cost of keeping decoded 
> metadata on the heap for messages on disk. There may well be a better 
> approach we have not considered. We are happy to test any alternative against 
> our workload, and thank you for taking a look.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to