[
https://issues.apache.org/jira/browse/QPID-8681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922346#comment-17922346
]
Robert Godfrey commented on QPID-8681:
--------------------------------------
[~rammantripragada] I can't really comment on what/how recognition works at
your employer. As has been mentioned a few times now, as someone who is
outside your employer I have no visibility on that, or what you and your
colleagues intended after raising the initial JIRA (where [~sudheersana] was
not mentioned at all). I personally treat JIRAs as essentially a request for a
committer to do "something", whether it is to investigate, or apply a PR that
is mentioned in a comment, or apply an attached patch file or whatever. Since
your initial JIRA made no mention that you (or a colleague) was going to
provide a PR or patch - but you did supply a screenshot with a fix - I took the
intention to be - "here is a problem, here is a fix, can you make the change?"
so I did. I'm not sure what sort of content might be in your new PR, but
unless it is a further code fix I'm not sure what value it will bring to the
codebase. We don't record code authors in the code, nor are the git logs
expected to be a complete record of the authors who have contributed - for
instance all history prior to Sep 2006 is lost losing all the contributions
made when it was initially developed inside JPMorganChase prior to be
transferred to Red Hat and then donated to Apache. Prior to being a committer
at Apache, I often used to simply forward patches to me colleagues who were,
who checked it in to Subversion (the version control system in use at the time)
without attribution. We do thank you and [~sudheersana] for your work in
identifying this issue and providing a resolution
> Addressing lock contention in Sorted Queues under high load by optimizing
> property fetching
> -------------------------------------------------------------------------------------------
>
> Key: QPID-8681
> URL: https://issues.apache.org/jira/browse/QPID-8681
> Project: Qpid
> Issue Type: Improvement
> Components: Broker-J
> Affects Versions: qpid-java-broker-7.0.9, qpid-java-broker-9.2.0
> Reporter: Sudheer Sana
> Assignee: Robert Godfrey
> Priority: Major
> Labels: contention, performance
> Fix For: qpid-java-broker-9.2.1
>
> Attachments: image-2025-01-27-22-41-16-668.png,
> image-2025-01-27-22-42-13-739.png
>
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> *Summary:*
> Apache Qpid Broker J provides Sorted Queues, which allow users to implement a
> message re-enqueue with delay feature. This feature is crucial in scenarios
> where messages dequeued from regular queues cannot be processed immediately
> due to certain preconditions (e.g., available resources, concurrency limits).
> These messages are re-enqueued to Sorted Queues, where they are sorted based
> on their delay expiry time. Periodic jobs then check these Sorted Queues for
> expired messages and move them back to the regular queues for processing.
> Under high load conditions (e.g., a re-enqueue rate of ~7,000 messages per
> second), we observed that the broker experiences contention issues, causing
> it to stop responding to REST API calls (which time out if REST timeouts are
> set on the client side otherwise just hung around). These APIs are used to
> periodically fetch queue statistics.
> *Analysis:*
> By analyzing Java Flight Recorder (JFR) data, we identified the root cause of
> the contention:
> # REST API calls to retrieve queue depths required querying some specific
> predefined properties from the broker.
> # However, for each requested property, the broker was inadvertently
> fetching all 26 properties, resulting in repeated and excessive lock
> acquisition attempts on the Sorted Queue data structure.
> # Among the requested properties, the *oldestMessageAge* property (used for
> delay queues) significantly contributed to the contention by increasing the
> number of lock requests.
> # On the application side, querying for the *oldestMessageAge* property is
> unnecessary when dealing with delay queues, so avoiding this query will
> further mitigate the contention issue.
> *Stack Trace:*
> The following stack trace illustrates the contention observed during the
> issue:
> {code:java}
> at
> org.apache.qpid.server.queue.SortedQueueEntryList.next(SortedQueueEntryList.java:292)
> at
> org.apache.qpid.server.queue.SortedQueueEntryList$QueueEntryIteratorImpl.atTail(SortedQueueEntryList.java:686)
> at
> org.apache.qpid.server.queue.SortedQueueEntryList$QueueEntryIteratorImpl.advance(SortedQueueEntryList.java:698)
> at
> org.apache.qpid.server.queue.SortedQueueEntryList.getOldestEntry(SortedQueueEntryList.java:350)
> at
> org.apache.qpid.server.queue.AbstractQueue.getOldestMessageArrivalTime(AbstractQueue.java:1518)
> at
> org.apache.qpid.server.queue.AbstractQueue.getOldestMessageAge(AbstractQueue.java:1546)
> at jdk.internal.reflect.GeneratedMethodAccessor167.invoke(Unknown
> Source:-1) at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:566) at
> org.apache.qpid.server.model.ConfiguredObjectMethodAttributeOrStatistic.getValue(ConfiguredObjectMethodAttributeOrStatistic.java:68)
> at
> org.apache.qpid.server.model.ConfiguredObjectMethodStatistic.getValue(ConfiguredObjectMethodStatistic.java:26)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.getStatistics(AbstractConfiguredObject.java:3181)
> at
> org.apache.qpid.server.queue.SortedQueueImplWithAccessChecking.getStatistics(SortedQueueImplWithAccessChecking.java:42)
> at
> org.apache.qpid.server.model.AbstractConfiguredObject.getStatistics(AbstractConfiguredObject.java:3168)
> at
> org.apache.qpid.server.management.plugin.servlet.query.ConfiguredObjectExpressionFactory$ConfiguredObjectPropertyExpression.getValue(ConfiguredObjectExpressionFactory.java:312)
> at
> org.apache.qpid.server.management.plugin.servlet.query.ConfiguredObjectExpressionFactory$ConfiguredObjectPropertyExpression.evaluate(ConfiguredObjectExpressionFactory.java:285)
> at
> org.apache.qpid.server.management.plugin.servlet.query.ConfiguredObjectExpressionFactory$ConfiguredObjectPropertyExpression.evaluate(ConfiguredObjectExpressionFactory.java:272)
> at
> org.apache.qpid.server.filter.ComparisonExpression.evaluate(ComparisonExpression.java:388)
> at
> org.apache.qpid.server.filter.ComparisonExpression.matches(ComparisonExpression.java:580)
> at
> org.apache.qpid.server.management.plugin.servlet.query.ConfiguredObjectQuery.filterObjects(ConfiguredObjectQuery.java:210)
> at
> org.apache.qpid.server.management.plugin.servlet.query.ConfiguredObjectQuery.<init>(ConfiguredObjectQuery.java:86)
> at
> org.apache.qpid.server.management.plugin.servlet.rest.QueryServlet.performQuery(QueryServlet.java:93)
> at
> org.apache.qpid.server.management.plugin.servlet.rest.QueryServlet.doGet(QueryServlet.java:56)
> at
> org.apache.qpid.server.management.plugin.servlet.rest.AbstractServlet.doGet(AbstractServlet.java:128)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790) {code}
> * Java Flight Recorder (JFR) data
> !image-2025-01-27-22-41-16-668.png!
> *Steps to Reproduce:*
> # Configure a Qpid Broker with Sorted Queues.
> # Enable REST APIs for queue statistics retrieval.
> # Simulate high load by re-enqueuing ~7,000 messages per second to the
> Sorted Queues.
> # Monitor broker performance and REST API response times.
> *Expected Behavior:* The broker should handle high re-enqueue rates without
> contention issues, and REST API calls for queue statistics should not time
> out.
> *Actual Behavior:* Under high load conditions, the broker experiences
> contention on the Sorted Queue data structure, leading to REST API timeouts.
> *Proposed Solution:*
> To address this issue, we propose optimizing the property fetching mechanism
> in the Qpid Broker in ConfiguredObjectExpressionFactory.java:
> # Modify the broker code (ConfiguredObjectExpressionFactory) to retrieve
> only the specifically requested properties rather than all 26 properties.
> !image-2025-01-27-22-42-13-739.png!
> # Avoid querying the oldestMessageAge property on the application side when
> dealing with delay queues, as it is not required for processing.
> # These optimizations will reduce the load on the Sorted Queue's locking
> mechanism and prevent redundant data fetches, thereby improving the broker's
> responsiveness under high load conditions.
> We have tested the proposed changes in our environment and observed
> significant performance improvements, including reduced lock contention and
> faster REST API responses under high load.
>
> _*(Originally opened by Ram Mantripragada, but Reporter updated to Sudheer
> Sana, following discussion below)*_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
