Jonathan Smith created QPID-8175:
------------------------------------
Summary: [c++ broker] QPID broker queue route leaks memory
Key: QPID-8175
URL: https://issues.apache.org/jira/browse/QPID-8175
Project: Qpid
Issue Type: Bug
Components: C++ Broker
Affects Versions: qpid-cpp-1.37.0
Environment: The setup I have is as follows:
On Centos 6.9 with 8GB of RAM, running _qpid-cpp-server_ with the default
configuration i.e.
{{ /usr/bin/qpidd --auth=no}}
Using the following packages
* qpid-tools 1.37
* qpid-cpp-server 1.37
Other than that the server is vanilla - It has no special packages installed.
Reporter: Jonathan Smith
When routing data with the use of qpid queue routes, I've noticed situations
where the memory starts to rise controlled when the queue on the source broker
fills up and the route cannot seem to keep up with the data flow.
The _qpidd_ process eventually consumers all the memory on the source broker
and is terminated by the kernel _oom-killer_
The problem seems to happen when a high rate of data is sent through a qpid
queue route from a headers exchange.
The queue starts to fill up as it selects incoming data and once it is full the
memory continues to rise as long as there is data flowing through the exchange
to the route queue.
Stopping this data and letting the queue 'drain down' through the destination
broker instance stops the memory rise but does not deallocate any memory.
I create a push route from source to destination broker with a 1GB source queue
with limit ring policy that subscribes to all traffic from the default headers
exchange (amq.match) and send data until the source queue fills up
For the sending of data, I have a single qpid-send instace sending to a
"xmatch: all" bindings on the headers exhange to a queue route on another
broker instance. The queue always manages to keep up for a few minutes then
inevitably starts to fill up.
After this point the memory continues to rise up to the point of exhaustion.
Stopping the qpid-send at any point stops the memory rise but does not
de-allocate any blocks.
>From the metrics on the exchange it looks like the issue seems to occur when
>the route is struggling to keep up with data the source queue is binding to on
>the exchange.
It is not known what is going on 'internally' when the queue route cannot
process an incoming message in time but I would have assumed that the message
would be discarded
and any memory used released but that is not what looks to be happening.
The following steps, executed sequentially on the source broker produces the
problem (on the dest broker just need to start it up, no queues are needed)
{code:java}
# Start qpid broker
/usr/bin/qpidd --auth=no
# Create queue and push route
qpid-config add queue queuerouteall --max-queue-size=1000000000
--limit-policy=ring -b <source broker address>
qpid-config bind amq.match queuerouteall qrakey all x-match=all -b <source
broker address>
qpid-route -s queue add <destination broker address> <source broker address>
amq.match queuerouteall
# Send high rate of data to source broker
qpid-send -b <source broker address> -a amq.match -m 0 --send-rate 5000
--content-size 1000000 --connection-options {reconnect:true}
{code}
I have also run this scenario on qpid-cpp-server and qpid-tools versions 0.30
and 1.36 and in each case the process consumes all the memory on the
server(even though the only queue had a ring limit of 1GB)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
