Nominal wrote:
I've been doing some testing against the latest 0.5 release of qpid and have
some questions on how the broker manages queues and the memory for them. I've
searched the archives but couldn't find any answers. At this point I have
narrowed my tests to only the c++ examples to eliminate any errors that may be
in code that I am writing.
I am using the simple direct examples for the cpp broker. The qpidd.conf is
empty so all the defaults are being used. When I start up qpid the memory use
is as follows
Every 2.0s: ps -eLF | grep qpid | grep -v grep
Wed Jun 10 11:48:59 2009
qpidd 29192 1 29192 0 5 20104 2868 0 11:47 ? 00:00:00
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29193 0 5 20104 2868 0 11:47 ? 00:00:00
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29194 0 5 20104 2868 0 11:47 ? 00:00:00
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29195 0 5 20104 2868 0 11:47 ? 00:00:00
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
qpidd 29192 1 29197 0 5 20104 2868 0 11:47 ? 00:00:00
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
The resident memory is 2868 from a clean start
I run ./declare_queues
Then I produce 50k messages on message_queue with: ./direct_producer localhost
5672 50000
resident size for each of the threads goes up to 53800
qpidd 29192 1 29192 0 5 32642 53800 0 11:47 ? 00:00:01
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
then I run the ./listener to consume the messages
qpidd 29192 1 29192 0 5 49051 54256 0 11:47 ? 00:00:02
/usr/sbin/qpidd --daemon --pid-dir /var/run/qpidd --data-dir /var/lib/qpidd
the resident memory size goes up again.
The only way I can free this memory is to restart the daemon. Is this how qpid
manages memory and is it something I should not be concerned with? How does
qpid determine when it has used enough memory to avoid swapping? Is this
something that is configured qpid.max_count qpid.max_size arguments? The way I
am expecting it to work in very lay terms is:
1. put message on queue, take up memory needed for message.
2. consume message successfully, free memory used for message.
The way it appears to be working is:
1. put message on queue, take up memory needed for message.
2. consume message successfully, never free used memory.
Any info on this would be greatly appreciated, I would really like to move
forward with using qpid for a messaging bus and have been very pleased with it
overall, I'm just confused at what I'm observing and if its normal or not.
qpidd is currently using the standard C++ new/delete to manage memory, so memory
freed by qpidd is not necessarily returned to the OS immediately, particularly
if the heap is fragmented. I'm not surprised at your results, but I haven't done
much heap profiling to date so I can't say much about when you would start to
see memory being released back to the OS.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
