I don't believe any profiling has been done on the headers exchange. If
you are on Linux, run oprofile on it
and either create a patch to correct the hot spot or mail the profile to
the list.
if needed I can also give mail you a stap (system tap) script for lock
analysis for the headers exchange.
regards,
Carl.
GS.Chandra N wrote:
Gordon, Tedd
After the patches to the headersexchange.cpp was applied, i can now
clearly see the incoming rates with the help of the tool Tedd sent.
The rates are at about 90-100 odd messages per sec (a drop from 5200
odd without subscriptions).
Sort of matches my earlier observations using qpid-tool. What next?
Do you think there might be any way to add the headers exchange
performance improvements to your list of priority fixes? I'm not on
the dev forums so i 'm not sure if the issue is communicated out there.
I will try to test the multiple subscriptions XML exchange next to if
that improves matters in any respect.
Thanks for the great support in fully un-reavelling the issue.
Rgds
gs
ps : I can also still reproduce the memory pile up by blocking the
broker using qpid-tool and setting the mgmgt-pub interval to 1 which
of-course i can avoid by not letting qpid-tool stick around and
increasing the mgmt-pub interval to 10 or more.
On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I found that if i kept the qpid-tool open long enough all my
clients and publishers would throw exceptions and come out.
Earlier when i was performing my tests i had 3 putty windows
opened to all the 3 servers and it seems as if, i was keeping
qpid-tool opened just long enough to cause build up but closing it
to run top and so on a so forth such that the memory piled up but
did not cause timeouts at the client either.
Observations
1.When there are subscriptions and a qpid-tool around messages
seem to pile up. Why is this so? *
*
2. When there are no subscribers and no qpid-tool the publishing
processes cause the CPu to go to 90% at the publishing box. But
when i started up subscribers, the publishing box CPU fell to 0%
and all the python processes piled up memory.
Concern : I'm not able to find out here if the publishers are
still able to send out messages at the speed that it should when
subscriptions are around. Or perhaps the broker is not able to
pull enough messages - iam not sure which of this is happeneing.
Probably latter? How do i tell?
If i use qpid-tool to connect to the broker at this point, every
time i hit show exchange i get the same stats with the same time
stamp. it looks like the broker is too busy trying to match
subscriptions even to refresh the stats.
3.Concern : Once the subscriptions are made, the CPU at the broker
box* *(high end dual core XEON with ht) goes to 90%. Even at this
level, i'm not sure the broker is able to match and discard all
the messages fast enough (due to 2nd observation). Or how do i tell?
Thanks
gs
ps : Is there a sure shot way to find out the message rates ? (I
currently use qpid-tool show exchange command to find the no of
msgRecieves and divide by time-dfference from 2 times the command
is run)
On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N
<[email protected] <mailto:[email protected]>> wrote:
I'm able to reproduce the issue, when i keep qpid-tool open
all the time with the mgmt switches.
However i'm not able to do so if qpid-tool is not open (did
not try with qpid-tool and no mgmt switches).
Thanks
gs
On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <[email protected]
<mailto:[email protected]>> wrote:
GS.Chandra N wrote:
ps : Please find attached the scripts that create the
issue
I'm afraid I wasn't able to recreate the issue with your
tests. Memory stayed reasonable and constant as the test
was running (5 servers, 5 clients).
I'm baffled at this point as to what it is you are seeing.
broker - runs on a dedicated box
qpidd -d --port=5672 --mgmt-enable=yes
--mgmt-pub-interval=1 --auth=no --log-source=yes
--log-to-file=/tmp/somename
Do you have qpid-tool or anything else running during the
test? Does turning management off have any impact on the
issue?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
<mailto:[email protected]>
