attached, works with linux kernel with private-futex support (RHEL5.3 or
F10 etc). If you run
an older kernel without private-futexes shout and I will give you that
script.
Carl.
GS.Chandra N wrote:
That ought to be interesting. I shall do the profiling and perhaps take a
look at the code too. Please do send me the script.
Thanks
gs
On Mon, Mar 16, 2009 at 6:09 AM, Carl Trieloff <[email protected]>wrote:
I don't believe any profiling has been done on the headers exchange. If you
are on Linux, run oprofile on it
and either create a patch to correct the hot spot or mail the profile to
the list.
if needed I can also give mail you a stap (system tap) script for lock
analysis for the headers exchange.
regards,
Carl.
GS.Chandra N wrote:
Gordon, Tedd
After the patches to the headersexchange.cpp was applied, i can now clearly
see the incoming rates with the help of the tool Tedd sent.
The rates are at about 90-100 odd messages per sec (a drop from 5200 odd
without subscriptions).
Sort of matches my earlier observations using qpid-tool. What next?
Do you think there might be any way to add the headers exchange performance
improvements to your list of priority fixes? I'm not on the dev forums so i
'm not sure if the issue is communicated out there.
I will try to test the multiple subscriptions XML exchange next to if that
improves matters in any respect.
Thanks for the great support in fully un-reavelling the issue.
Rgds
gs
ps : I can also still reproduce the memory pile up by blocking the broker
using qpid-tool and setting the mgmgt-pub interval to 1 which of-course i
can avoid by not letting qpid-tool stick around and increasing the mgmt-pub
interval to 10 or more.
On Thu, Mar 12, 2009 at 3:45 PM, GS.Chandra N <[email protected]>wrote:
Hi,
I found that if i kept the qpid-tool open long enough all my clients and
publishers would throw exceptions and come out.
Earlier when i was performing my tests i had 3 putty windows opened to all
the 3 servers and it seems as if, i was keeping qpid-tool opened just long
enough to cause build up but closing it to run top and so on a so forth such
that the memory piled up but did not cause timeouts at the client either.
Observations
1.When there are subscriptions and a qpid-tool around messages seem to
pile up. Why is this so? *
*
2. When there are no subscribers and no qpid-tool the publishing processes
cause the CPu to go to 90% at the publishing box. But when i started up
subscribers, the publishing box CPU fell to 0% and all the python processes
piled up memory.
Concern : I'm not able to find out here if the publishers are still able
to send out messages at the speed that it should when subscriptions are
around. Or perhaps the broker is not able to pull enough messages - iam not
sure which of this is happeneing. Probably latter? How do i tell?
If i use qpid-tool to connect to the broker at this point, every time i
hit show exchange i get the same stats with the same time stamp. it looks
like the broker is too busy trying to match subscriptions even to refresh
the stats.
3.Concern : Once the subscriptions are made, the CPU at the broker box* *(high
end dual core XEON with ht) goes to 90%. Even at this level, i'm not sure
the broker is able to match and discard all the messages fast enough (due to
2nd observation). Or how do i tell?
Thanks
gs
ps : Is there a sure shot way to find out the message rates ? (I currently
use qpid-tool show exchange command to find the no of msgRecieves and divide
by time-dfference from 2 times the command is run)
On Thu, Mar 12, 2009 at 3:11 PM, GS.Chandra N <[email protected]>wrote:
I'm able to reproduce the issue, when i keep qpid-tool open all the time
with the mgmt switches.
However i'm not able to do so if qpid-tool is not open (did not try with
qpid-tool and no mgmt switches).
Thanks
gs
On Thu, Mar 12, 2009 at 1:31 AM, Gordon Sim <[email protected]> wrote:
GS.Chandra N wrote:
ps : Please find attached the scripts that create the issue
I'm afraid I wasn't able to recreate the issue with your tests. Memory
stayed reasonable and constant as the test was running (5 servers, 5
clients).
I'm baffled at this point as to what it is you are seeing.
broker - runs on a dedicated box
qpidd -d --port=5672 --mgmt-enable=yes --mgmt-pub-interval=1
--auth=no --log-source=yes --log-to-file=/tmp/somename
Do you have qpid-tool or anything else running during the test? Does
turning management off have any impact on the issue?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
#! stap
# This script tries to identify contended user-space locks by hooking
# into the futex system call.
global thread_thislock # short
global thread_blocktime #
global lock_waits # long-lived stats on (tid,lock) blockage elapsed time
global process_names # long-lived pid-to-execname mapping
probe syscall.futex {
## op == 0 -> FUTEX_WAIT
if (op != 0) next # we don't care about originators of WAKE events
t = tid ()
process_names[pid()] = execname()
thread_thislock[t] = $uaddr
thread_blocktime[t] = gettimeofday_us()
}
probe syscall.futex.return {
t = tid()
ts = thread_blocktime[t]
if (ts) {
elapsed = gettimeofday_us() - ts
lock_waits[pid(), thread_thislock[t]] <<< elapsed
delete thread_blocktime[t]
delete thread_thislock[t]
}
}
probe end {
foreach ([pid+, lock] in lock_waits)
printf ("%s[%d] lock %p contended %d times, %d avg us\n",
process_names[pid], pid, lock, @count(lock_waits[pid,lock]),
@avg(lock_waits[pid,lock]))
}
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
