On 07/01/2014 03:52 PM, Ihar Hrachyshka wrote:
On 01/07/14 15:55, Alexei Kornienko wrote:
And in addition I've provided some links to existing
implementation with places that IMHO cause bottlenecks. From my
point of view that code is doing obviously stupid things (like
closing/opening sockets for each message sent).
That indeed sounds bad.
That is enough for me to rewrite it even without additional
proofs that it's wrong.
[Full disclosure: I'm not as involved into oslo.messaging internals as
you probably are, so I may speak out dumb things.]
I wonder whether there are easier ways to fix that particular issue
without rewriting everything from scratch. Like, provide a pool of
connections and make send() functions use it instead of creating new
connections (?)
I did a little investigation, using the test scripts from Alexei.
Looking at the protocol trace with wireshark, you can certainly see lots
of redundant stuff. However I did *not* see connections being opened and
closed.
What I did see was the AMQP (0-9-1) channel being opened and closed, and
the exchange being declared for every published message. Both these
actions are synchronous. This clearly impacts throughput and, in my view
more importantly, will limit scalability by giving the broker redundant
work.
However, I think this could be avoided (at least to the extreme extent
it occurs at present) by a simple patch. I posted this up to gerrit in
case anyone is interested: https://review.openstack.org/#/c/104194/
With this in place the ingress improves significantly[1]. The bottleneck
then appears to be the acking of messages. This means the queue depth
tends to get higher than before, and the traffic appears more 'bursty'.
Each message is acked individually, whereas some degree of batching
might be possible which would likely improve the efficiency there a
little bit.
Anyway, just an observation I thought it would be interesting to share.
--Gordon.
[1] From less than 400 msgs/sec to well over 1000 msgs/sec on my laptop
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
