On 06/10/2014 12:03 PM, Dina Belova wrote:
Hello, stackers!
Oslo.messaging is future of how different OpenStack components
communicate with each other, and really I’d love to start discussion
about how we can make this library even better then it’s now and how can
we refactor it make more production-ready.
As we all remember, oslo.messaging was initially inspired to be created
as a logical continuation of nova.rpc - as a separated library, with
lots of transports supported, etc. That’s why oslo.messaging inherited
not only advantages of now did the nova.rpc work (and it were lots of
them), but also some architectural decisions that currently sometimes
lead to the performance issues (we met some of them while Ceilometer
performance testing [1] during the Icehouse).
For instance, simple testing messaging server (with connection pool and
eventlet) can process 700 messages per second. The same functionality
implemented using plain kombu (without connection pool and eventlet)
driver is processing ten times more - 7000-8000 messages per second.
So we have the following suggestions about how we may make this process
better and quicker (and really I’d love to collect your feedback, folks):
1) Currently we have main loop running in the Executor class, and I
guess it’ll be much better to move it to the Server class, as it’ll make
relationship between the classes easier and will leave Executor only one
task - process the message and that’s it (in blocking or eventlet mode).
Moreover, this will make further refactoring much easier.
2) Some of the drivers implementations (such as impl_rabbit and
impl_qpid, for instance) are full of useless separated classes that in
reality might be included to other ones. There are already some changes
making the whole structure easier [2], and after the 1st issue will be
solved Dispatcher and Listener also will be able to be refactored.
3) If we’ll separate RPC functionality and messaging functionality it’ll
make code base clean and easily reused.
4) Connection pool can be refactored to implement more efficient
connection reusage.
Folks, are you ok with such a plan? Alexey Kornienko already started
some of this work [2], but really we want to be sure that we chose the
correct vector of development here.
For the impl_qpid driver, I think there would need to be quite
significant changes to make it efficient. At present there are several
synchronous roundtrips for every RPC call made[1]. Notifications are not
treated any differently than RPCs (and sending a call is no different to
sending a cast).
I agree the connection pooling is not efficient. For qpid at least it
creates too many connections for no real benefit[2].
I think this may be a result of trying to fit the same high-level design
to two entirely different underlying APIs.
For me at least, this also makes it hard to spot issues by reading the
code. The qpid specific 'unit' tests for oslo.messaging also fail for me
everytime when an actual qpidd broker is running (I haven't yet got to
the bottom of that).
I'm personally not sure that the changes to impl_qpid you linked to have
much impact on either efficiency or readability, safety of the code. I
think there could be a lot of work required to significantly improve
that driver, and I wonder if that would be better spent on e.g. the AMQP
1.0 driver which I believe will perform much better and will offer more
choice in deployment.
--Gordon
[1] For both the request and the response, the sender is created every
time, which results in at least one roundtrip to the broker. Again, for
both the request and the response, the message is then sent with a
blocking send, meaning a further synchronous round trip for each. So for
an RPC call, instead of just one roundtrip, there are at least four.
[2] In my view, what matters more than per-connection throughput for
olso.messaging, is the scalability of the system as you add many RPC
clients and servers. Excessive creation of connections by each process
will have a negative impact on this. I don't believe the current code
gets anywhere close to the limits of the underlying connection and
suspect it would be more efficient and faster to multiplex different
streams down the same connection. This would be especially true where
using eventlet I suspect.
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
