Hi Li Ma,
Thank you very much for your reply
On 06.03.15 05:01, Li Ma wrote:
Hi all, actually I'm writing the same mail topic for zeromq driver,
but I haven't done it yet. Thank you for proposing this topic,
ozamiatin.
1. ZeroMQ functionality
Actually I proposed a session topic in the coming summit to show our
production system, named 'Distributed Messaging System for OpenStack
at Scale'. I don't know whether it will be allowed to present.
Otherwise, if it is possible, I can share my experience in design
summit.
Currently, AWCloud (the company I'm working) deployed more than 20
private clouds and 3 public clouds for our customers in production,
scaling from 10 to 500 physical nodes without any performance issue.
The performance dominates all the existing drivers in every aspect.
All is using ZeroMQ driver. We started improving ZeroMQ driver in
Icehouse and currently the modified driver has switched to
oslo.messaging.
As all knows, ZeroMQ has been unmaintainable for long. My colleagues
and I continuously contribute patches to upstream. The progress is a
little bit slow because we are doing everything just in our spare time
and the review procedure is also not efficient.
Here are two important patches [1], [2], for matchmaker redis. When
they are landed, I think ZeroMQ driver is capable of running in small
deployments.
It's good to hear you have a living deployment with zmq driver.
Is there a big divergence between your production and upstream versions
of the driver? Besides [1] and [2] fixes for redis we have [5] and [6]
critical multi-backend related issues for using the driver in real-world
deployment.
The only functionality for large-scale deployment that lacks in the
current upstream codebase is socket pool scheduling (to provide
lifecycle management, including recycle and reuse zeromq sockets). It
was done several months ago and we are willing to contribute. I plan
to propose a blueprint in the next release.
Pool, recycle and reuse sounds good for performance.
We also need a refactoring of the driver to reduce redundant entities
or reconsider them (like ZmqClient or InternalContext) and to reduce
code replications (like with topics).
There is also some topics management needed.
Clear code == less bugs == easy understand == easy contribute.
We need a discussion (with related spec and UMLs) about what the driver
architecture should be (I'm in progress with that).
2. Why ZeroMQ matters for OpenStack
ZeroMQ is the only driver that depends on a stable library not an open
source product. This is the most important thing that comes up my
mind. When we deploy clouds with RabbitMQ or Qpid, we need
comprehensive knowledge from their community, from deployment best
practice to performance tuning for different scales. As an open source
product, no doubt that bugs are always there. You need to push lots of
things in different communities rather than OpenStack community.
Finally, it is not that working, you all know it, right?
ZeroMQ library itself is just encapsulation of sockets and is stable
enough and widely used in large-scale cluster communication for long.
We can build our own messaging system for inter-component RPC. We can
improve it for OpenStack and have the governance for codebase. We
don't need to rely on different products out of the community.
Actually, only ZeroMQ provides the possibility.
IMO, we can just keep it and improve it and finally it becomes another
choice for operators.
Zmq is also an opensource product and it has it's own community,
but I agree that rabbit is a complicated "blackbox" software we depend on.
While zmq is just a library and it is much more simpler (more reliable)
inside.
Zmq driver is more lower-level than rabbit and qpid therefore it
provides more flexibility.
By now it is the only driver where brokerless implementation is possible.
3. ZeroMQ integration
I've been working on the integration of ZeroMQ and DevStack for a
while and actually it is working right now. I updated the deployment
guide [3].
That's true it works! :)
I think it is the time to bring a non-voting gate for ZeroMQ and we
can make the functional tests work.
You can turn it with 'check experimental'. It is broken now.
4. ZeroMQ blueprints
We'd love to provide blueprints to improve ZeroMQ, as ozamiatin does.
According to my estimation, ZeroMQ can be another choice for
production in 1-2 release cycles due to bp review and patch review
procedure.
5. ZeroMQ discussion
Here I'd like to say sorry for this driver. Due to spare time and
timezone, I'm not available for IRC or other meeting or discussions.
But if it is possible, should we create a subgroup for ZeroMQ and
schedule meetings for it? If we can schedule in advance or at a fixed
date & time, I'm in.
That's great idea
+1 for zmq subgroup and meetings
6. Feedback to ozamiatin's suggestions
I'm with you in most all the proposals, but for packages, I think we
can just separate all the components in a sub-directory. This step is
enough at the current stage.
Packaging the components are complicated. I don't think it is possible
for oslo.messaging to break into two packages, like oslo.messaging and
oslo.messaging.zeromq. And I cannot see the benefit clearly.
Subfolder is actually what I mean (python package like '_drivers')
it should stay in oslo.messaging. Separate package like
oslo.messaging.zeromq is overkill.
As Doug proposed we can do consistently to AMQP-driver.
For priorities, I think the number 1, 6 and 7 have the high priority,
especially 7. Because ZeroMQ is pretty new for everyone, we do need
more paper work to promote and introduce it to the community. By the
way, I made a wiki before and everyone is welcome to update it [4].
[1] https://review.openstack.org/#/c/152471/
[2] https://review.openstack.org/#/c/155673/
[3] https://review.openstack.org/#/c/130943/
[4] https://wiki.openstack.org/wiki/ZeroMQ
[5] https://bugs.launchpad.net/oslo.messaging/+bug/1282297
[6] https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1381972
Thanks a lot,
Li Ma (Nick)
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Regards,
Oleksii Zamiatin
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
