Re: [Openstack] is anyone using zeromq for RPC?
On 2014/9/13 14:32, James Page wrote: The scale potential is very appealing and is something I want to test - - hopefully in the next month or so. Canonical are interested in helping to maintain this driver and hopefully we help any critical issues prior to Juno release. That sounds good. I just went through all the bugs reported in the community. The only critical bug which makes ZeroMQ malfunction is https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the corresponding review is under way: https://review.openstack.org/#/c/84938/ Others are tagged to 'zmq' in https://bugs.launchpad.net/oslo.messaging By the way, some bugs as far as I know, are outdated. They should be re-verified one by one later. Cheers, Li Ma ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Li On 17/09/14 11:58, Li Ma wrote: The scale potential is very appealing and is something I want to test - - hopefully in the next month or so. Canonical are interested in helping to maintain this driver and hopefully we help any critical issues prior to Juno release. That sounds good. I just went through all the bugs reported in the community. The only critical bug which makes ZeroMQ malfunction is https://bugs.launchpad.net/oslo.messaging/+bug/1301723 and the corresponding review is under way: https://review.openstack.org/#/c/84938/ Agreed Others are tagged to 'zmq' in https://bugs.launchpad.net/oslo.messaging Looking through Doug's suggested list of information and collating what I know of from our work last week: 1) documentation for how to configure and use zeromq with oslo.messaging (note, not the version in oslo-incubator, the version in the messaging library repository) As part of our sprint, I worked on automating deployment of OpenStack + 0MQ using Ubuntu + Juju (service orchestration tool). I can re-jig that work into some general documentation on how best to configure ZeroMQ with OpenStack - the current documentation is a bit raw and does not talk about how to configure the oslo-messaging-zmq-receiver at all. I also plan some packaging updates for Debian/Ubuntu in our next dev cycle to make this a little easier to configure and digest - for example, right now no systemd unit/upstart configuration/sysv init script is provided to manage the zmq-receiver. I'd also like to document the current design of the ZMQ driver - Doug - - where is the best place todo this? I thought in the source tree somewhere. 2) a list of the critical bugs that need to be fixed + any existing patches associated with those bugs, so they can be reviewed early in kilo This blocks operation of nova+neutron environements: https://bugs.launchpad.net/oslo.messaging/+bug/1301723 Summary: Message was sent to wrong node with zmq as rpc_backend Patch: https://review.openstack.org/84938 Also notifcations are effectively unimplemented which prevents use with Ceilometer so I'd also add: https://bugs.launchpad.net/oslo.messaging/+bug/1368154 Summary: https://bugs.launchpad.net/oslo.messaging/+bug/ Patch: https://review.openstack.org/120745 3) an analysis of what it would take to be able to run functional tests for zeromq on our CI infrastructure, not necessarily the full tempest run or devstack-gate job, probably functional tests we place in the tree with the driver (we will be doing this for all of the drivers) + besides writing new functional tests, we need to bring the unit tests for zeromq into the oslo.messaging repository Kapil Thangavelu started work on both functional tests for the ZMQ driver last week; the output from the sprint is here: https://github.com/ostack-musketeers/oslo.messaging it covers the ZMQ driver (including messaging through the zmq-receiver proxy) and the associated MatchMakers (local, ring, redis) at a varying levels of coverage, but I feel it moves things in the right direction - Kapil's going to raise a review for this in the next couple of days. Doug - has any structure been agreed within the oslo.messaging tree for unit/functional test splits? Right now we have them all in one place. Edward Hope-Morley also worked on getting devstack working with ZMQ: https://github.com/ostack-musketeers/devstack that's still WIP but again we'll get any changes submitted for review ASAP. 4) and some improvements that we would like to make longer term a) Connection re-use on outbound messaging avoiding the current tcp setup overhead for every sent message. This may also bring further performance benefits due to underlying messaging batching in ZMQ. b) Moving from tcp PUSH/PULL sockets between servers to DEALER/DEALER (or something similar) to allow for heartbeating and more immediate failure detection c) Crypto support Cheers James - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBCAAGBQJUGXaSAAoJEL/srsug59jDo4oQAJP4wrsxRtf1OBJ3/fyXtH0I 6UQ28LH9ILpguA8XA7ekK3EOIlWVwuXzenBWkHOrdt6uXJ8eTQzYs01DGg7LvlKD 5FR5LbB7EZoXxC/RjiKA5dlFoc36AhirFYpPkrnWMIv91cRgzxXnbENmfxKRkMbk QXzATFuf2CUEjXo6H+sTBfdsdq3sA3mEhqjBQ4zzkO1Mw2Jp3T8vqNueZtA6xnPV /HzRJbFtzo6A29+AJvseHc4glMwZ5OLXQ49Opuuo0eOYlkNUyj6fb18aX7vuoGNK 166+Inl9L5o2Sjwm/p24Q3DZCOJJVKKhxB2HEcObIOJSkUlLepoEwoY4pAh27LWH o4yB0Ou4R8wJGD27xhixdrU4bt/qAJjekek/pZvTFVVaNp/LOto/ZjAEZ06XLFbB EFrB6ZQ7Us0FJ0I7vz/ivzSro21+5gmzght+z7z+JHWoe0BC61W+kij+2VdWzz4q SQQxXVX+M18J7D1dLUD01k1Ky1/xoLG+ssf9Hh1GSh6VlVqiKBp9m72VL8eJDh5S cVyr1k5qaR7QU7ttMPaAmTkJlYzI+jaAvJIIjyQNSIUtjPtXmQAaAsCfSDwWBKOs SdgOKHtWU9TiMqtV7LrFIBY3nx+wgiKB3WQFnqlwiod/zsqlKw6g4rdU/7xFVFR3 pfRBUCgU1mIVeY0+VUyT =EW/H -END PGP SIGNATURE- ___ Mailing
Re: [Openstack] is anyone using zeromq for RPC?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi Li/Doug On 12/09/14 11:35, Li Ma wrote: - documentation for how to configure and use zeromq with oslo.messaging (note, not the version in oslo-incubator, the version in the messaging library repository) - a list of the critical bugs that need to be fixed - any existing patches associated with those bugs, so they can be reviewed early in kilo - an analysis of what it would take to be able to run functional tests for zeromq on our CI infrastructure, not necessarily the full tempest run or devstack-gate job, probably functional tests we place in the tree with the driver (we will be doing this for all of the drivers) - besides writing new functional tests, we need to bring the unit tests for zeromq into the oslo.messaging repository I'll first do the bug analysis and deployment doc. Hopefully the draft will be finished next week. I can help with that; my team spent some time last week sprinting on ZeroMQ support in Oslo Messaging/OpenStack to assess its potential for scale and current state so I think we've already answered quite a few of those questions, as well as started work on getting unit and functional tests up and running for the ZMQ driver. We also managed to persuade Pieter Hintjens (ZeroMQ upstream) to pop by and review the overall design of ZMQ driver - he gave it a general +1 and we have a list of longer term improvements to make. Li - I'll ping you directly to co-ordinate on this activity next week and then we can come back to list to report the details. It would also be useful, frankly, to have some idea of whether there is interest beyond the two teams who have reported trying to use it so far. If it’s only interesting to you, we might split the driver out of oslo.messaging and let you maintain it separately. If there is wide interest, I would lean towards keeping it in the tree where it is now. Yes, it makes sense. I'm not sure who is using it or trying to use it. As far as I know, ZeroMQ never got attention although it is the only broker-less method in OpenStack. The scale potential is very appealing and is something I want to test - - hopefully in the next month or so. Canonical are interested in helping to maintain this driver and hopefully we help address any critical issues prior to Juno release. Regards James - -- James Page Ubuntu and Debian Developer james.p...@ubuntu.com jamesp...@debian.org -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBCAAGBQJUE+TsAAoJEL/srsug59jDfGwP/1hQWsp1X9Ycfo2qa/ObCMG+ 5/AMvHzivTwZ8Z3Oo2Zdcwsv1aCTdyAJnr3D//VNvCTg+wsIkcKccNo90Ez0vGpT BgeutdxfreZ7W8YVGr16chTgciwXKDzcQ+B9QFdM/H2fYXvnFBFW9sLSIOJ0cTo8 N1QgRgMI9WXsQ06+0DHP44DGMDwtUK5d4ONe5Tc9aMiaHplaSMDabxXXDzBDH+cG PFyGQzX0jiLoZlMXZhHJwvEAZC6arpxGKssdipYg8xiG9j773n8n1HGZMqWyiJgD BjrVwwoSgVyGzCxtNeTlY8RfrYv1TU7GA3WfZKARjOWYUamsx7pFJwCj45h8DyIX BDtUQHBvk3TnkQ3TKuBgWG9rHAYQHWsuUOWng75kxuQGKEIWCU0vv43Eh3YTloIO D6XrdflosDdfAZ40EwLX/3Id8rxrpLUWPSuz8a3eHEyBbpcVthd03ix6T0WE2u5g 2EmSzYFTmU3DN2F14PsYur2tCzFPVebUHpvLxCURwEz1jrWQ5k19U3wBLYjZO2K2 H9XsXWFNXUSwbJeHAubztg+vZtNwlbYjXMGjDW50Yz0x0cMuGsgrTFBTEfxNTGiV 2aZZHuzczVpjxsuJmv3OSIGbWWap4knft1R7T0pbg9RTMcbcX5iSggL/7v9iyg2p wxr+FPJuQqaV9ZeRDVEA =zHZi -END PGP SIGNATURE- ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On 2014/9/11 5:13, Doug Hellmann wrote: Here are a few things we need, in no particular order: - documentation for how to configure and use zeromq with oslo.messaging (note, not the version in oslo-incubator, the version in the messaging library repository) - a list of the critical bugs that need to be fixed - any existing patches associated with those bugs, so they can be reviewed early in kilo - an analysis of what it would take to be able to run functional tests for zeromq on our CI infrastructure, not necessarily the full tempest run or devstack-gate job, probably functional tests we place in the tree with the driver (we will be doing this for all of the drivers) - besides writing new functional tests, we need to bring the unit tests for zeromq into the oslo.messaging repository I'll first do the bug analysis and deployment doc. Hopefully the draft will be finished next week. It would also be useful, frankly, to have some idea of whether there is interest beyond the two teams who have reported trying to use it so far. If it’s only interesting to you, we might split the driver out of oslo.messaging and let you maintain it separately. If there is wide interest, I would lean towards keeping it in the tree where it is now. Yes, it makes sense. I'm not sure who is using it or trying to use it. As far as I know, ZeroMQ never got attention although it is the only broker-less method in OpenStack. cheers, Li Ma ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On 2014/9/8 23:11, Doug Hellmann wrote: I would love to try to get them done step by step. And I would be happy to have your help doing that! Do you or your team have resources to contribute? I'm from Awcloud, an IaaS provider in China. We built our OpenStack clouds based on ZeroMQ in multiple DCs during this year. As I stated in the last thread, we've contributed several patches. The last one to make it work is reworked now. AFAIK, ZeroMQ is non-AMQP low-level communication method, so there are some gaps which cannot be ignored, how it works, how the nodes communicate with each other via ZeroMQ driver, how to make it run, how to debug, any tuning methods for production, etc. Actually, it seems that every aspect is totally different from AMQP-based methods, like qpid or rabbitmq. Everything needs to be documented to guide people who are interested in this driver. I just noticed that dropping ZeroMQ is on the plan: https://etherpad.openstack.org/p/kilo-oslo-summit-topics Damnit, I lost previous discussions on this topic. I'm mainly working in other projects in most of the time. Right now, I'm willing to contribute from now on. My colleagues are also available to help. Yes, Josh mentioned that in another message in this thread. It does sound like a potentially useful approach, although given the expectations of oslo.messaging right now I’m not sure how to make it work or if it would actually be easier than updating the current driver. It is definitely worth investigating. Doug Yes, it's great that someone is working on that way. ZeroMQ driver test cases are developing. https://bugs.launchpad.net/oslo.messaging/+bug/1302941 I'm not sure the progress, but I can try to link him and see if we can work together to get it done. Meanwhile, I'll start to investigate other must-do. OK, actually I'm not very clear about the priorities of must-do. So, Doug, what do you think of it? cheers, Li Ma ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 10, 2014, at 7:27 AM, Li Ma skywalker.n...@gmail.com wrote: On 2014/9/8 23:11, Doug Hellmann wrote: I would love to try to get them done step by step. And I would be happy to have your help doing that! Do you or your team have resources to contribute? I'm from Awcloud, an IaaS provider in China. We built our OpenStack clouds based on ZeroMQ in multiple DCs during this year. As I stated in the last thread, we've contributed several patches. The last one to make it work is reworked now. AFAIK, ZeroMQ is non-AMQP low-level communication method, so there are some gaps which cannot be ignored, how it works, how the nodes communicate with each other via ZeroMQ driver, how to make it run, how to debug, any tuning methods for production, etc. Actually, it seems that every aspect is totally different from AMQP-based methods, like qpid or rabbitmq. Everything needs to be documented to guide people who are interested in this driver. I just noticed that dropping ZeroMQ is on the plan: https://etherpad.openstack.org/p/kilo-oslo-summit-topics That list is of *proposed* topics to discuss. It is not a final list, yet, since we’re still in the early phases of planning our summit discussions. Damnit, I lost previous discussions on this topic. I'm mainly working in other projects in most of the time. Right now, I'm willing to contribute from now on. My colleagues are also available to help. That’s good to hear, and I hope between your team and Antonio’s maybe we can pull together a small group to own the zeromq driver and make it fully functional. Yes, Josh mentioned that in another message in this thread. It does sound like a potentially useful approach, although given the expectations of oslo.messaging right now I’m not sure how to make it work or if it would actually be easier than updating the current driver. It is definitely worth investigating. Doug Yes, it's great that someone is working on that way. ZeroMQ driver test cases are developing. https://bugs.launchpad.net/oslo.messaging/+bug/1302941 I'm not sure the progress, but I can try to link him and see if we can work together to get it done. Meanwhile, I'll start to investigate other must-do. OK, actually I'm not very clear about the priorities of must-do. So, Doug, what do you think of it? Here are a few things we need, in no particular order: - documentation for how to configure and use zeromq with oslo.messaging (note, not the version in oslo-incubator, the version in the messaging library repository) - a list of the critical bugs that need to be fixed - any existing patches associated with those bugs, so they can be reviewed early in kilo - an analysis of what it would take to be able to run functional tests for zeromq on our CI infrastructure, not necessarily the full tempest run or devstack-gate job, probably functional tests we place in the tree with the driver (we will be doing this for all of the drivers) - besides writing new functional tests, we need to bring the unit tests for zeromq into the oslo.messaging repository It would also be useful, frankly, to have some idea of whether there is interest beyond the two teams who have reported trying to use it so far. If it’s only interesting to you, we might split the driver out of oslo.messaging and let you maintain it separately. If there is wide interest, I would lean towards keeping it in the tree where it is now. Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On 2014/9/8 23:11, Doug Hellmann wrote: It’s great to know you were able to get a version working for your needs. I really thought it was fairly broken — did you have to patch the driver at all? Actually it was not that broken. Patches our team provided are merged before. https://review.openstack.org/#/c/84310/ https://review.openstack.org/#/c/84618/ And here's another abandoned patch: https://review.openstack.org/#/c/84938/ I'll explain it in the bug list later and rework on it. With these patch, ZeroMQ will be OK. (1) Provide tuning parameters for ZeroMQ context and sockets in order to fine-tune ZeroMQ for different deployment scenario. (2) Provide comprehensive test cases for ZeroMQ. (3) Support Devstack. (4) Provide installation tutorial in OpenStack docs. (5) Provide suggestions on real deployment architecture in OpenStack docs. We also need to add functional testing for the driver in the CI environment. Yes, almost forgot that CI is the most important for ZeroMQ driver, and the documentation as well. As I listed above, we need a plan for how to build the whole ZeroMQ support environment step by step, especially the priority that I'm not very clear. Anyway, I think it seems not realistic to land everything for Juno. cheers, Li Ma ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
Hi Doug, It is sad that it may be abandoned in future OpenStack release. ZeroMQ is fantastic messaging library because it is broker-less and can be fully distributed. It is really useful in large-scale deployment scenario. Currently we are using ZeroMQ as the message queue implementation in our production systems from Havana. We've deployed this method in multiple DCs. Abandon ZeroMQ is a great loss. AFAIK, the following can be done to make it active in OpenStack ecosystem. (1) Provide tuning parameters for ZeroMQ context and sockets in order to fine-tune ZeroMQ for different deployment scenario. (2) Provide comprehensive test cases for ZeroMQ. (3) Support Devstack. (4) Provide installation tutorial in OpenStack docs. (5) Provide suggestions on real deployment architecture in OpenStack docs. I would love to try to get them done step by step. By the way, I just noticed that kombu supports ZeroMQ. I wonder how it support direct/topic/fanout. Maybe we can try the kombu way to simplify the current impl_zmq implementation. But it needs further investigation. cheers, Li Ma - Original Message - From: Doug Hellmann d...@doughellmann.com To: Sandy Walsh sandy.wa...@rackspace.com Cc: openstack@lists.openstack.org Sent: 星期日, 2014年 9 月 07日 上午 1:01:38 Subject: Re: [Openstack] is anyone using zeromq for RPC? On Sep 6, 2014, at 11:00 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Well, yeah, but that doesn't help the operators that have it deployed already and didn't see this thread. Their update process will be: 1. install 2. wtf?! 3. scramble. I expect to make a lot of noise about it between now and the middle/end of kilo when we actually remove the driver. As I said it's not at all clear the driver even works in its current state, so I don't know if anyone could be using it as it is. Doug Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
Hi All, We tested briefly ZeroMQ with Havana last year, but we couldn't find any good documentation on how to implement it, and we were not able to get it working. We also got the impression that the support was not at all mature, so we decided to use RabbitMQ instead. However, I must say that the broker-less design of ZeroMQ is very appealing, and we would like to give it a try, assuming 1) the documentation is improved 2) there is some assurance that support for ZeroMQ is not going to be dropped. I can help with 1) if there is someone that knows a bit of the internals and can bootstrap me, because I have no first hand experience on how message queues are used in OpenStack, and little experience with ZeroMQ. About 2), well this is a decision for the developers, but IMHO there *should* be support for ZeroMQ in OpenStack: its broker-less architecture would eliminate a SPoF (the message broker), could ease the deployment (especially in HA setup) and grant very high performance. My 2 cents... Antonio Messina -- antonio.s.mess...@gmail.com antonio.mess...@uzh.ch +41 (0)44 635 42 22 S3IT: Service and Support for Science IT http://www.s3it.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 8, 2014, at 10:00 AM, Li Ma skywalker.n...@gmail.com wrote: Hi Doug, It is sad that it may be abandoned in future OpenStack release. ZeroMQ is fantastic messaging library because it is broker-less and can be fully distributed. It is really useful in large-scale deployment scenario. Yes, I’m also disappointed that we don’t have anyone interested in supporting it. Currently we are using ZeroMQ as the message queue implementation in our production systems from Havana. We've deployed this method in multiple DCs. Abandon ZeroMQ is a great loss. AFAIK, the following can be done to make it active in OpenStack ecosystem. It’s great to know you were able to get a version working for your needs. I really thought it was fairly broken — did you have to patch the driver at all? (1) Provide tuning parameters for ZeroMQ context and sockets in order to fine-tune ZeroMQ for different deployment scenario. (2) Provide comprehensive test cases for ZeroMQ. (3) Support Devstack. (4) Provide installation tutorial in OpenStack docs. (5) Provide suggestions on real deployment architecture in OpenStack docs. We also need to add functional testing for the driver in the CI environment. I would love to try to get them done step by step. And I would be happy to have your help doing that! Do you or your team have resources to contribute? By the way, I just noticed that kombu supports ZeroMQ. I wonder how it support direct/topic/fanout. Maybe we can try the kombu way to simplify the current impl_zmq implementation. But it needs further investigation. Yes, Josh mentioned that in another message in this thread. It does sound like a potentially useful approach, although given the expectations of oslo.messaging right now I’m not sure how to make it work or if it would actually be easier than updating the current driver. It is definitely worth investigating. Doug cheers, Li Ma - Original Message - From: Doug Hellmann d...@doughellmann.com To: Sandy Walsh sandy.wa...@rackspace.com Cc: openstack@lists.openstack.org Sent: 星期日, 2014年 9 月 07日 上午 1:01:38 Subject: Re: [Openstack] is anyone using zeromq for RPC? On Sep 6, 2014, at 11:00 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Well, yeah, but that doesn't help the operators that have it deployed already and didn't see this thread. Their update process will be: 1. install 2. wtf?! 3. scramble. I expect to make a lot of noise about it between now and the middle/end of kilo when we actually remove the driver. As I said it's not at all clear the driver even works in its current state, so I don't know if anyone could be using it as it is. Doug Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 8, 2014, at 10:35 AM, Antonio Messina antonio.s.mess...@gmail.com wrote: Hi All, We tested briefly ZeroMQ with Havana last year, but we couldn't find any good documentation on how to implement it, and we were not able to get it working. We also got the impression that the support was not at all mature, so we decided to use RabbitMQ instead. However, I must say that the broker-less design of ZeroMQ is very appealing, and we would like to give it a try, assuming 1) the documentation is improved 2) there is some assurance that support for ZeroMQ is not going to be dropped. I can help with 1) if there is someone that knows a bit of the internals and can bootstrap me, because I have no first hand experience on how message queues are used in OpenStack, and little experience with ZeroMQ. Unfortunately, the existing Oslo team doesn’t have a lot of experience with ZeroMQ either (hence this thread). It sounds like Li Ma’s team has made it work, though, so maybe you could work together. We should prioritize documentation and then functional testing, I think. About 2), well this is a decision for the developers, but IMHO there *should* be support for ZeroMQ in OpenStack: its broker-less architecture would eliminate a SPoF (the message broker), could ease the deployment (especially in HA setup) and grant very high performance. I agree, it would be useful to support it. This is purely a resource allocation problem for me. I don't have anyone willing to do the work needed to ensure the driver is functional and can be deployed sanely (although maybe I’ve found a couple of volunteers now :-). There is another effort going on to support AMQP 1.0, which (as I understand it) includes similar broker-less deployment options. Before we decide whether to invest in ZeroMQ for that reason alone, it would be useful to know if AMQP 1.0 support makes potential ZeroMQ support less interesting. Doug My 2 cents... Antonio Messina -- antonio.s.mess...@gmail.com antonio.mess...@uzh.ch +41 (0)44 635 42 22 S3IT: Service and Support for Science IT http://www.s3it.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
The team has been focused on library graduation this cycle, so some bugs have not received the attention we would like. Now that we’re in the feature-freeze period, some reviewer resources should be freed up. I will add those patches to the priority list to see what we can do about them. Doug On Sep 5, 2014, at 5:13 PM, Noel Burton-Krahn n...@pistoncloud.com wrote: Thanks for the link, Doug. I'll take a look. We found a few problems and race conditions with rabbit's failover. Is that being addressed in Juno? I didn't see anything in the blueprint. We've posted patches, but they've been waiting for merge for a while. We're hoping it would be simpler to avoid rabbit altogether. https://review.openstack.org/#/c/109373/ https://review.openstack.org/#/c/110511/ I'll take a look at kombu's zeromq -- Noel On Fri, Sep 5, 2014 at 1:57 PM, Doug Hellmann d...@doughellmann.com wrote: Have you been following the work on the AMQP 1.0 driver going into the Juno release of oslo.messaging? https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation Doug On Sep 5, 2014, at 4:50 PM, Noel Burton-Krahn n...@pistoncloud.com wrote: We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 5, 2014, at 5:02 PM, Joshua Harlow harlo...@yahoo-inc.com wrote: What about just using the kombu transport instead? https://github.com/celery/kombu/blob/master/kombu/transport/zmq.py Then people have a way to use oslo.messaging (the rabbit impl just uses kombu) with zmq without needing a specific zmq implementation. Or if the kombu one doesn't work well, maybe help make it better? The usage patterns defined by oslo.messaging needs more than the connection code, especially for the RPC call and response pattern we rely on heavily. I don’t know if our current kombu-based driver would work with the zmq connectivity kombu itself provides because our driver assumes the presence of an exchange to route messages. Either way, there is not right now anyone on the Oslo team interested in focusing on that work no matter where it would be done, and so I am trying to find someone who might like to contribute to it because they actually want to use zeromq. Doug From: Noel Burton-Krahn n...@pistoncloud.com Date: Friday, September 5, 2014 at 1:50 PM To: Doug Hellmann d...@doughellmann.com Cc: openstack@lists.openstack.org openstack@lists.openstack.org, Nick Bartos n...@pistoncloud.com Subject: Re: [Openstack] is anyone using zeromq for RPC? We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
From: Doug Hellmann [d...@doughellmann.com] On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Well, yeah, but that doesn't help the operators that have it deployed already and didn't see this thread. Their update process will be: 1. install 2. wtf?! 3. scramble. Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Sep 6, 2014, at 11:00 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Well, yeah, but that doesn't help the operators that have it deployed already and didn't see this thread. Their update process will be: 1. install 2. wtf?! 3. scramble. I expect to make a lot of noise about it between now and the middle/end of kilo when we actually remove the driver. As I said it's not at all clear the driver even works in its current state, so I don't know if anyone could be using it as it is. Doug Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
We couldn't get the zmq driver to work so we stuck with our patched rabbit version. On 2014-09-06 10:15 AM, Doug Hellmann d...@doughellmann.com wrote: On Sep 6, 2014, at 11:00 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] On Sep 5, 2014, at 11:13 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. If a maintainer is found, we can extract the files or resurrect them in place at any point in the future. Well, yeah, but that doesn't help the operators that have it deployed already and didn't see this thread. Their update process will be: 1. install 2. wtf?! 3. scramble. I expect to make a lot of noise about it between now and the middle/end of kilo when we actually remove the driver. As I said it's not at all clear the driver even works in its current state, so I don't know if anyone could be using it as it is. Doug Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] is anyone using zeromq for RPC?
The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
On Fri, Sep 5, 2014 at 2:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. I haven't seen any work on zmq in DevStack for some time either. We'll follow Oslo in dropping it if that decision is made. dt -- Dean Troyer dtro...@gmail.com ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
Have you been following the work on the AMQP 1.0 driver going into the Juno release of oslo.messaging? https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation Doug On Sep 5, 2014, at 4:50 PM, Noel Burton-Krahn n...@pistoncloud.com wrote: We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
What about just using the kombu transport instead? https://github.com/celery/kombu/blob/master/kombu/transport/zmq.py Then people have a way to use oslo.messaging (the rabbit impl just uses kombu) with zmq without needing a specific zmq implementation. Or if the kombu one doesn't work well, maybe help make it better? From: Noel Burton-Krahn n...@pistoncloud.commailto:n...@pistoncloud.com Date: Friday, September 5, 2014 at 1:50 PM To: Doug Hellmann d...@doughellmann.commailto:d...@doughellmann.com Cc: openstack@lists.openstack.orgmailto:openstack@lists.openstack.org openstack@lists.openstack.orgmailto:openstack@lists.openstack.org, Nick Bartos n...@pistoncloud.commailto:n...@pistoncloud.com Subject: Re: [Openstack] is anyone using zeromq for RPC? We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.commailto:d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.orgmailto:openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
Thanks for the link, Doug. I'll take a look. We found a few problems and race conditions with rabbit's failover. Is that being addressed in Juno? I didn't see anything in the blueprint. We've posted patches, but they've been waiting for merge for a while. We're hoping it would be simpler to avoid rabbit altogether. https://review.openstack.org/#/c/109373/ https://review.openstack.org/#/c/110511/ I'll take a look at kombu's zeromq -- Noel On Fri, Sep 5, 2014 at 1:57 PM, Doug Hellmann d...@doughellmann.com wrote: Have you been following the work on the AMQP 1.0 driver going into the Juno release of oslo.messaging? https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation Doug On Sep 5, 2014, at 4:50 PM, Noel Burton-Krahn n...@pistoncloud.com wrote: We are looking at moving to zeromq to get away from rabbit bugs like https://bugs.launchpad.net/oslo.messaging/+bug/1338732. Cheers, -- Noel On Fri, Sep 5, 2014 at 12:50 PM, Doug Hellmann d...@doughellmann.com wrote: The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] is anyone using zeromq for RPC?
From: Doug Hellmann [d...@doughellmann.com] The zmq driver in oslo.messaging, used for internal communication between OpenStack services, has been without a maintainer for a significant period of time. It isn’t actively tested, and it isn’t clear whether or not it works. The Oslo team would like to drop support for it in Kilo, but before we do that we would like to find out if (a) anyone uses it and (b) if any of those people would like to contribute to maintaining it. Why not just break it out into a separate repo? It would still be loadable via stevedore. This would let people keep using it if they desire (and, more importantly, maintain it externally) vs. the all-or- nothing inclusion/exclusion decision. Thanks, Doug ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
