Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On 12/05/15 13:32, Fox, Kevin M wrote: Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) Ah yes, I forgot about Barbican, but as another application-facing API it is indeed in the same boat as all of the others we've been talking about in this thread by the sound of it. I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. So just keep in mind that this problem is not in any way specific to Barbican. If we can get Nova to automatically create a Keystone user account for every server and provide the credentials/token in the metadata, that would be awesome. Nova is the hardest place to get new features into though, so it might be wise to consider other options. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Yes, that's correct. EC2 provides a machine account for each VM, along with a rotating key (I guess equivalent to a token for our purposes) accessible through the metadata. You can dynamically assign roles to the machine user in order to control its access to all of the AWS API actions. cheers, Zane. Thanks, Kevin *From:* Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] *Sent:* Tuesday, May 12, 2015 10:06 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.gov mailto:kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.com mailto:zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On 12/05/15 09:44, Stan Lagun wrote: +1 for making Murano Engine - Murano Agent communication plugable so that one can switch to Zaqar or anything else. Cool, thanks. However watching RabbitMQ development for years I know hard can it be to build efficient and reliable system and I'm just not sure Zaqar can compete with such battle-proven thing like RabbitMQ yet. Yep, it's certainly a hard problem. Zaqar has some different constraints though - Rabbit is optimised for low-latency and throughput. For Zaqar it's more about reliability and scalability. I don't think the scalability problem is going to be completely solved overnight, but I think the important thing at this stage is that the API be stable and not present any barriers to improving scalability on the back end - and I think the v2 API will do that. For the kinds of workloads that something like Murano could throw at it, I think it's more than capable already. The only advantage I see is multi-tenancy. The big advantage is in having a single interface across OpenStack for this. But yes, multi-tenancy is the main prerequisite for that to happen. But I do believe it can be relatively easy be implemented with RabbitMQ. At lease in Murano. Don't want to go off topic here. The main idea is to use https://github.com/rabbitmq/rabbitmq-auth-backend-amqp and dynamically grant agent permissions only to his dedicated input queue so that it cannot access anything else. Not just other tenants queues but also queues of other VMs in the same tenant. In case of Murano we will not need to maintain additional secrets or databases. Neither it will be needed to create RabbitMQ users/vhosts as all of this becomes virtual. And agent will not be holding any RabbitMQ passwords at all There is more to multi-tenancy than just authentication/authorisation. It's also things like making sure one tenant's use of resources doesn't affect another tenant's (e.g. creating a denial of service by maxing out capacity); being able to measure and bill for usage; quotas to prevent accidental overuse c. So while adding auth to your current solution is definitely a good thing and I would encourage you to do so (in addition to making the whole thing pluggable :) I don't think that even an authenticated Rabbit can become the single point for messaging across OpenStack. There's too much stuff missing for it to be a serious contender for public clouds in particular. That's not to say Rabbit couldn't be used as a back-end to Zaqar, although it should be noted that the Zaqar team already investigated that in depth and found that it didn't meet their needs (at least at the time). cheers, Zane. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com On Tue, May 12, 2015 at 10:52 AM, Renat Akhmerov rakhme...@mirantis.com mailto:rakhme...@mirantis.com wrote: Zane, Fully agree with you vision here. On 12 May 2015, at 07:15, Zane Bitter zbit...@redhat.com mailto:zbit...@redhat.com wrote: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. Any volunteers? * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] I added an agenda item for the summit in https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral to discuss this. Everyone is welcome. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Cool picture of a wonderful future :) Thanks for reading. Please grab me at summit if you want to know more; I am always happy to bend the ear of anyone who will listen at length on this topic. As usual, I'll be the tall dude with the weird accent ;) With the great pleasure. (P.S. your accent is cool!) Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
see the email thread Re: [openstack-dev] [trove][zaqar] Trove and Zaqar integration. Summit working session Thanks, Kevin From: Georgy Okrokvertskhov Sent: Wednesday, May 13, 2015 11:16:41 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action +1. Is it a separate e-mail thread or IRC meeting? Thanks Gosha On Wed, May 13, 2015 at 10:21 AM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: it seems like the trove/zaqar guys are talking about a very similar topic at the exact same time. :/ Maybe it would be a good time for all parties to get together and chat? Thanks, Kevin From: Stan Lagun [sla...@mirantis.commailto:sla...@mirantis.com] Sent: Tuesday, May 12, 2015 11:52 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action On Wed, May 13, 2015 at 2:46 AM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: Awesome. When is it/where do I go to look up that info? http://sched.co/3Clo Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.comhttp://www.mirantis.com/ Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On Wed, May 13, 2015 at 5:42 PM, Zane Bitter zbit...@redhat.com wrote: There is more to multi-tenancy than just authentication/authorisation. It's also things like making sure one tenant's use of resources doesn't affect another tenant's (e.g. creating a denial of service by maxing out capacity); being able to measure and bill for usage; quotas to prevent accidental overuse c. This is a indeed a strong point that makes sense for shared MQ broker. However I'm not convinced that Zaqar approach is an optimal one: 1. There are a several dozens of message-oriented middleware solutions around (QPID, ActiveMQ, Camel, Kafka, HortnetQ and many more). Many of them are well know, well tested, have significant user base, published books etc. Does no one of them could be used instead with additional wrapping or special configs or additional plugins/extensions? So that is would left just to implement the integration layer instead of build MQ solution from scratch? 2. Even RabbitMQ has a lot of things pluggable and just maybe all those requirements can be met by a set of RabbitMQ plugins. It is also possible to contribute something to RabbitMQ or any other OSS MQ. 3. I'm not sure that noisy neighbor problem and other QoS related issues are a big problem here. The question is who is the consumer of MQ? If consumers are user-land applications running on OpenStack instance VMs a better solution could be just to set dedicated RabbitMQ/somethingMQ on a VM for those apps instead of a single shared server. I do understand that a common shared resource is easier to mange and administrate. On the other hand it is less secure and subject to QoS issues. There are so many software configuration management tools around (docker containers and container managers, prebuilt images, puppet manifests and so on) that it is not a real challenge any more. And if consumers are other OpenStack services QoS is unlikely to be a problem because in each OpenStack distribution there are fixed number of tested services with known shape of MQ workload. 4. Another option could be just to bring new MQ broker per tenant. For example to run docker RabbitMQ container per tenant + something small to manage them (to sync containers with tenant list). Docker has resource quotas that can be used. 5. Stable API is good. However open and standardized API like AMQP is even better. Maybe all of those possible solutions are bad and I'm sure there are downsides to each approach. It is just not obvious that development of yet another MQ solution worth it. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
it seems like the trove/zaqar guys are talking about a very similar topic at the exact same time. :/ Maybe it would be a good time for all parties to get together and chat? Thanks, Kevin From: Stan Lagun [sla...@mirantis.com] Sent: Tuesday, May 12, 2015 11:52 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action On Wed, May 13, 2015 at 2:46 AM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: Awesome. When is it/where do I go to look up that info? http://sched.co/3Clo Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
+1. Is it a separate e-mail thread or IRC meeting? Thanks Gosha On Wed, May 13, 2015 at 10:21 AM, Fox, Kevin M kevin@pnnl.gov wrote: it seems like the trove/zaqar guys are talking about a very similar topic at the exact same time. :/ Maybe it would be a good time for all parties to get together and chat? Thanks, Kevin -- *From:* Stan Lagun [sla...@mirantis.com] *Sent:* Tuesday, May 12, 2015 11:52 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action On Wed, May 13, 2015 at 2:46 AM, Fox, Kevin M kevin@pnnl.gov wrote: Awesome. When is it/where do I go to look up that info? http://sched.co/3Clo Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On Wed, May 13, 2015 at 2:46 AM, Fox, Kevin M kevin@pnnl.gov wrote: Awesome. When is it/where do I go to look up that info? http://sched.co/3Clo Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Zane, Fully agree with you vision here. On 12 May 2015, at 07:15, Zane Bitter zbit...@redhat.com wrote: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. Any volunteers? * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] I added an agenda item for the summit in https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral to discuss this. Everyone is welcome. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Cool picture of a wonderful future :) Thanks for reading. Please grab me at summit if you want to know more; I am always happy to bend the ear of anyone who will listen at length on this topic. As usual, I'll be the tall dude with the weird accent ;) With the great pleasure. (P.S. your accent is cool!) Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Thanks, Kevin From: Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] Sent: Tuesday, May 12, 2015 10:06 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.commailto:zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we work together to come up with consistent solutions to these problems that keep popping up across OpenStack, we can prove wrong all the naysayers who think that application-facing services are only for proprietary clouds. I wrote up my vision for why that's important and what there first steps are here: http://www.zerobanana.com/archive/2015/04/24#a-vision
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we work together to come up with consistent solutions to these problems that keep popping up across OpenStack, we can prove wrong all the naysayers who think that application-facing services are only for proprietary clouds. I wrote up my vision for why that's important and what there first steps are here: http://www.zerobanana.com/archive/2015/04/24#a-vision-for-openstack Note that there are some subtleties that not everyone here will be able to contribute directly to fixing. For example, as I highlight in that post, Keystone is built around the concept that applications never talk to the cloud. But there are lots of other things people can work on now that would really make a big difference. For Mistral and Murano specifically, and in rough order of priority: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. * Encourage any deployers and distributors you know (or, ahem, may work for ;) to make Zaqar available as an option. * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] * Make Zaqar
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
+1 for Keystone based solution. Keystone is an identity service for OpenStack so we have to make sure that Keystone supports our use case for authentication and authorization for clients inside VMs. Thanks Gosha On Tue, May 12, 2015 at 10:32 AM, Fox, Kevin M kevin@pnnl.gov wrote: Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Thanks, Kevin -- *From:* Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] *Sent:* Tuesday, May 12, 2015 10:06 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we work together to come up with consistent solutions to these problems that keep popping up
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
We are going to have a design session at the summit entirely dedicated to Murano guest agent security and isolation. Everyone is welcomed to attend and discuss security requirements, concerns and possible solutions. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, May 12, 2015 at 8:32 PM, Fox, Kevin M kevin@pnnl.gov wrote: Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Thanks, Kevin -- *From:* Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] *Sent:* Tuesday, May 12, 2015 10:06 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Awesome. When is it/where do I go to look up that info? Thanks, Kevin From: Stan Lagun [sla...@mirantis.com] Sent: Tuesday, May 12, 2015 1:33 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action We are going to have a design session at the summit entirely dedicated to Murano guest agent security and isolation. Everyone is welcomed to attend and discuss security requirements, concerns and possible solutions. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com On Tue, May 12, 2015 at 8:32 PM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Thanks, Kevin From: Georgy Okrokvertskhov [gokrokvertsk...@mirantis.commailto:gokrokvertsk...@mirantis.com] Sent: Tuesday, May 12, 2015 10:06 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.commailto:zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
+1 for making Murano Engine - Murano Agent communication plugable so that one can switch to Zaqar or anything else. However watching RabbitMQ development for years I know hard can it be to build efficient and reliable system and I'm just not sure Zaqar can compete with such battle-proven thing like RabbitMQ yet. The only advantage I see is multi-tenancy. But I do believe it can be relatively easy be implemented with RabbitMQ. At lease in Murano. Don't want to go off topic here. The main idea is to use https://github.com/rabbitmq/rabbitmq-auth-backend-amqp and dynamically grant agent permissions only to his dedicated input queue so that it cannot access anything else. Not just other tenants queues but also queues of other VMs in the same tenant. In case of Murano we will not need to maintain additional secrets or databases. Neither it will be needed to create RabbitMQ users/vhosts as all of this becomes virtual. And agent will not be holding any RabbitMQ passwords at all Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, May 12, 2015 at 10:52 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Zane, Fully agree with you vision here. On 12 May 2015, at 07:15, Zane Bitter zbit...@redhat.com wrote: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. Any volunteers? * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] I added an agenda item for the summit in https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral to discuss this. Everyone is welcome. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Cool picture of a wonderful future :) Thanks for reading. Please grab me at summit if you want to know more; I am always happy to bend the ear of anyone who will listen at length on this topic. As usual, I'll be the tall dude with the weird accent ;) With the great pleasure. (P.S. your accent is cool!) Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
I would encourage everyone to listen Zane. I think Zaqar is the way to go for Murano. It is not the first time when someone asks as to switch MQ technologies. Remember our journey with ZeroMQ? Lets meet with Zaqar team on the summit and discuss what we can do together. As I remember Zaqar even had AMQ support on their roadmap but nothing prevents us to use their HTTP based client. It will be also great if we can share the client with Heat. I don't see any reason why we can't use one generic agent for software orchestration side. The only different between Heat and Murano there is in declarative vs. imperative approach. Most of the time Heat software orchestration is enough but in some cases like executing the same script multiple times on master node to register minions imperative approach is just more flexible. Thanks Gosha On Tue, May 12, 2015 at 6:44 AM, Stan Lagun sla...@mirantis.com wrote: +1 for making Murano Engine - Murano Agent communication plugable so that one can switch to Zaqar or anything else. However watching RabbitMQ development for years I know hard can it be to build efficient and reliable system and I'm just not sure Zaqar can compete with such battle-proven thing like RabbitMQ yet. The only advantage I see is multi-tenancy. But I do believe it can be relatively easy be implemented with RabbitMQ. At lease in Murano. Don't want to go off topic here. The main idea is to use https://github.com/rabbitmq/rabbitmq-auth-backend-amqp and dynamically grant agent permissions only to his dedicated input queue so that it cannot access anything else. Not just other tenants queues but also queues of other VMs in the same tenant. In case of Murano we will not need to maintain additional secrets or databases. Neither it will be needed to create RabbitMQ users/vhosts as all of this becomes virtual. And agent will not be holding any RabbitMQ passwords at all Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, May 12, 2015 at 10:52 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Zane, Fully agree with you vision here. On 12 May 2015, at 07:15, Zane Bitter zbit...@redhat.com wrote: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. Any volunteers? * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] I added an agenda item for the summit in https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral to discuss this. Everyone is welcome. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Cool picture of a wonderful future :) Thanks for reading. Please grab me at summit if you want to know more; I am always happy to bend the ear of anyone who will listen at length on this topic. As usual, I'll be the tall dude with the weird accent ;) With the great pleasure. (P.S. your accent is cool!) Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
+1 From: Zane Bitter [zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we work together to come up with consistent solutions to these problems that keep popping up across OpenStack, we can prove wrong all the naysayers who think that application-facing services are only for proprietary clouds. I wrote up my vision for why that's important and what there first steps are here: http://www.zerobanana.com/archive/2015/04/24#a-vision-for-openstack Note that there are some subtleties that not everyone here will be able to contribute directly to fixing. For example, as I highlight in that post, Keystone is built around the concept that applications never talk to the cloud. But there are lots of other things people can work on now that would really make a big difference. For Mistral and Murano specifically, and in rough order of priority: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. * Encourage any deployers and distributors you know (or, ahem, may work for ;) to make Zaqar available as an option. * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] * Make Zaqar an alternative to Rabbit for communicating to the Murano agent. * Use your experience in implementing notifications over email and the like in Mistral to help the Zaqar team to add the notification features they've long been planning. These could take the form of microservices listening on a Zaqar queue. You get the reliable, asynchronous queuing semantics for free and *every* service and user can benefit from your work. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hi Stan we wanted interact with murano applications from mistral. Currently there is no support in mistral how to execute scripts on VM via murano agent (maybe I miss something). We noticed std.ssh mistral action so we consider SSH as one of the options. I think that it is not good idea due to networking obstacles I just wanted to confirm that. Thanks for pointing out Zaquar I didn't know about it. Filip On 05/10/2015 04:21 AM, Stan Lagun wrote: Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com mailto:rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov mailto:kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hi there is VPN mechanism in neutron we could consider for future how to get around these networking obstacles if we would like to use direct SSH. 1) every private created by murano would create VPN gateway on public interface of the router [1] neutron vpn-service-create --name myvpn --description My vpn service router1 mysubnet 2) any service like mistral which needs directly access VM via SSH (or other protocols) would connect to that VPN and then it could directly access VM on its fixed IP This mechanism would probably resolve network obstacles. But it requires more effort to analyse it. [1] https://wiki.openstack.org/wiki/Neutron/VPNaaS/HowToInstall Filip On 05/08/2015 10:22 AM, Renat Akhmerov wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Stan, At the beginning we considered whether we could implement action on a murano application via mistral workflow. We thought that it could be beneficial to use workflow engine to implement some non-trivial action e.g. reconfiguration of some complex application within murano environment. Of course there would be disadvantages like dependency on mistral. I would recommend to discuss it later on some meeting (IRC, hangouts or summit) with Radek to explain use-cases better. Filip On 05/11/2015 10:44 AM, Stan Lagun wrote: Filip, Currently there is no support in mistral how to execute scripts on VM via murano agent Mistral can call Murano application action that will do the job via agent. Actions are intended to be called by 3rd party systems with single HTTP request Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis On Mon, May 11, 2015 at 11:27 AM, Filip Blaha filip.bl...@hp.com mailto:filip.bl...@hp.com wrote: Hi there is VPN mechanism in neutron we could consider for future how to get around these networking obstacles if we would like to use direct SSH. 1) every private created by murano would create VPN gateway on public interface of the router [1] neutron vpn-service-create --name myvpn --description My vpn service router1 mysubnet 2) any service like mistral which needs directly access VM via SSH (or other protocols) would connect to that VPN and then it could directly access VM on its fixed IP This mechanism would probably resolve network obstacles. But it requires more effort to analyse it. [1] https://wiki.openstack.org/wiki/Neutron/VPNaaS/HowToInstall Filip On 05/08/2015 10:22 AM, Renat Akhmerov wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin Mkevin@pnnl.gov mailto:kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe mailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Filip, Currently there is no support in mistral how to execute scripts on VM via murano agent Mistral can call Murano application action that will do the job via agent. Actions are intended to be called by 3rd party systems with single HTTP request Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, May 11, 2015 at 11:27 AM, Filip Blaha filip.bl...@hp.com wrote: Hi there is VPN mechanism in neutron we could consider for future how to get around these networking obstacles if we would like to use direct SSH. 1) every private created by murano would create VPN gateway on public interface of the router [1] neutron vpn-service-create --name myvpn --description My vpn service router1 mysubnet 2) any service like mistral which needs directly access VM via SSH (or other protocols) would connect to that VPN and then it could directly access VM on its fixed IP This mechanism would probably resolve network obstacles. But it requires more effort to analyse it. [1] https://wiki.openstack.org/wiki/Neutron/VPNaaS/HowToInstall Filip On 05/08/2015 10:22 AM, Renat Akhmerov wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Sorry, I reread what I wrote and it came off.. harsher then I had intended. I don't mean I will never support it, simply that I can't support it as it is. I have security researchers that use this particular cloud, and don't want to give Murano a black eye if they discover something... I would help if I could, but unfortunatly, I already have my fingers in too many pies. If I do manage to get a bit of time, maybe I can help. we'll see. I'm not convinced it will be a simple thing to fix though. Rabbit wasn't build with multitenancy in mind, so fixing it right might take a fair amount of work. http polling is not a good long term solution. Uses more power/resources then necessary. Zaquar, being designed for multi tenancy messaging from the get go, seems like a good way to go. Unfortunatly, they still only support http polling I think, but have it on their roadmap to fix. Most of the hard work is switching over to Zaquar though. So once they add a non polling option, it should be relatively cheep to add support. Thanks, Kevin From: Stan Lagun [sla...@mirantis.com] Sent: Sunday, May 10, 2015 3:24 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Kevin, I do agree that lack of RabbitMQ multi-tenancy is a problem. However as this is developers mailing list I would suggest to contribute and make Murano guest agent secure for the benefit of all users and all existing applications instead of spending comparable amount of time developing intricate solutions that will make everything way more complex. If Murano application developers need to write additional Mistral workflow for each deployment step that would make application development be extremely harder and Murano be mostly useless. There are several approaches that can be taken to solve agent isolation problem and all of them are relatively easy to implement. This task is one of out top priorities for the Liberty and will be solved very soon anyway. Another approach that IMO also better than SSH is to use HOT software config that uses HTTP polling and doesn't suffer from lack of tenant isolation. I do want to see better Mistral integration in Murano as well as many other tools like puppet etc. And there are some good use cases for Mistral. But when it comes to the most basic things that Murano was designed to do from the foundation I want to make sure that Murano can do them the best way possible without requiring users to learn additional DSLs/tools or go extra step and involve additional services where not necessary. If something important is missing in Murano that makes usage of Mistral for deployment more attractive I'd rather focus on improving Murano and bringing those features to Murano. We can even use Mistral under the hood as long as we not make users to write both MuranoPL and Mistral DSL code for trivial things like service restart. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com On Sun, May 10, 2015 at 8:44 PM, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: Im planning on deploying murano but wont be supporting the murano guest agent. The lack of multi tenant security is a big problem I think. Thanks, Kevin From: Stan Lagun Sent: Saturday, May 09, 2015 7:21:17 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we work together to come up with consistent solutions to these problems that keep popping up across OpenStack, we can prove wrong all the naysayers who think that application-facing services are only for proprietary clouds. I wrote up my vision for why that's important and what there first steps are here: http://www.zerobanana.com/archive/2015/04/24#a-vision-for-openstack Note that there are some subtleties that not everyone here will be able to contribute directly to fixing. For example, as I highlight in that post, Keystone is built around the concept that applications never talk to the cloud. But there are lots of other things people can work on now that would really make a big difference. For Mistral and Murano specifically, and in rough order of priority: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. * Encourage any deployers and distributors you know (or, ahem, may work for ;) to make Zaqar available as an option. * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] * Make Zaqar an alternative to Rabbit for communicating to the Murano agent. * Use your experience in implementing notifications over email and the like in Mistral to help the Zaqar team to add the notification features they've long been planning. These could take the form of microservices listening on a Zaqar queue. You get the reliable, asynchronous queuing semantics for free and *every* service and user can benefit from your work. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Thanks for reading. Please grab me at summit if
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Kevin, I do agree that lack of RabbitMQ multi-tenancy is a problem. However as this is developers mailing list I would suggest to contribute and make Murano guest agent secure for the benefit of all users and all existing applications instead of spending comparable amount of time developing intricate solutions that will make everything way more complex. If Murano application developers need to write additional Mistral workflow for each deployment step that would make application development be extremely harder and Murano be mostly useless. There are several approaches that can be taken to solve agent isolation problem and all of them are relatively easy to implement. This task is one of out top priorities for the Liberty and will be solved very soon anyway. Another approach that IMO also better than SSH is to use HOT software config that uses HTTP polling and doesn't suffer from lack of tenant isolation. I do want to see better Mistral integration in Murano as well as many other tools like puppet etc. And there are some good use cases for Mistral. But when it comes to the most basic things that Murano was designed to do from the foundation I want to make sure that Murano can do them the best way possible without requiring users to learn additional DSLs/tools or go extra step and involve additional services where not necessary. If something important is missing in Murano that makes usage of Mistral for deployment more attractive I'd rather focus on improving Murano and bringing those features to Murano. We can even use Mistral under the hood as long as we not make users to write both MuranoPL and Mistral DSL code for trivial things like service restart. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sun, May 10, 2015 at 8:44 PM, Fox, Kevin M kevin@pnnl.gov wrote: Im planning on deploying murano but wont be supporting the murano guest agent. The lack of multi tenant security is a big problem I think. Thanks, Kevin -- *From:* Stan Lagun *Sent:* Saturday, May 09, 2015 7:21:17 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Im planning on deploying murano but wont be supporting the murano guest agent. The lack of multi tenant security is a big problem I think. Thanks, Kevin From: Stan Lagun Sent: Saturday, May 09, 2015 7:21:17 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis mailto:sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.govmailto:kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Thanks for confirmation, that trying direct from mistral ssh to VM via fixed IP is not good idea. Btw. It would probably not work even if mistral run on the same network node hosting the router for the tenant because neutron creates separate network namespace (ip netns qrouter-x) for each router and VMs are accessible only from that namespace not from default. Filip On 05/06/2015 06:31 PM, Georgy Okrokvertskhov wrote: On Wed, May 6, 2015 at 9:26 AM, Fox, Kevin M kevin@pnnl.gov mailto:kevin@pnnl.gov wrote: If your Mistral engine is on the same host as the network node hosting the router for the tenant, then it would probably work there are a lot of conditions in that statement though... Too many for my tastes. :/ While I dislike agents running in the vm's, this still might be a good use case for one... This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. From Murano's perspective though, maybe it shouldn't care. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? Maybe the OpenStack Deployer chooses what's the best route for their cloud? Thanks, Kevins +1 for MQ. That is the path which proved itself to be working in most of the cases. -1 for ssh as this is a big headache. Thanks, Gosha From: Filip Blaha [filip.bl...@hp.com mailto:filip.bl...@hp.com] Sent: Wednesday, May 06, 2015 8:42 AM To: openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 mailto:cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com http://www.mirantis.com/ Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hello, Ad ... The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. ) IMO it is defined by user what networks are accessible from VM – i.e., there can be several ‚public networks‘ Ad There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM.) And In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. Yes and no ☺ In case of SSH the direction is obvious – from Mistral to VM. But in case of MQ it is nearly the same, but both VM and Mistral are accessing the MQ – so the direction is Mistral to MQ, and VM to MQ. In this case it is important on what network the MQ is running – is MQ running on VM (managed by nova), or on O~S node? In both cases we have to solve how neutron network will be available to O~S node: · MQ is on VM (managed by nova) o VM with Murano agent has to be on the same network, or via router as MQ o Mistral (and of course Murano engine) has to be configured to have access to VM with MQ e.g., via floating IP, or manually configured namespaces ? · MQ is on O~S node o VM with Murano agent has to be configured to access ‚public network‘ with MQ o Mistral and (Murano engine) will have access to MQ (as they are running with all O~S nodes) Gosha) In production environment - do you have ‚management network‘ on which MQ, VMs-with-Murano-agent, and Murano-engine, Mistral are running ? Anyway I like more idea of using MQ for execution of actions (such as ssh) instead of direct ssh. Regards, Radek From: Georgy Okrokvertskhov [mailto:gokrokvertsk...@mirantis.com] Sent: Wednesday, May 06, 2015 6:40 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.commailto:filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5mailto:cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
yes. I agree that direction is important from only networking piont of view. Usually is more probable that VM on neutron network will be able to access O~S service ( VM -- rabbit) then opposite direction from O~S service to VM running on neutron network (mistral -- VM). Filip On 05/06/2015 06:39 PM, Georgy Okrokvertskhov wrote: Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.com mailto:filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org mailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 mailto:cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com http://www.mirantis.com/ Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Yes, Rabbit MQ is kind of shared. Each VM gets its own Queue which is dynamically created in MQ when application is being deployed. Technically we can create separate MQ users and virtual hosts for each VM, but this is an overkill for now. So by default it is just separate Queue with random generated name. If you want more protection you wiull have to change Murano default behaviour by adding a new vhost for each tenant. Thanks Gosha On Thu, May 7, 2015 at 10:26 AM, Fox, Kevin M kevin@pnnl.gov wrote: So... rabbit is not multitenant I think. You share a rabbit across multiple tenants's vms? How do you protect one tenant's vm's from getting commands sent to it by another tenant? Thanks, Kevin -- *From:* Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] *Sent:* Thursday, May 07, 2015 9:18 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hi, When we use Murano in production there is a MQ service which is running on OpenStack controllers but it listens on public interface. It means that both Murano which is running on OpenStack controllers and Agent on VMs have an access to this MQ via external (public) network. When Murano creates a new deployment it actually deploys a private network and attach it to the router which acts as a gateway to external networking. So it is specific application deployment topology which allows VMs to communicate with MA via external network. Thanks Gosha On Thu, May 7, 2015 at 1:28 AM, Filip Blaha filip.bl...@hp.com wrote: yes. I agree that direction is important from only networking piont of view. Usually is more probable that VM on neutron network will be able to access O~S service ( VM -- rabbit) then opposite direction from O~S service to VM running on neutron network (mistral -- VM). Filip On 05/06/2015 06:39 PM, Georgy Okrokvertskhov wrote: Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hi, When we use Murano in production there is a MQ service which is running on OpenStack controllers but it listens on public interface. It means that both Murano which is running on OpenStack controllers and Agent on VMs have an access to this MQ via external (public) network. When Murano creates a new deployment it actually deploys a private network and attach it to the router which acts as a gateway to external networking. So it is specific application deployment topology which allows VMs to communicate with MA via external network. Thanks Gosha On Thu, May 7, 2015 at 1:28 AM, Filip Blaha filip.bl...@hp.com wrote: yes. I agree that direction is important from only networking piont of view. Usually is more probable that VM on neutron network will be able to access O~S service ( VM -- rabbit) then opposite direction from O~S service to VM running on neutron network (mistral -- VM). Filip On 05/06/2015 06:39 PM, Georgy Okrokvertskhov wrote: Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
So... rabbit is not multitenant I think. You share a rabbit across multiple tenants's vms? How do you protect one tenant's vm's from getting commands sent to it by another tenant? Thanks, Kevin From: Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] Sent: Thursday, May 07, 2015 9:18 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hi, When we use Murano in production there is a MQ service which is running on OpenStack controllers but it listens on public interface. It means that both Murano which is running on OpenStack controllers and Agent on VMs have an access to this MQ via external (public) network. When Murano creates a new deployment it actually deploys a private network and attach it to the router which acts as a gateway to external networking. So it is specific application deployment topology which allows VMs to communicate with MA via external network. Thanks Gosha On Thu, May 7, 2015 at 1:28 AM, Filip Blaha filip.bl...@hp.commailto:filip.bl...@hp.com wrote: yes. I agree that direction is important from only networking piont of view. Usually is more probable that VM on neutron network will be able to access O~S service ( VM -- rabbit) then opposite direction from O~S service to VM running on neutron network (mistral -- VM). Filip On 05/06/2015 06:39 PM, Georgy Okrokvertskhov wrote: Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.commailto:filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5mailto:cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hi, From Murano experience I can tell you that ssh to VM in general case will not work. In order to have an ssh access you will have to assign floating IPs so that Mistral service will be able to connect to VM. That is exactly the reason why Murano uses agent and MQ mechanism when client on VM initiates a connection. I believe the same issue was in Sahara when they used direct ssh connections to VMs. Thanks Gosha On Wed, May 6, 2015 at 9:00 AM, Pospisil, Radek radek.pospi...@hp.com wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
If your Mistral engine is on the same host as the network node hosting the router for the tenant, then it would probably work there are a lot of conditions in that statement though... Too many for my tastes. :/ While I dislike agents running in the vm's, this still might be a good use case for one... This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. From Murano's perspective though, maybe it shouldn't care. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? Maybe the OpenStack Deployer chooses what's the best route for their cloud? Thanks, Kevin From: Filip Blaha [filip.bl...@hp.com] Sent: Wednesday, May 06, 2015 8:42 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Connection direction here is important only in the frame of networking connectivity problem solving. The networking in OpenStack in general works in such a way so that connections from VM are allowed to almost anywhere. In Murano production deployment we use separate MQ instance so that VMs have no access to OpenStack MQ. In the sense who initiates task execution it always a Murano service which publishes tasks (shell script + necessary files) in the MQ so that agent can pull them and execute. Thanks Gosha On Wed, May 6, 2015 at 9:31 AM, Filip Blaha filip.bl...@hp.com wrote: Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On Wed, May 6, 2015 at 9:26 AM, Fox, Kevin M kevin@pnnl.gov wrote: If your Mistral engine is on the same host as the network node hosting the router for the tenant, then it would probably work there are a lot of conditions in that statement though... Too many for my tastes. :/ While I dislike agents running in the vm's, this still might be a good use case for one... This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. From Murano's perspective though, maybe it shouldn't care. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? Maybe the OpenStack Deployer chooses what's the best route for their cloud? Thanks, Kevins +1 for MQ. That is the path which proved itself to be working in most of the cases. -1 for ssh as this is a big headache. Thanks, Gosha From: Filip Blaha [filip.bl...@hp.com] Sent: Wednesday, May 06, 2015 8:42 AM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hello one more note on that. There is difference in direction who initiates connection. In case of murano agent -- rabbit MQ is connection initiated from VM to openstack service(rabbit). In case of std.ssh mistral action is direction opposite from openstack service (mistral) to ssh server on VM. Filip On 05/06/2015 06:00 PM, Pospisil, Radek wrote: Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Murano] [Mistral] SSH workflow action
Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Hello, I think that the generic question is - can be O~S services also accessible on Neutron networks, so VM (created by Nova) can access it? We (I and Filip) were discussing this today and we were not make a final decision. Another example is Murano agent running on VMs - it connects to RabbitMQ which is also accessed by Murano engine Regards, Radek -Original Message- From: Blaha, Filip Sent: Wednesday, May 06, 2015 5:43 PM To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello We are considering implementing actions on services of a murano environment via mistral workflows. We are considering whether mistral std.ssh action could be used to run some command on an instance. Example of such action in murano could be restart action on Mysql DB service. Mistral workflow would ssh to that instance running Mysql and run service mysql restart. From my point of view trying to use SSH to access instances from mistral workflow is not good idea but I would like to confirm it. The biggest problem I see there is openstack networking. Mistral service running on some openstack node would not be able to access instance via its fixed IP (e.g. 10.0.0.5) via SSH. Instance could accessed via ssh from namespace of its gateway router e.g. ip netns exec qrouter-... ssh cirros@10.0.0.5 but I think it is not good to rely on implementation detail of neutron and use it. In multinode openstack deployment it could be even more complicated. In other words I am asking whether we can use std.ssh mistral action to access instances via ssh on theirs fixed IPs? I think no but I would like to confirm it. Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
