[openstack-dev] [Mistral][Heat] Feedback on the Mistral DSL
Hi Mistral folks, Congrats on getting the 0.0.2 release out. I had a look at Renat's screencast and the examples, and I wanted to share some feedback based on my experience with Heat. Y'all will have to judge for yourselves to what extent this experience is applicable to Mistral. (Assume that everything I know about it was covered in the screencast and you won't be far wrong.) The first thing that struck me looking at https://github.com/stackforge/mistral-extra/tree/master/examples/create_vm is that I have to teach Mistral how to talk to Nova. I can't overstate how surprising this is as a user, because Mistral is supposed to want to become a part of OpenStack. It should know how to talk to Nova! There is actually an existing DSL for interacting with OpenStack[1], and here's what the equivalent operation looks like: os server create $server_name --image $image_id --flavor $flavor_id --nic net-id=$network_id Note that this is approximately exactly 96.875% shorter (or 3200% shorter, if you're in advertising). This approach reminds me a bit of TOSCA, in the way that it requires you to define every node type before you use it. (Even TOSCA is moving away from this by developing a Simple Profile that includes the most common ones in the box - an approach I assume/hope you're considering also.) The stated reason for this is that they want TOSCA templates to run on any cloud regardless of its underlying features (rather than take a lowest-common-denominator approach, as other attempts at hybrid clouds have done). Contrast that with Heat, which is unapologetically an orchestration system *for OpenStack*. I note from the screencast that Mistral's stated mission is to: Provide a mechanism to define and execute tasks and workflows *in OpenStack clouds* (My emphasis.) IMO the design doesn't reflect the mission. You need to decide whether you are trying to build the OpenStack workflow DSL or the workflow DSL to end all workflow DSLs. That problem could be solved by including built-in definitions for core OpenStack service in a similar way to std.* (i.e. take the TOSCA Simple Profile approach), but I'm actually not sure that goes far enough. The lesson of Heat is that we do best when we orchestrate *only* OpenStack APIs. For example, when we started working on Heat, there was no autoscaling in OpenStack so we implemented it ourselves inside Heat. Two years later, there's still no autoscaling in OpenStack other than what we implemented, and we've been struggling for a year to try to split Heat's implementation out into a separate API so that everyone can use it. Looking at things like std.email, I feel a similar way about them. OpenStack is missing something equivalent to SNS, where a message on a queue can trigger an email or another type of notification, and a lot of projects are going to eventually need something like that. It would be really unfortunate if all of them went out and invented it independently. It's much better to implement such things as their own building blocks that can be combined together in complex ways rather than adding that complexity to a bunch of services. Such a notification service could even be extended to do std.http-like ReST calls, although personally the whole idea of OpenStack services calling out to arbitrary HTTP APIs makes me extremely uncomfortable. Much better IMO to just post messages to queues and let the receiver (long) poll for it. So I would favour a DSL that is *much* simpler, and replaces all of std.* with functions that call OpenStack APIs, and only OpenStack APIs, including the API for posting messages to Marconi queues, which would be the method of communication to the outside world. (If the latter part sounds a bit like SWF, it's for a good reason, but the fact that it would allow access directly to all of the OpenStack APIs before resorting to an SDK makes it much more powerful, as well as providing a solid justification for why this should be part of OpenStack.) The ideal way to get support for all of the possible OpenStack APIs would be to do it by introspection on python-openstackclient. That means you'd only have to do the work once and it will stay up to date. This would avoid the problem we have in Heat, where we have to implement each resource type separately. (This is the source of a great deal of Heat's value to users - the existence of tested resource plugins - but also the thing that stops us from iterating the code quicker.) I'm also unsure that it's a good idea for things like timers to be set up inside the DSL. I would prefer that the DSL just define workflows and export entry points to them. Then have various ways to trigger them: from the API manually, from a message to a Marconi queue, from a timer, c. The latter two you'd set up through the Mistral API. If a user wanted a single document that set up one or more workflows and their triggers, a Heat template
Re: [openstack-dev] [Mistral][Heat] Feedback on the Mistral DSL
Hi Zane, Great feedback! My comments below... On 07 May 2014, at 22:29, Zane Bitter zbit...@redhat.com wrote: The first thing that struck me looking at https://github.com/stackforge/mistral-extra/tree/master/examples/create_vm is that I have to teach Mistral how to talk to Nova. I can't overstate how surprising this is as a user, because Mistral is supposed to want to become a part of OpenStack. It should know how to talk to Nova! There is actually an existing DSL for interacting with OpenStack[1], and here's what the equivalent operation looks like: os server create $server_name --image $image_id --flavor $flavor_id --nic net-id=$network_id Note that this is approximately exactly 96.875% shorter (or 3200% shorter, if you're in advertising). This approach reminds me a bit of TOSCA, in the way that it requires you to define every node type before you use it. (Even TOSCA is moving away from this by developing a Simple Profile that includes the most common ones in the box - an approach I assume/hope you're considering also.) The stated reason for this is that they want TOSCA templates to run on any cloud regardless of its underlying features (rather than take a lowest-common-denominator approach, as other attempts at hybrid clouds have done). Contrast that with Heat, which is unapologetically an orchestration system *for OpenStack*. Fully understandable. Originally we did want to build a really generic service with minimal knowledge about an infrastructure in which it would work. And we wanted to build something really simple and were afraid of blowing the framework with too many features related to OpenStack services. Because if we had started adding, for example, Nova actions then we would have probably had to include others like Cinder, Neutron etc. etc. and the question was “Where should we stop? Where is the line that we shouldn’t cross?”. However, since then our opinion (at least my personal) has changed regarding this question and that’s we we’ve designed our action subsystem easy to enable plugins. So along with “std” namespace that is given out of the box there will be others like “nova”, may be implemented as a separate additional project for better arch decomposition but that’s a different question.. This work is not finished yet, one of the things we wanted to do is to gather as much feedback as possible before the summit and at the summit itself but, I’m now 99% sure we need to move forward to having all the action packs to enable users to natively use all core OpenStack services. So basically I agree with you on that one. What you saw in the demo was mostly intended to demonstrate how you can extend existing actions by describing adapters to them right in DSL. But I guess it’s my fault that I didn’t deliver that message. I note from the screencast that Mistral's stated mission is to: Provide a mechanism to define and execute tasks and workflows *in OpenStack clouds* (My emphasis.) IMO the design doesn't reflect the mission. You need to decide whether you are trying to build the OpenStack workflow DSL or the workflow DSL to end all workflow DSLs. That problem could be solved by including built-in definitions for core OpenStack service in a similar way to std.* (i.e. take the TOSCA Simple Profile approach), but I'm actually not sure that goes far enough. The lesson of Heat is that we do best when we orchestrate *only* OpenStack APIs. Yes, it is related with my previous comment. I really do believe we shouldn’t be trying to build a OpenStack agnostic workflow DSL (and service). Excessive generalization is evil as well as excessive specialization so we need to find a good balance. But we’re definitely building a workflow service for OpenStack and are intended to moving to making OpenStack services to easy and natively use in Mistral. And please note that this part just wasn’t the main focus of the first phase of development. We rather wanted to try out the approach itself and show the community what we got as a result of our research. Even though the release version is 0.0.2 (for purely technical reasons) we still consider it sort of a PoC in a sense that we may want to change DSL and design pretty significantly. For example, when we started working on Heat, there was no autoscaling in OpenStack so we implemented it ourselves inside Heat. Two years later, there's still no autoscaling in OpenStack other than what we implemented, and we've been struggling for a year to try to split Heat's implementation out into a separate API so that everyone can use it. Looking at things like std.email, I feel a similar way about them. OpenStack is missing something equivalent to SNS, where a message on a queue can trigger an email or another type of notification, and a lot of projects are going to eventually need something like that. It would be really unfortunate if all of them went out and
