On Tue, 2016-01-12 at 13:24 -0500, Ryan Brown wrote: > On 01/12/2016 10:50 AM, Jiri Tomasek wrote: > > On 01/12/2016 04:22 PM, Ryan Brown wrote: > > > On 01/12/2016 06:52 AM, Jiri Tomasek wrote: > > > > On 01/11/2016 04:51 PM, Dan Prince wrote: > > > > > Background info: > > > > > > > > > > We've got a problem in TripleO at the moment where many of > > > > > our > > > > > workflows can be driven by the command line only. This causes > > > > > some > > > > > problems for those trying to build a UI around the workflows > > > > > in that > > > > > they have to duplicate deployment logic in potentially > > > > > multiple places. > > > > > There are specs up for review which outline how we might > > > > > solve this > > > > > problem by building what is called TripleO API [1]. > > > > > > > > > > Late last year I began experimenting with an OpenStack > > > > > service called > > > > > Mistral which contains a generic workflow API. Mistral > > > > > supports > > > > > defining workflows in YAML and then creating, managing, and > > > > > executing > > > > > them via an OpenStack API. Initially the effort was focused > > > > > around the > > > > > idea of creating a workflow in Mistral which could supplant > > > > > our > > > > > "baremetal introspection" workflow which currently lives in > > > > > python- > > > > > tripleoclient. I create a video presentation which outlines > > > > > this effort > > > > > [2]. This particular workflow seemed to fit nicely within the > > > > > Mistral > > > > > tooling. > > > > > > > > > > ---- > > > > > > > > > > More recently I've turned my attention to what it might look > > > > > like if we > > > > > were to use Mistral as a replacement for the TripleO API > > > > > entirely. This > > > > > brings forth the question of would TripleO be better off > > > > > building out > > > > > its own API... or would relying on existing OpenStack APIs be > > > > > a better > > > > > solution? > > > > > > > > > > Some things I like about the Mistral solution: > > > > > > > > > > - The API already exists and is generic. > > > > > > > > > > - Mistral already supports interacting with many of the > > > > > OpenStack API's > > > > > we require [3]. Integration with keystone is baked in. Adding > > > > > support > > > > > for new clients seems straightforward (I've had no issues in > > > > > adding > > > > > support for ironic, inspector, and swift actions). > > > > > > > > > > - Mistral actions are pluggable. We could fairly easily wrap > > > > > some of > > > > > our more complex workflows (perhaps those that aren't easy to > > > > > replicate > > > > > with pure YAML workflows) by creating our own TripleO Mistral > > > > > actions. > > > > > This approach would be similar to creating a custom Heat > > > > > resource... > > > > > something we have avoided with Heat in TripleO but I think it > > > > > is > > > > > perhaps more reasonable with Mistral and would allow us to > > > > > again build > > > > > out our YAML workflows to drive things. This might allow us > > > > > to build > > > > > off some of the tripleo-common consolidation that is already > > > > > underway > > > > > ... > > > > > > > > > > - We could achieve a "stable API" by simply maintaining input > > > > > parameters for workflows in a stable manner. Or perhaps > > > > > workflows get > > > > > versioned like a normal API would be as well. > > > > > > > > > > - The purist part of me likes Mistral quite a bit. It fits > > > > > nicely with > > > > > the deploy OpenStack with OpenStack. I sort of feel like if > > > > > we have to > > > > > build our own API in TripleO part of this vision has failed > > > > > and could > > > > > even be seen as a massive technical debt which would likely > > > > > be hard to > > > > > build a community around outside of TripleO. > > > > > > > > > > - Some of the proposed validations could perhaps be > > > > > implemented as new > > > > > Mistral actions as well. I'm not convinced we require TripleO > > > > > API just > > > > > to support a validations mechanism yet. Perhaps validations > > > > > seem hard > > > > > because we are simply trying to do them in the wrong places > > > > > anyway? > > > > > (like for example perhaps we should validate network > > > > > connectivity at > > > > > inspection time rather than during provisioning). > > > > > > > > > > - Power users might find a workflow built around a Mistral > > > > > API more > > > > > easy to interact with and expand upon. Perhaps this ends up > > > > > being > > > > > something that gets submitted as a patchset back to the > > > > > TripleO that we > > > > > accept into our upstream "stock" workflow sets. > > > > > > > > > > ---- > > > > > > > > > > Last week we landed the last patches [4] to our undercloud to > > > > > enable > > > > > installing Mistral by simply setting: enable_mistral = true > > > > > in > > > > > undercloud.conf. NOTE: you'll need to be using a recent trunk > > > > > repo from > > > > > Delorean so that you have the recently added Mistral packages > > > > > for this > > > > > to work. Although the feature is disable by default this > > > > > should enable > > > > > those wishing to tinker with Mistral as a new TripleO > > > > > undercloud > > > > > service an easy path forwards. > > > > > > > > > > [1] https://review.openstack.org/#/c/230432 > > > > > [2] https://www.youtube.com/watch?v=bnAT37O-sdw > > > > > [3] http://git.openstack.org/cgit/openstack/mistral/tree/mist > > > > > ral/action > > > > > s/openstack/mapping.json > > > > > [4] https://etherpad.openstack.org/p/tripleo-undercloud-workf > > > > > low > > > > > > > > > > > > > > > _____________________________________________________________ > > > > > _____________ > > > > > > > > > > > > > > > OpenStack Development Mailing List (not for usage questions) > > > > > Unsubscribe: > > > > > [email protected]?subject:unsubscribe > > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > -dev > > > > > > > > Hi, I have a few questions: > > > > > > > > Is Mistral action able to access/manipulate local files? E.g. > > > > access the > > > > templates installed at undercloud's > > > > /usr/share/openstack-tripleo-heat-templates? > > > > > > I believe with mistral there would be an intermediate step of > > > uploading the templates to Swift first. Heat can read templates > > > from > > > swift, and any mistral workflows would be able to read the > > > templates > > > out, modify them, and save back to swift. > > > > Correct, but from the Mistral usage standpoint, having the > > flexibility > > is a good thing IMO regardless of the example I have chosen. > > > > > > > > Object stores FTW > > > > > > > I Mistral action able to call either OpenStack service python > > > > client or > > > > OpenStack service API directly? > > > > > > > > What is the response from the Mistral action in the workflow? > > > > Lets say > > > > we'd use Mistral to get a list of available environments (we do > > > > this in > > > > tripleo-common now) So we call Mistral API to trigger a > > > > workflow that > > > > has single action which gets the list of environments. Is > > > > mistral able > > > > to provide this list as a response, or it is able just to > > > > trigger a > > > > workflow? > > > > > > > > In similar manner, is Mistral able to provide a way to get > > > > workflow in > > > > progress data? E.g. We have a Mistral workflow for nodes > > > > introspection. > > > > This workflow contains several actions calling to ironic and > > > > ironic-inspector apis. GUI calls Mistral API to trigger the > > > > workflow and > > > > now it needs to have a way to track the progress of that > > > > workflow. How > > > > would this be achieved? I think forcing GUI to poll the APIs > > > > that are > > > > called in the actions and try to implement logic that estimates > > > > what > > > > state the workflow is to report about it is not a valid > > > > solution. We > > > > need the workflow API (Mistral) to provide a lets say web > > > > sockets > > > > connection and push the status of actions along with relevant > > > > data, so > > > > GUI can listen to those. > > > > > > I can see a few options for progress polling/push. > > > > > > Mistral: > > > 1. Send job to REST API (we didn't have to build) > > > 1. It spins off as many jobs as it needs (we build) > > > 1. Poll mistral to see how many are done > > > > > > TripleO API > > > 1. Build the REST API (we build) > > > 1. Send requests to start introspection (we build) > > > 1. Poll TripleO API until things are done > > > > > > Mistral + Zaqar: > > > 1. Send job to REST API (we didn't have to build) > > > 1. It spins off as many jobs as it needs (we build) > > > 1. Send updates to Zaqar (we didn't have to build) > > > 1. Get websocket updates to the GUI from Zaqar (we didn't have to > > > build) > > > > > > Zaqar is (of course) designed to deliver updates like this, so > > > every > > > project on the face of the planet doesn't have to rebuild > > > websocket > > > notifications, which is a good thing. > > > > Thanks, this sounds very good. So the Mistral + Zaqar workflow > > means > > that as part of workflow action, Mistral notifies Zaqar that > > certain > > thing happened. Zaqar then notifies listening GUI providing > > information > > to identify what API call the GUI needs to make to retrieve > > relevant > > data or even provide the data itself (e.g. Mistral fails, some > > error > > occurs etc.) > > I don't know what the mistral-zaqar integration is at the moment, but > if > it's not there you could post to zaqar in the custom TripleO actions.
Exactly, This is what I had in mind as well. And if zaqarclient is in good shape integrating it w/ Mistral should be quite straightforward. > > > > > > > > I am about to implement your nodes workflow in the GUI to test > > > > how it > > > > works. > > > > > > > > Jirka > > > > > > > > _______________________________________________________________ > > > > ___________ > > > > > > > > OpenStack Development Mailing List (not for usage questions) > > > > Unsubscribe: > > > > [email protected]?subject:unsubscribe > > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d > > > > ev > > > > > > > Jirka > > > > ___________________________________________________________________ > > _______ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: [email protected]?subject:unsu > > bscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
