Winson, Renat, I have a few questions, because some of aspects aren't clear to me.
1) How does the end user will pass env variables to workflow?Will you add one more optional parameter to execution-create command? mistral execution-create wf wf_input wf_params wf_env If yes than what is wf_env will be, json file? 2) Retuning to first example: ... action: std.sql conn_str={$.env.conn_str} query={$.query} ... $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? 3) Can user has a few environments? On Wed, Dec 24, 2014 at 8:20 AM, Renat Akhmerov <rakhme...@mirantis.com> wrote: > Thanks Winson, > > Since we discussed all this already I just want to confirm that I fully > support this model, it will significantly help us make much more concise, > readable and maintainable workflows. I spent a lot of time thinking about > it and don’t see any problems with it. Nice job! > > However, all additional comments and questions are more than welcomed! > > > Renat Akhmerov > @ Mirantis Inc. > > > > On 24 Dec 2014, at 04:32, W Chan <m4d.co...@gmail.com> wrote: > > After some online discussions with Renat, the following is a revision of > the proposal to address the following related blueprints. > * > https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment > * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context > * > https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values > * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context > > Please refer to the following threads for backgrounds. > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html > > > *Workflow Context Scope* > 1. context to workflow is passed to all its subflows and subtasks/actions > (aka children) only explicitly via inputs > 2. context are passed by value (copy.deepcopy) to children > 3. change to context is passed to parent only when it's explicitly > published at the end of the child execution > 4. change to context at the parent (after a publish from a child) is > passed to subsequent children > > *Environment Variables* > Solves the problem for quickly passing pre-defined inputs to a WF > execution. In the WF spec, environment variables are referenced as > $.env.var1, $.env.var2, etc. We should implement an API and DB model > where users can pre-defined different environments with their own set of > variables. An environment can be passed either by name from the DB or > adhoc by dict in start_workflow. On workflow execution, a copy of the > environment is saved with the execution object. Action inputs are still > declared explicitly in the WF spec. This does not solve the problem where > common inputs are specified over and over again. So if there are multiple > SQL tasks in the WF, the WF author still needs to supply the conn_str > explicitly for each task. In the example below, let's say we have a SQL > Query Action that takes a connection string and a query statement as > inputs. The WF author can specify that the conn_str input is supplied from > the $.env.conn_str. > > *Example:* > > # Assume this SqlAction is registered as std.sql in Mistral's Action table. > class SqlAction(object): > def __init__(self, conn_str, query): > ... > > ... > > version: "2.0" > workflows: > demo: > type: direct > input: > - query > output: > - records > tasks: > query: > action: std.sql conn_str={$.env.conn_str} query={$.query} > publish: > records: $ > > ... > > my_adhoc_env = { > "conn_str": "mysql://admin:secrete@localhost/test" > } > > ... > > # adhoc by dict > start_workflow(wf_name, wf_inputs, env=my_adhoc_env) > > OR > > # lookup by name from DB model > start_workflow(wf_name, wf_inputs, env="my_lab_env") > > > *Define Default Action Inputs as Environment Variables* > Solves the problem where we're specifying the same inputs to subflows and > subtasks/actions over and over again. On command execution, if action > inputs are not explicitly supplied, then defaults will be lookup from the > environment. > > *Example:* > Using the same example from above, the WF author can still supply both > conn_str and query inputs in the WF spec. However, the author also has the > option to supply that as default action inputs. An example environment > structure is below. "__actions" should be reserved and immutable. Users > can specific one or more default inputs for the sql action as nested dict > under "__actions". Recursive YAQL eval should be supported in the env > variables. > > version: "2.0" > workflows: > demo: > type: direct > input: > - query > output: > - records > tasks: > query: > action: std.sql query={$.query} > publish: > records: $ > > ... > > my_adhoc_env = { > "sql_server": "localhost", > "__actions": { > "std.sql": { > "conn_str": "mysql://admin:secrete@{$.env.sql_server}/test" > } > } > } > > > *Default Input Values Supplied Explicitly in WF Spec* > Please refer to this blueprint > <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values> > for background. This is a different use case. To support, we just need to > set the correct order of precedence in applying values. > 1. Input explicitly given to the sub flow/task in the WF spec > 2. Default input supplied from env > 3. Default input supplied at WF spec > > *Putting this together...* > At runtime, the WF context would be similar to the following example. > This will be use to recursively eval the inputs for subflow/tasks/actions. > > ctx = { > "var1": …, > "var2": …, > "my_server_ip": "10.1.23.250", > "env": { > "sql_server": "localhost", > "__actions": { > "std.sql": { > "conn": "mysql://admin:secrete@{$.env.sql_server}/test" > }, > "my.action": { > "endpoint": "http://{$.my_server_ip}/v1/foo" > } > } > } > } > > *Runtime Context* > Please refer to this thread > <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html> > for > the background and discussion. The only change here is that on run_action, > we will pass the runtime data as kwargs to all action invocation. This > means all Action classes should have at least **kwargs added to the > __init__ method. > > runtime = { > "execution_id": ..., > "task_id": ..., > ... > } > > ... > > action = SomeAction(..., runtime=runtime) > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev