Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
On 27 Dec 2014, at 01:39, W Chan m4d.co...@gmail.com wrote: What you’re saying is that whatever is under “$.env” is just the exact same environment that we passed when we started the workflow? If yes then it definitely makes sense to me (it just allows to explicitly access environment, not through the implicit variable lookup). Please confirm. Yes. the $.env that I original proposed would be the same dict as the one supplied at start_workflow. Although we have to agree whether the variables in the environment are allowed to change after the WF started. Unless there's a valid use case, I would lean toward making env immutable. Let’s make them immutable. One thing that I strongly suggest is that we clearly define all reserved keys like “env”, “__actions” etc. I think it’d be better if they all started with the same prefix, for example, double underscore. Agree. How about using double underscore for env as well (i.e. $.__env.var1, $.__env.var2)? Yes. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
What you’re saying is that whatever is under “$.env” is just the exact same environment that we passed when we started the workflow? If yes then it definitely makes sense to me (it just allows to explicitly access environment, not through the implicit variable lookup). Please confirm. Yes. the $.env that I original proposed would be the same dict as the one supplied at start_workflow. Although we have to agree whether the variables in the environment are allowed to change after the WF started. Unless there's a valid use case, I would lean toward making env immutable. One thing that I strongly suggest is that we clearly define all reserved keys like “env”, “__actions” etc. I think it’d be better if they all started with the same prefix, for example, double underscore. Agree. How about using double underscore for env as well (i.e. $.__env.var1, $.__env.var2)? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
One thing that I strongly suggest is that we clearly define all reserved keys like “env”, “__actions” etc. I think it’d be better if they all started with the same prefix, for example, double underscore. I absolutely agree here. We should use specific keywords with __ prefix like we used __executions. On Thu, Dec 25, 2014 at 8:51 AM, Renat Akhmerov rakhme...@mirantis.com wrote: On 24 Dec 2014, at 23:37, W Chan m4d.co...@gmail.com wrote: * 2) Retuning to first example: ** ... ** action: std.sql conn_str={$.env.conn_str} query={$.query} ** ... ** $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? * I was actually thinking the environment will use the reserved word env in the WF context. The value for the env key will be the dict supplied either DB lookup by name, by dict, or by JSON from CLI. Ok, probably here’s the place where I didn’t understand you before. I thought “env” here is just a arbitrary key that users themselves may want to have to just group some variables under a single umbrella. What you’re saying is that whatever is under “$.env” is just the exact same environment that we passed when we started the workflow? If yes then it definitely makes sense to me (it just allows to explicitly access environment, not through the implicit variable lookup). Please confirm. One thing that I strongly suggest is that we clearly define all reserved keys like “env”, “__actions” etc. I think it’d be better if they all started with the same prefix, for example, double underscore. The nested dict for __actions (and all other keys with double underscore) is special system purpose, in this case declaring defaults for action inputs. Similar to __execution where it's for containing runtime data for the WF execution. Yes, that’s clear Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
Winson, Renat, I have a few questions, because some of aspects aren't clear to me. 1) How does the end user will pass env variables to workflow?Will you add one more optional parameter to execution-create command? mistral execution-create wf wf_input wf_params wf_env If yes than what is wf_env will be, json file? 2) Retuning to first example: ... action: std.sql conn_str={$.env.conn_str} query={$.query} ... $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? 3) Can user has a few environments? On Wed, Dec 24, 2014 at 8:20 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Thanks Winson, Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job! However, all additional comments and questions are more than welcomed! Renat Akhmerov @ Mirantis Inc. On 24 Dec 2014, at 04:32, W Chan m4d.co...@gmail.com wrote: After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints. * https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context Please refer to the following threads for backgrounds. * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html *Workflow Context Scope* 1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs 2. context are passed by value (copy.deepcopy) to children 3. change to context is passed to parent only when it's explicitly published at the end of the child execution 4. change to context at the parent (after a publish from a child) is passed to subsequent children *Environment Variables* Solves the problem for quickly passing pre-defined inputs to a WF execution. In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc. We should implement an API and DB model where users can pre-defined different environments with their own set of variables. An environment can be passed either by name from the DB or adhoc by dict in start_workflow. On workflow execution, a copy of the environment is saved with the execution object. Action inputs are still declared explicitly in the WF spec. This does not solve the problem where common inputs are specified over and over again. So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task. In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs. The WF author can specify that the conn_str input is supplied from the $.env.conn_str. *Example:* # Assume this SqlAction is registered as std.sql in Mistral's Action table. class SqlAction(object): def __init__(self, conn_str, query): ... ... version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql conn_str={$.env.conn_str} query={$.query} publish: records: $ ... my_adhoc_env = { conn_str: mysql://admin:secrete@localhost/test } ... # adhoc by dict start_workflow(wf_name, wf_inputs, env=my_adhoc_env) OR # lookup by name from DB model start_workflow(wf_name, wf_inputs, env=my_lab_env) *Define Default Action Inputs as Environment Variables* Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again. On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment. *Example:* Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec. However, the author also has the option to supply that as default action inputs. An example environment structure is below. __actions should be reserved and immutable. Users can specific one or more default inputs for the sql action as nested dict under __actions. Recursive YAQL eval should be supported in the env variables. version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql query={$.query} publish:
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
On 24 Dec 2014, at 14:06, Anastasia Kuznetsova akuznets...@mirantis.com wrote: 1) How does the end user will pass env variables to workflow?Will you add one more optional parameter to execution-create command? mistral execution-create wf wf_input wf_params wf_env If yes than what is wf_env will be, json file? Yes. IMO it should be possible to specify either a string (name of a previously stored environment) or a json file (so called ad-hoc environment). 2) Retuning to first example: ... action: std.sql conn_str={$.env.conn_str} query={$.query} ... $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? So far we agreed that ‘key' should not be a registered key. Environment (optionally specified) is just another storage of variables going after workflow context in a lookup chain. So that if somewhere in a wf we have an expression $.something then this “something” will be first looked in workflow context and if it doesn’t exist there then looked in the specified environment. But if we want to explicitly group a set of variables we can use any (except for reserved as __actions ) key, for example, “env”. 3) Can user has a few environments? Yes. That’s one of the goals of introducing a concept of environment. So that same workflows could be running in different environments (e.g with different email settings, any kinds of passports etc.). Renat Akhmerov @ Mirantis Inc.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
Renat, thanks for response! One more question: So that same workflows could be running in different environments Asking about using a few environments I meant within one workflow. For example I need to work with two DB and I have two environments: env1 = {conn_str: ip, user: user, password: passwd} and env2 = {conn_str: ip2, user: user2, password: passwd2}. Will it be possible to do something like this: tasks: connect_first_db: action: std.sql conn_str={$.env1.conn_str} query={$.query} publish: records: $ connect_second_db: action: std.sql conn_str={$.env2.conn_str} query={$.query} publish: records: $ Thanks, Anastasia Kuznetsova On Wed, Dec 24, 2014 at 2:19 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 24 Dec 2014, at 14:06, Anastasia Kuznetsova akuznets...@mirantis.com wrote: 1) How does the end user will pass env variables to workflow?Will you add one more optional parameter to execution-create command? mistral execution-create wf wf_input wf_params wf_env If yes than what is wf_env will be, json file? Yes. IMO it should be possible to specify either a string (name of a previously stored environment) or a json file (so called ad-hoc environment). 2) Retuning to first example: ... action: std.sql conn_str={$.env.conn_str} query={$.query} ... $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? So far we agreed that ‘key' should not be a registered key. Environment (optionally specified) is just another storage of variables going after workflow context in a lookup chain. So that if somewhere in a wf we have an expression $.something then this “something” will be first looked in workflow context and if it doesn’t exist there then looked in the specified environment. But if we want to explicitly group a set of variables we can use any (except for reserved as __actions ) key, for example, “env”. 3) Can user has a few environments? Yes. That’s one of the goals of introducing a concept of environment. So that same workflows could be running in different environments (e.g with different email settings, any kinds of passports etc.). Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
Trying to clarify a few things... * 2) Retuning to first example: ** ... ** action: std.sql conn_str={$.env.conn_str} query={$.query} ** ... ** $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? * I was actually thinking the environment will use the reserved word env in the WF context. The value for the env key will be the dict supplied either DB lookup by name, by dict, or by JSON from CLI. The nested dict for __actions (and all other keys with double underscore) is special system purpose, in this case declaring defaults for action inputs. Similar to __execution where it's for containing runtime data for the WF execution. * 3) Can user has a few environments?* I don't think we intend to mix one or more environments in a WF execution. The key was to supply any named environment at WF execution time. So the WF auth only needs to know the variables will be under $.env. If we allow one or more environments in a WF execution, this means each environment needs to be referred to by name (i.e. in your example env1 and env2). We then would lost the ability to swap any named environments for different executions of the same WF. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
On 24 Dec 2014, at 23:37, W Chan m4d.co...@gmail.com wrote: 2) Retuning to first example: ... action: std.sql conn_str={$.env.conn_str} query={$.query} ... $.env - is it a name of environment or it will be a registered syntax to getting access to values from env ? I was actually thinking the environment will use the reserved word env in the WF context. The value for the env key will be the dict supplied either DB lookup by name, by dict, or by JSON from CLI. Ok, probably here’s the place where I didn’t understand you before. I thought “env” here is just a arbitrary key that users themselves may want to have to just group some variables under a single umbrella. What you’re saying is that whatever is under “$.env” is just the exact same environment that we passed when we started the workflow? If yes then it definitely makes sense to me (it just allows to explicitly access environment, not through the implicit variable lookup). Please confirm. One thing that I strongly suggest is that we clearly define all reserved keys like “env”, “__actions” etc. I think it’d be better if they all started with the same prefix, for example, double underscore. The nested dict for __actions (and all other keys with double underscore) is special system purpose, in this case declaring defaults for action inputs. Similar to __execution where it's for containing runtime data for the WF execution. Yes, that’s clear Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints. * https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context Please refer to the following threads for backgrounds. * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html *Workflow Context Scope* 1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs 2. context are passed by value (copy.deepcopy) to children 3. change to context is passed to parent only when it's explicitly published at the end of the child execution 4. change to context at the parent (after a publish from a child) is passed to subsequent children *Environment Variables* Solves the problem for quickly passing pre-defined inputs to a WF execution. In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc. We should implement an API and DB model where users can pre-defined different environments with their own set of variables. An environment can be passed either by name from the DB or adhoc by dict in start_workflow. On workflow execution, a copy of the environment is saved with the execution object. Action inputs are still declared explicitly in the WF spec. This does not solve the problem where common inputs are specified over and over again. So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task. In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs. The WF author can specify that the conn_str input is supplied from the $.env.conn_str. *Example:* # Assume this SqlAction is registered as std.sql in Mistral's Action table. class SqlAction(object): def __init__(self, conn_str, query): ... ... version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql conn_str={$.env.conn_str} query={$.query} publish: records: $ ... my_adhoc_env = { conn_str: mysql://admin:secrete mysql://admin:secrete@@localhost /test } ... # adhoc by dict start_workflow(wf_name, wf_inputs, env=my_adhoc_env) OR # lookup by name from DB model start_workflow(wf_name, wf_inputs, env=my_lab_env) *Define Default Action Inputs as Environment Variables* Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again. On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment. *Example:* Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec. However, the author also has the option to supply that as default action inputs. An example environment structure is below. __actions should be reserved and immutable. Users can specific one or more default inputs for the sql action as nested dict under __actions. Recursive YAQL eval should be supported in the env variables. version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql query={$.query} publish: records: $ ... my_adhoc_env = { sql_server: localhost, __actions: { std.sql: { conn_str: mysql://admin:secrete@{$.env.sql_server}/test } } } *Default Input Values Supplied Explicitly in WF Spec* Please refer to this blueprint https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values for background. This is a different use case. To support, we just need to set the correct order of precedence in applying values. 1. Input explicitly given to the sub flow/task in the WF spec 2. Default input supplied from env 3. Default input supplied at WF spec *Putting this together...* At runtime, the WF context would be similar to the following example. This will be use to recursively eval the inputs for subflow/tasks/actions. ctx = { var1: …, var2: …, my_server_ip: 10.1.23.250, env: { sql_server: localhost, __actions: { std.sql: { conn: mysql://admin:secrete@{$.env.sql_server}/test }, my.action: { endpoint: http://{$.my_server_ip}/v1/foo; } } } } *Runtime Context*
Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs
Thanks Winson, Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job! However, all additional comments and questions are more than welcomed! Renat Akhmerov @ Mirantis Inc. On 24 Dec 2014, at 04:32, W Chan m4d.co...@gmail.com wrote: After some online discussions with Renat, the following is a revision of the proposal to address the following related blueprints. * https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context https://blueprints.launchpad.net/mistral/+spec/mistral-global-context * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context Please refer to the following threads for backgrounds. * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html * http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html Workflow Context Scope 1. context to workflow is passed to all its subflows and subtasks/actions (aka children) only explicitly via inputs 2. context are passed by value (copy.deepcopy) to children 3. change to context is passed to parent only when it's explicitly published at the end of the child execution 4. change to context at the parent (after a publish from a child) is passed to subsequent children Environment Variables Solves the problem for quickly passing pre-defined inputs to a WF execution. In the WF spec, environment variables are referenced as $.env.var1, $.env.var2, etc. We should implement an API and DB model where users can pre-defined different environments with their own set of variables. An environment can be passed either by name from the DB or adhoc by dict in start_workflow. On workflow execution, a copy of the environment is saved with the execution object. Action inputs are still declared explicitly in the WF spec. This does not solve the problem where common inputs are specified over and over again. So if there are multiple SQL tasks in the WF, the WF author still needs to supply the conn_str explicitly for each task. In the example below, let's say we have a SQL Query Action that takes a connection string and a query statement as inputs. The WF author can specify that the conn_str input is supplied from the $.env.conn_str. Example: # Assume this SqlAction is registered as std.sql in Mistral's Action table. class SqlAction(object): def __init__(self, conn_str, query): ... ... version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql conn_str={$.env.conn_str} query={$.query} publish: records: $ ... my_adhoc_env = { conn_str: mysql://admin:secrete mysql://admin:secrete@@localhost/test } ... # adhoc by dict start_workflow(wf_name, wf_inputs, env=my_adhoc_env) OR # lookup by name from DB model start_workflow(wf_name, wf_inputs, env=my_lab_env) Define Default Action Inputs as Environment Variables Solves the problem where we're specifying the same inputs to subflows and subtasks/actions over and over again. On command execution, if action inputs are not explicitly supplied, then defaults will be lookup from the environment. Example: Using the same example from above, the WF author can still supply both conn_str and query inputs in the WF spec. However, the author also has the option to supply that as default action inputs. An example environment structure is below. __actions should be reserved and immutable. Users can specific one or more default inputs for the sql action as nested dict under __actions. Recursive YAQL eval should be supported in the env variables. version: 2.0 workflows: demo: type: direct input: - query output: - records tasks: query: action: std.sql query={$.query} publish: records: $ ...