Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Renat Akhmerov]

> On 27 Dec 2014, at 01:39, W Chan  wrote:
> 
> > What you’re saying is that whatever is under “$.env” is just the exact same 
> > environment that we passed when we started the workflow? If yes then it 
> > definitely makes sense to me (it just allows to explicitly access 
> > environment, not through the implicit variable lookup). Please confirm.
> Yes. the $.env that I original proposed would be the same dict as the one 
> supplied at start_workflow.  Although we have to agree whether the variables 
> in the environment are allowed to change after the WF started.  Unless 
> there's a valid use case, I would lean toward making env immutable.

Let’s make them immutable. 

> > One thing that I strongly suggest is that we clearly define all reserved 
> > keys like “env”, “__actions” etc. I think it’d be better if they all 
> > started with the same prefix, for example, double underscore.
> 
> Agree. How about using double underscore for env as well (i.e. $.__env.var1, 
> $.__env.var2)?

Yes.


Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[W Chan]

> What you’re saying is that whatever is under “$.env” is just the exact same 
> environment that we passed when we started the workflow? If yes then it 
> definitely makes sense to me (it just allows to explicitly access 
> environment, not through the implicit variable lookup). Please confirm.

Yes. the $.env that I original proposed would be the same dict as the one
supplied at start_workflow.  Although we have to agree whether the
variables in the environment are allowed to change after the WF started.
Unless there's a valid use case, I would lean toward making env immutable.

> One thing that I strongly suggest is that we clearly define all reserved
keys like “env”, “__actions” etc. I think it’d be better if they all
started with the same prefix, for example, double underscore.

Agree. How about using double underscore for env as well (i.e.
$.__env.var1, $.__env.var2)?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Nikolay Makhotkin]

>
> One thing that I strongly suggest is that we clearly define all reserved
> keys like “env”, “__actions” etc. I think it’d be better if they all
> started with the same prefix, for example, double underscore.


I absolutely agree here. We should use specific keywords with "__" prefix
like we used "__executions".

On Thu, Dec 25, 2014 at 8:51 AM, Renat Akhmerov 
wrote:

>
> On 24 Dec 2014, at 23:37, W Chan  wrote:
>
> >* 2) Retuning to first example:
> *>* ...
> *>*  action: std.sql conn_str={$.env.conn_str} query={$.query}
> *>* ...
> *>* $.env - is it a name of environment or it will be a registered syntax to 
> getting access to values from env ?
> *
>
> I was actually thinking the environment will use the reserved word "env" in 
> the WF context.  The value for the "env" key will be the dict supplied either 
> DB lookup by name, by dict, or by JSON from CLI.
>
> Ok, probably here’s the place where I didn’t understand you before. I
> thought “env” here is just a arbitrary key that users themselves may want
> to have to just group some variables under a single umbrella. What you’re
> saying is that whatever is under “$.env” is just the exact same environment
> that we passed when we started the workflow? If yes then it definitely
> makes sense to me (it just allows to explicitly access environment, not
> through the implicit variable lookup). Please confirm.
>
> One thing that I strongly suggest is that we clearly define all reserved
> keys like “env”, “__actions” etc. I think it’d be better if they all
> started with the same prefix, for example, double underscore.
>
> The nested dict for "__actions" (and all other keys with double underscore) 
> is special system purpose, in this case declaring defaults for action inputs. 
>  Similar to "__execution" where it's for containing runtime data for the WF 
> execution.
>
> Yes, that’s clear
>
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Best Regards,
Nikolay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Renat Akhmerov]

> On 24 Dec 2014, at 23:37, W Chan  wrote:
> > 2) Retuning to first example:
> > ...
> >  action: std.sql conn_str={$.env.conn_str} query={$.query}
> > ...
> > $.env - is it a name of environment or it will be a registered syntax to 
> > getting access to values from env ?
> I was actually thinking the environment will use the reserved word "env" in 
> the WF context.  The value for the "env" key will be the dict supplied either 
> DB lookup by name, by dict, or by JSON from CLI.
Ok, probably here’s the place where I didn’t understand you before. I thought 
“env” here is just a arbitrary key that users themselves may want to have to 
just group some variables under a single umbrella. What you’re saying is that 
whatever is under “$.env” is just the exact same environment that we passed 
when we started the workflow? If yes then it definitely makes sense to me (it 
just allows to explicitly access environment, not through the implicit variable 
lookup). Please confirm.

One thing that I strongly suggest is that we clearly define all reserved keys 
like “env”, “__actions” etc. I think it’d be better if they all started with 
the same prefix, for example, double underscore.

> The nested dict for "__actions" (and all other keys with double underscore) 
> is special system purpose, in this case declaring defaults for action inputs. 
>  Similar to "__execution" where it's for containing runtime data for the WF 
> execution.

Yes, that’s clear


Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[W Chan]

Trying to clarify a few things...

>* 2) Retuning to first example:
*>* ...
*>*  action: std.sql conn_str={$.env.conn_str} query={$.query}
*>* ...
*>* $.env - is it a name of environment or it will be a registered
syntax to getting access to values from env ?
*

I was actually thinking the environment will use the reserved word
"env" in the WF context.  The value for the "env" key will be the dict
supplied either DB lookup by name, by dict, or by JSON from CLI.

The nested dict for "__actions" (and all other keys with double
underscore) is special system purpose, in this case declaring defaults
for action inputs.  Similar to "__execution" where it's for containing
runtime data for the WF execution.

>* 3) Can user has a few environments?*

I don't think we intend to mix one or more environments in a WF
execution.  The key was to supply any named environment at WF
execution time.  So the WF auth only needs to know the variables will
be under $.env.  If we allow one or more environments in a WF
execution, this means each environment needs to be referred to by name
(i.e. in your example env1 and env2).  We then would lost the ability
to swap any named environments for different executions of the same
WF.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Anastasia Kuznetsova]

Renat,

thanks for response!
One more question:

 So that same workflows could be running in different environments


Asking about using a few environments I meant within one workflow. For
example I need to work with two DB and I have two environments: env1 =
{"conn_str": ip, "user": user, "password": passwd} and env2 = {"conn_str":
ip2, "user": user2, "password": passwd2}. Will it be possible to do
something like this:

tasks:
   connect_first_db:
  action: std.sql conn_str={$.env1.conn_str} query={$.query}
  publish:
 records: $
   connect_second_db:
  action: std.sql conn_str={$.env2.conn_str} query={$.query}
  publish:
 records: $


Thanks,
Anastasia Kuznetsova

On Wed, Dec 24, 2014 at 2:19 PM, Renat Akhmerov 
wrote:

>
> On 24 Dec 2014, at 14:06, Anastasia Kuznetsova 
> wrote:
>
> 1) How does the end user will pass env variables to workflow?Will you add
> one more optional parameter to execution-create command?
> mistral execution-create wf wf_input wf_params wf_env
> If yes than what is wf_env will be, json file?
>
>
> Yes. IMO it should be possible to specify either a string (name of a
> previously stored environment) or a json file (so called ad-hoc
> environment).
>
> 2) Retuning to first example:
> ...
>  action: std.sql conn_str={$.env.conn_str} query={$.query}
> ...
> $.env - is it a name of environment or it will be a registered syntax to
> getting access to values from env ?
>
>
> So far we agreed that ‘key' should not be a registered key. Environment
> (optionally specified) is just another storage of variables going after
> workflow context in a lookup chain. So that if somewhere in a wf we have an
> expression $.something then this “something” will be first looked in
> workflow context and if it doesn’t exist there then looked in the specified
> environment.
> But if we want to explicitly group a set of variables we can use any
> (except for reserved as "__actions" ) key, for example, “env”.
>
> 3) Can user has a few environments?
>
>
> Yes. That’s one of the goals of introducing a concept of environment. So
> that same workflows could be running in different environments (e.g with
> different email settings, any kinds of passports etc.).
>
>
> Renat Akhmerov
> @ Mirantis Inc.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Renat Akhmerov]

> On 24 Dec 2014, at 14:06, Anastasia Kuznetsova  
> wrote:
> 
> 1) How does the end user will pass env variables to workflow?Will you add one 
> more optional parameter to execution-create command? 
> mistral execution-create wf wf_input wf_params wf_env
> If yes than what is wf_env will be, json file?

Yes. IMO it should be possible to specify either a string (name of a previously 
stored environment) or a json file (so called ad-hoc environment).

> 2) Retuning to first example:
> ...
>  action: std.sql conn_str={$.env.conn_str} query={$.query}
> ...
> $.env - is it a name of environment or it will be a registered syntax to 
> getting access to values from env ?

So far we agreed that ‘key' should not be a registered key. Environment 
(optionally specified) is just another storage of variables going after 
workflow context in a lookup chain. So that if somewhere in a wf we have an 
expression $.something then this “something” will be first looked in workflow 
context and if it doesn’t exist there then looked in the specified environment.
But if we want to explicitly group a set of variables we can use any (except 
for reserved as "__actions" ) key, for example, “env”.

> 3) Can user has a few environments?

Yes. That’s one of the goals of introducing a concept of environment. So that 
same workflows could be running in different environments (e.g with different 
email settings, any kinds of passports etc.).


Renat Akhmerov
@ Mirantis Inc.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Anastasia Kuznetsova]

Winson, Renat,

I have a few questions, because some of aspects aren't clear to me.

1) How does the end user will pass env variables to workflow?Will you add
one more optional parameter to execution-create command?
mistral execution-create wf wf_input wf_params wf_env
If yes than what is wf_env will be, json file?
2) Retuning to first example:
...
 action: std.sql conn_str={$.env.conn_str} query={$.query}
...
$.env - is it a name of environment or it will be a registered syntax to
getting access to values from env ?
3) Can user has a few environments?

On Wed, Dec 24, 2014 at 8:20 AM, Renat Akhmerov 
wrote:

> Thanks Winson,
>
> Since we discussed all this already I just want to confirm that I fully
> support this model, it will significantly help us make much more concise,
> readable and maintainable workflows. I spent a lot of time thinking about
> it and don’t see any problems with it. Nice job!
>
> However, all additional comments and questions are more than welcomed!
>
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
>
> On 24 Dec 2014, at 04:32, W Chan  wrote:
>
> After some online discussions with Renat, the following is a revision of
> the proposal to address the following related blueprints.
> *
> https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment
> * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context
> *
> https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values
> * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context
>
> Please refer to the following threads for backgrounds.
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html
> *
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html
>
>
> *Workflow Context Scope*
> 1. context to workflow is passed to all its subflows and subtasks/actions
> (aka children) only explicitly via inputs
> 2. context are passed by value (copy.deepcopy) to children
> 3. change to context is passed to parent only when it's explicitly
> published at the end of the child execution
> 4. change to context at the parent (after a publish from a child) is
> passed to subsequent children
>
> *Environment Variables*
> Solves the problem for quickly passing pre-defined inputs to a WF
> execution.  In the WF spec, environment variables are referenced as
> $.env.var1, $.env.var2, etc.  We should implement an API and DB model
> where users can pre-defined different environments with their own set of
> variables.  An environment can be passed either by name from the DB or
> adhoc by dict in start_workflow.  On workflow execution, a copy of the
> environment is saved with the execution object.  Action inputs are still
> declared explicitly in the WF spec.  This does not solve the problem where
> common inputs are specified over and over again.  So if there are multiple
> SQL tasks in the WF, the WF author still needs to supply the conn_str
> explicitly for each task.  In the example below, let's say we have a SQL
> Query Action that takes a connection string and a query statement as
> inputs.  The WF author can specify that the conn_str input is supplied from
> the $.env.conn_str.
>
> *Example:*
>
> # Assume this SqlAction is registered as std.sql in Mistral's Action table.
> class SqlAction(object):
> def __init__(self, conn_str, query):
> ...
>
> ...
>
> version: "2.0"
> workflows:
> demo:
> type: direct
> input:
> - query
> output:
> - records
> tasks:
> query:
> action: std.sql conn_str={$.env.conn_str} query={$.query}
> publish:
> records: $
>
> ...
>
> my_adhoc_env = {
> "conn_str": "mysql://admin:secrete@localhost/test"
> }
>
> ...
>
> # adhoc by dict
> start_workflow(wf_name, wf_inputs, env=my_adhoc_env)
>
> OR
>
> # lookup by name from DB model
> start_workflow(wf_name, wf_inputs, env="my_lab_env")
>
>
> *Define Default Action Inputs as Environment Variables*
> Solves the problem where we're specifying the same inputs to subflows and
> subtasks/actions over and over again.  On command execution, if action
> inputs are not explicitly supplied, then defaults will be lookup from the
> environment.
>
> *Example:*
> Using the same example from above, the WF author can still supply both
> conn_str and query inputs in the WF spec.  However, the author also has the
> option to supply that as default action inputs.  An example environment
> structure is below.  "__actions" should be reserved and immutable.  Users
> can specific one or more default inputs for the sql action as nested dict
> under "__actions".  Recursive YAQL eval should be supported in the env
> variables.
>
> version: "2.0"
> workflows:
> demo:
> type: direct
> input:
> - query
> output:
> - records
> tasks:
>   

Re: [openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[Renat Akhmerov]

Thanks Winson,

Since we discussed all this already I just want to confirm that I fully support 
this model, it will significantly help us make much more concise, readable and 
maintainable workflows. I spent a lot of time thinking about it and don’t see 
any problems with it. Nice job!

However, all additional comments and questions are more than welcomed!


Renat Akhmerov
@ Mirantis Inc.



> On 24 Dec 2014, at 04:32, W Chan  wrote:
> 
> After some online discussions with Renat, the following is a revision of the 
> proposal to address the following related blueprints.
> * 
> https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment 
> 
> * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context 
> 
> * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values 
> 
> * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context 
> 
> 
> Please refer to the following threads for backgrounds.
> * 
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html 
> 
> * 
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html 
> 
> * 
> http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html 
> 
> 
> 
> Workflow Context Scope
> 1. context to workflow is passed to all its subflows and subtasks/actions 
> (aka children) only explicitly via inputs
> 2. context are passed by value (copy.deepcopy) to children
> 3. change to context is passed to parent only when it's explicitly published 
> at the end of the child execution
> 4. change to context at the parent (after a publish from a child) is passed 
> to subsequent children
> 
> Environment Variables
> Solves the problem for quickly passing pre-defined inputs to a WF execution.  
> In the WF spec, environment variables are referenced as $.env.var1, 
> $.env.var2, etc.  We should implement an API and DB model where users can 
> pre-defined different environments with their own set of variables.  An 
> environment can be passed either by name from the DB or adhoc by dict in 
> start_workflow.  On workflow execution, a copy of the environment is saved 
> with the execution object.  Action inputs are still declared explicitly in 
> the WF spec.  This does not solve the problem where common inputs are 
> specified over and over again.  So if there are multiple SQL tasks in the WF, 
> the WF author still needs to supply the conn_str explicitly for each task.  
> In the example below, let's say we have a SQL Query Action that takes a 
> connection string and a query statement as inputs.  The WF author can specify 
> that the conn_str input is supplied from the $.env.conn_str.
> 
> Example:
> 
> # Assume this SqlAction is registered as std.sql in Mistral's Action table.
> class SqlAction(object):
> def __init__(self, conn_str, query):
> ...
> 
> ...
> 
> version: "2.0"
> workflows:
> demo:
> type: direct
> input:
> - query
> output:
> - records
> tasks:
> query:
> action: std.sql conn_str={$.env.conn_str} query={$.query}
> publish:
> records: $
> 
> ...
> 
> my_adhoc_env = {
> "conn_str": "mysql://admin:secrete 
> @localhost/test"
> }
> 
> ...
> 
> # adhoc by dict
> start_workflow(wf_name, wf_inputs, env=my_adhoc_env)
> 
> OR
> 
> # lookup by name from DB model
> start_workflow(wf_name, wf_inputs, env="my_lab_env")
> 
> Define Default Action Inputs as Environment Variables
> Solves the problem where we're specifying the same inputs to subflows and 
> subtasks/actions over and over again.  On command execution, if action inputs 
> are not explicitly supplied, then defaults will be lookup from the 
> environment.
> 
> Example:
> Using the same example from above, the WF author can still supply both 
> conn_str and query inputs in the WF spec.  However, the author also has the 
> option to supply that as default action inputs.  An example environment 
> structure is below.  "__actions" should be reserved and immutable.  Users can 
> specific one or more default inputs for the sql action as nested dict under 
> "__actions".  Recursive YAQL eval should be supported in the env variables.
> 
> version: "2.0"
> workflows:
> demo:
> type: direct
> input:
> - query
> output:
> - records
> tasks:
> query:
> action

[openstack-dev] [Mistral] Converging discussions on WF context and WF/task/action inputs

[W Chan]

After some online discussions with Renat, the following is a revision of
the proposal to address the following related blueprints.
*
https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment
* https://blueprints.launchpad.net/mistral/+spec/mistral-global-context
*
https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values
* https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context

Please refer to the following threads for backgrounds.
*
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html
*
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html
*
http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html


*Workflow Context Scope*
1. context to workflow is passed to all its subflows and subtasks/actions
(aka children) only explicitly via inputs
2. context are passed by value (copy.deepcopy) to children
3. change to context is passed to parent only when it's explicitly
published at the end of the child execution
4. change to context at the parent (after a publish from a child) is passed
to subsequent children

*Environment Variables*
Solves the problem for quickly passing pre-defined inputs to a WF
execution.  In the WF spec, environment variables are referenced as
$.env.var1, $.env.var2, etc.  We should implement an API and DB model where
users can pre-defined different environments with their own set of
variables.  An environment can be passed either by name from the DB or
adhoc by dict in start_workflow.  On workflow execution, a copy of the
environment is saved with the execution object.  Action inputs are still
declared explicitly in the WF spec.  This does not solve the problem where
common inputs are specified over and over again.  So if there are multiple
SQL tasks in the WF, the WF author still needs to supply the conn_str
explicitly for each task.  In the example below, let's say we have a SQL
Query Action that takes a connection string and a query statement as
inputs.  The WF author can specify that the conn_str input is supplied from
the $.env.conn_str.

*Example:*

# Assume this SqlAction is registered as std.sql in Mistral's Action table.
class SqlAction(object):
def __init__(self, conn_str, query):
...

...

version: "2.0"
workflows:
demo:
type: direct
input:
- query
output:
- records
tasks:
query:
action: std.sql conn_str={$.env.conn_str} query={$.query}
publish:
records: $

...

my_adhoc_env = {
"conn_str": "mysql://admin:secrete @localhost
/test"
}

...

# adhoc by dict
start_workflow(wf_name, wf_inputs, env=my_adhoc_env)

OR

# lookup by name from DB model
start_workflow(wf_name, wf_inputs, env="my_lab_env")


*Define Default Action Inputs as Environment Variables*
Solves the problem where we're specifying the same inputs to subflows and
subtasks/actions over and over again.  On command execution, if action
inputs are not explicitly supplied, then defaults will be lookup from the
environment.

*Example:*
Using the same example from above, the WF author can still supply both
conn_str and query inputs in the WF spec.  However, the author also has the
option to supply that as default action inputs.  An example environment
structure is below.  "__actions" should be reserved and immutable.  Users
can specific one or more default inputs for the sql action as nested dict
under "__actions".  Recursive YAQL eval should be supported in the env
variables.

version: "2.0"
workflows:
demo:
type: direct
input:
- query
output:
- records
tasks:
query:
action: std.sql query={$.query}
publish:
records: $

...

my_adhoc_env = {
"sql_server": "localhost",
"__actions": {
"std.sql": {
"conn_str": "mysql://admin:secrete@{$.env.sql_server}/test"
 }
}
}


*Default Input Values Supplied Explicitly in WF Spec*
Please refer to this blueprint

for background.  This is a different use case.  To support, we just need to
set the correct order of precedence in applying values.
1. Input explicitly given to the sub flow/task in the WF spec
2. Default input supplied from env
3. Default input supplied at WF spec

*Putting this together...*
At runtime, the WF context would be similar to the following example.  This
will be use to recursively eval the inputs for subflow/tasks/actions.

ctx = {
   "var1": …,
   "var2": …,
   "my_server_ip": "10.1.23.250",
   "env": {
"sql_server": "localhost",
"__actions": {
"std.sql": {
"conn": "mysql://admin:secrete@{$.env.sql_server}/test"
},
"my.action": {
"endpoint": "http://{$.my_server_ip}/v1/foo";
}