Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[ELISHA, Moshe (Moshe)]

I agree with the task object as well.
It should be an object you can get info like the start time, end time, error 
code, ...

-Original Message-
From: Renat Akhmerov [mailto:rakhme...@mirantis.com] 
Sent: יום ב 07 ספטמבר 2015 19:59
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL 
function


> On 07 Sep 2015, at 19:18, Stan Lagun <sla...@mirantis.com> wrote:
> 
> I believe this is a good change. $.task_name requires you that $ be 
> pointing to a tasks dictionary. But in the middle of the query like 
> [1.2.3].select($ + 1)  "$" will change its value. With a function approach 
> you can write [1, 2, 3].select($ + task(taskName)). However the name "task" 
> looks confusing as to my understanding tasks may have attributes other than 
> result. It may make sense to use task(taskName).result instead.

Yes, I like the idea that task() should be more than just a result. Good point!

Renat Akhmerov
@ Mirantis Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Stan Lagun]

I believe this is a good change. $.task_name requires you that $ be
pointing to a tasks dictionary. But in the middle of the query like
[1.2.3].select($ + 1)  "$" will change its value. With a function approach
you can write [1, 2, 3].select($ + task(taskName)). However the name "task"
looks confusing as to my understanding tasks may have attributes other than
result. It may make sense to use task(taskName).result instead.

Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis



On Sun, Sep 6, 2015 at 12:14 AM, Dmitri Zimine 
wrote:

> Yes meant to ask for consistency of referencing to task results. So it’s
> task(task_name) regardless of where.
>
> One use case in favor of this is tooling: I refactor workflow with an
> automated tool which wants to automatically rename the task name
> EVERYWHERE. You guys know well by now that renaming the task is a source of
> too many frustrating errors :)
>
> What other think?
>
> DZ.
>
> On Sep 3, 2015, at 4:23 AM, Renat Akhmerov  wrote:
>
>
> On 02 Sep 2015, at 21:01, Dmitri Zimine  wrote:
>
> Agree,
>
> with one detail: make it explicit -  task(task_name).
>
>
> So do you suggest we just replace res() with task() and it looks like
>
> task() - get task result when we are in “publish”
> task(task_name) - get task result from anywhere
>
> ?
>
> Is that correct you mean we must always specify a task name? The reason
> I’d like to have a simplified form (w/o task name) is that I see a lot of
> workflows that we have to repeat task name in publish so that it just look
> too verbose to me. Especially in case of very long task name.
>
> Consider something like this:
>
> tasks:
>   *get_volumes_by_names*:
> with-items: name in <% $.vol_names %>
> workflow: get_volume_by_name name=<% $.name %>
> publish:
>   volumes: <% $.*get_volumes_by_names* %>
>
> So in publish we have to repeat a task name, there’s no other way now. I’d
> like to soften this requirement, but if you still want to use task names
> you’ll be able to.
>
>
> res - we often see folks confused by result of what (action, task,
> workflow) although we cleaned up our lingo: action-output, task-result,
> workflow-output…. but still worth being explicit.
>
> And full result is being thought as the root context $.
>
> Publishing to global context may be ok for now, IMO.
>
>
> Not sure what you meant by "Publishing to global context”. Can you clarify
> please?
>
>
> Renat Akhmerov
> @ Mirantis Inc.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Renat Akhmerov]

> On 07 Sep 2015, at 19:18, Stan Lagun  wrote:
> 
> I believe this is a good change. $.task_name requires you that $ be pointing 
> to a tasks dictionary. But in the middle of the query like [1.2.3].select($ + 
> 1)  "$" will change its value. With a function approach
> you can write [1, 2, 3].select($ + task(taskName)). However the name "task" 
> looks confusing as to my understanding tasks may have attributes other than 
> result. It may make sense to use task(taskName).result instead.

Yes, I like the idea that task() should be more than just a result. Good point!

Renat Akhmerov
@ Mirantis Inc.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Dmitri Zimine]

Yes meant to ask for consistency of referencing to task results. So it’s 
task(task_name) regardless of where. 

One use case in favor of this is tooling: I refactor workflow with an automated 
tool which wants to automatically rename the task name EVERYWHERE. You guys 
know well by now that renaming the task is a source of too many frustrating 
errors :)

What other think? 

DZ. 

On Sep 3, 2015, at 4:23 AM, Renat Akhmerov  wrote:

> 
>> On 02 Sep 2015, at 21:01, Dmitri Zimine  wrote:
>> 
>> Agree, 
>> 
>> with one detail: make it explicit -  task(task_name). 
> 
> So do you suggest we just replace res() with task() and it looks like
> 
> task() - get task result when we are in “publish”
> task(task_name) - get task result from anywhere
> 
> ?
> 
> Is that correct you mean we must always specify a task name? The reason I’d 
> like to have a simplified form (w/o task name) is that I see a lot of 
> workflows that we have to repeat task name in publish so that it just look 
> too verbose to me. Especially in case of very long task name.
> 
> Consider something like this:
> 
> tasks:
>   get_volumes_by_names:
> with-items: name in <% $.vol_names %>
> workflow: get_volume_by_name name=<% $.name %>
> publish:
>   volumes: <% $.get_volumes_by_names %>
> 
> So in publish we have to repeat a task name, there’s no other way now. I’d 
> like to soften this requirement, but if you still want to use task names 
> you’ll be able to.
> 
> 
>> res - we often see folks confused by result of what (action, task, workflow) 
>> although we cleaned up our lingo: action-output, task-result, 
>> workflow-output…. but still worth being explicit.
>> 
>> And full result is being thought as the root context $.
>> 
>> Publishing to global context may be ok for now, IMO.
> 
> Not sure what you meant by "Publishing to global context”. Can you clarify 
> please?
> 
> 
> Renat Akhmerov
> @ Mirantis Inc.
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Renat Akhmerov]

> On 02 Sep 2015, at 21:01, Dmitri Zimine  wrote:
> 
> Agree, 
> 
> with one detail: make it explicit -  task(task_name). 

So do you suggest we just replace res() with task() and it looks like

task() - get task result when we are in “publish”
task(task_name) - get task result from anywhere

?

Is that correct you mean we must always specify a task name? The reason I’d 
like to have a simplified form (w/o task name) is that I see a lot of workflows 
that we have to repeat task name in publish so that it just look too verbose to 
me. Especially in case of very long task name.

Consider something like this:

tasks:
  get_volumes_by_names:
with-items: name in <% $.vol_names %>
workflow: get_volume_by_name name=<% $.name %>
publish:
  volumes: <% $.get_volumes_by_names %>

So in publish we have to repeat a task name, there’s no other way now. I’d like 
to soften this requirement, but if you still want to use task names you’ll be 
able to.


> res - we often see folks confused by result of what (action, task, workflow) 
> although we cleaned up our lingo: action-output, task-result, 
> workflow-output…. but still worth being explicit.
> 
> And full result is being thought as the root context $.
> 
> Publishing to global context may be ok for now, IMO.

Not sure what you meant by "Publishing to global context”. Can you clarify 
please?


Renat Akhmerov
@ Mirantis Inc.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Nikolay Makhotkin]

Hi,

I also thought about that recently. So, I absolutely agree with this
proposal. It would be nice to see this feature in Liberty.

On Wed, Sep 2, 2015 at 2:17 PM, Renat Akhmerov <rakhme...@mirantis.com>
wrote:

> FYI
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
>
> Begin forwarded message:
>
> *From: *Renat Akhmerov <rakhme...@mirantis.com>
> *Subject: **[openstack-dev][mistral][yaql] Addressing task result using
> YAQL function*
> *Date: *2 Sep 2015 17:16:27 GMT+6
> *To: *"OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
>
> Hi,
>
> I’d like to propose a few changes after transition to yaql 1.0:
>
> We already moved from using “$.__execution” in DSL to "execution()” and
> from “$.__env” to “env()” where “execution()” and “env()" are registered
> yaql functions. We had to do it because double underscored are prohibited
> in the new yaql.
>
>
> In the spirit of these changes I’m proposing a similar change for
> addressing task result in Mistral DSL.
>
> Currently we have the syntax “$.task_name” that we can use in yaql
> expressions to refer to corresponding task result. The current
> implementation is of that is kind of tricky and, IMO, conceptually wrong
> because referencing this kind of key leads to DB read operation that’s
> requried to fetch task result (it may be big as megabytes so it can’t be
> stored in workflow context all the time. In other words, we create a
> dictionary with side effect and change the initial semantics of dictionary.
> Along with mentioned trickiness of this approach it’s not convenient, for
> example, to debug the code because we can accidentally cause a DB
> operation.
>
> So the solution I’m proposing is to have an explicit yaql function called
> “res” or “result” to extract task result.
>
> *res()* - extracts the result of the current task, i.e. in “publish”
> clause.
> *res(‘task_name’)* - extracts the result of the task with the specified
> name. Can also be used in “publish” clause, if needed
>
> This approach seems more flexible (cause we can add any other functions
> w/o having to make significant changes in WF engine) and expressive because
> user won’t confuse $.task_name with addressing a regular workflow context
> variable.
>
> Of course, this to some extent breaks backwards compatibility. But we
> already changed yaql version which was necessary anyway so it seems like a
> good time to do it.
>
> I’d very much like to have your input on this.
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
>
>
>


-- 
Best Regards,
Nikolay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Renat Akhmerov]

Hi,

I’d like to propose a few changes after transition to yaql 1.0:

We already moved from using “$.__execution” in DSL to "execution()” and from 
“$.__env” to “env()” where “execution()” and “env()" are registered yaql 
functions. We had to do it because double underscored are prohibited in the new 
yaql.


In the spirit of these changes I’m proposing a similar change for addressing 
task result in Mistral DSL.

Currently we have the syntax “$.task_name” that we can use in yaql expressions 
to refer to corresponding task result. The current implementation is of that is 
kind of tricky and, IMO, conceptually wrong because referencing this kind of 
key leads to DB read operation that’s requried to fetch task result (it may be 
big as megabytes so it can’t be stored in workflow context all the time. In 
other words, we create a dictionary with side effect and change the initial 
semantics of dictionary. Along with mentioned trickiness of this approach it’s 
not convenient, for example, to debug the code because we can accidentally 
cause a DB operation. 

So the solution I’m proposing is to have an explicit yaql function called “res” 
or “result” to extract task result.

res() - extracts the result of the current task, i.e. in “publish” clause.
res(‘task_name’) - extracts the result of the task with the specified name. Can 
also be used in “publish” clause, if needed

This approach seems more flexible (cause we can add any other functions w/o 
having to make significant changes in WF engine) and expressive because user 
won’t confuse $.task_name with addressing a regular workflow context variable.

Of course, this to some extent breaks backwards compatibility. But we already 
changed yaql version which was necessary anyway so it seems like a good time to 
do it.

I’d very much like to have your input on this.

Renat Akhmerov
@ Mirantis Inc.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function

[Dmitri Zimine]

Agree, 

with one detail: make it explicit -  task(task_name). 

res - we often see folks confused by result of what (action, task, workflow) 
although we cleaned up our lingo: action-output, task-result, workflow-output…. 
but still worth being explicit.

And full result is being thought as the root context $.

Publishing to global context may be ok for now, IMO.

DZ>

On Sep 2, 2015, at 4:16 AM, Renat Akhmerov  wrote:

> Hi,
> 
> I’d like to propose a few changes after transition to yaql 1.0:
> 
> We already moved from using “$.__execution” in DSL to "execution()” and from 
> “$.__env” to “env()” where “execution()” and “env()" are registered yaql 
> functions. We had to do it because double underscored are prohibited in the 
> new yaql.
> 
> 
> In the spirit of these changes I’m proposing a similar change for addressing 
> task result in Mistral DSL.
> 
> Currently we have the syntax “$.task_name” that we can use in yaql 
> expressions to refer to corresponding task result. The current implementation 
> is of that is kind of tricky and, IMO, conceptually wrong because referencing 
> this kind of key leads to DB read operation that’s requried to fetch task 
> result (it may be big as megabytes so it can’t be stored in workflow context 
> all the time. In other words, we create a dictionary with side effect and 
> change the initial semantics of dictionary. Along with mentioned trickiness 
> of this approach it’s not convenient, for example, to debug the code because 
> we can accidentally cause a DB operation. 
> 
> So the solution I’m proposing is to have an explicit yaql function called 
> “res” or “result” to extract task result.
> 
> res() - extracts the result of the current task, i.e. in “publish” clause.
> res(‘task_name’) - extracts the result of the task with the specified name. 
> Can also be used in “publish” clause, if needed
> 
> This approach seems more flexible (cause we can add any other functions w/o 
> having to make significant changes in WF engine) and expressive because user 
> won’t confuse $.task_name with addressing a regular workflow context variable.
> 
> Of course, this to some extent breaks backwards compatibility. But we already 
> changed yaql version which was necessary anyway so it seems like a good time 
> to do it.
> 
> I’d very much like to have your input on this.
> 
> Renat Akhmerov
> @ Mirantis Inc.
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev